Research · Benchmark

RobloxBench

A lean public suite for scoring AI models on Roblox development work — Luau, Open Cloud, security, and analytics — not generic coding puzzles.

Method borrows from HumanEval (executable unit tests), SWE-bench (fail-to-pass bug repair), and LiveCodeBench (scenario breadth and honest skips when tooling is missing). Full write-up: benchmarks/robloxbench/METHODOLOGY.md in the shipLua repository.

Seed suite 0.1.0

32 tasks across 5 categories. Scores are pass rates over graded tasks; skipped tasks (for example missing Luau CLI) are excluded from the denominator.

Luau scripting
Write Luau modules graded by unit tests (HumanEval-style).
Roblox API Q&A
Short factual answers on Open Cloud, OAuth, webhooks, and analytics APIs.
Bug fixing
Repair broken scripts until fail-to-pass tests go green (SWE-bench-lite).
DataStore & remotes
Remotes, DataStores, replication secrets, and purchase grants.
Analytics reasoning
Reason over synthetic Analytics Query–shaped samples — never invented live KPIs.

Leaderboard

No results yet

Published runs will appear here from results/leaderboard.json. We do not show estimated or placeholder model scores.

RobloxBench is research infrastructure from shipLua. It is not affiliated with or endorsed by Roblox Corporation. Run the harness yourself from the repo; API keys stay in your environment.