How grades work

A grade says how much work a task needs before you use it.

Every task gets a number from 1 to 10 and a confidence letter. The number is the effort to fix it. The letter says how much of the grade a person has checked.

Reading a grade

7LKC

7Quick fix before use: minutes of workLKThe problem: the answer is in the environmentCAutomated, with an open question behind it

The scale

Nothing below 8 should be used as it is.

GradeBucketWhat it meansFix effort

10useTwo reviewers found nothing, and it was stable over three runs. Rare by design.None

9useA reviewer found nothing that affects scoring.None

8useUsable as it is, with minor cautions, such as ground truth that may drift. The highest grade automated checks can give.None now

7fixQuick fix before use.Minutes

6fixSmall fix before use.Under an hour

5fixModerate fix before use.A few hours

4fixMajor fix before use.A day or more

3tossCheaper to rewrite. The environment or data can be reused.Rewrite

2tossCheaper to rewrite. Only the idea survives.Rewrite

1tossNothing worth keeping.Start over

NGNo gradeCould not be assessed: no reference solution, the grader needs vendor credentials, or the environment asks for unsafe host access.Resubmit with what is missing

Grades have some room. A task with several quick fixes can end up a 5.

The confidence letter

A person reviewed the task. Required for 9 and 10.

Automated. The grade rests only on file checks and container runs.

Automated, and part of the grade rests on a model's answer or an open question: a flag, an unsure answer, a partial reward, or checks that have not run.

Automated audits top out at 8. A wrong answer key can pass every automated check we have, so 9 and 10 need a person. An 8 · C means we found nothing; an 8 · A means a person checked and agreed.

Qualifiers

LKThe answer is in the environmentInflates

LXThe tests check less than the task asksInflates

HKThe score can be had without the workInflates

WSIt does not test the skill it namesInflates

SMThe tests and the instruction disagreeDeflates

RNThe task cannot be run or reviewedDeflates

GBGrading depends on something that can breakDeflates

Any grade can carry any qualifier. Inflating problems make models look better than they are; deflating ones make them look worse. An answer leaked through Git history left in the image is a one-line fix, so 7 LK. An answer published online needs a new task, so 2 LK.

How automated checks set the grade

Rubric 1.0

task.toml does not parse · RNGit history kept in the image (flag until confirmed)Tests or solution copied into the image (flag until confirmed)Grader trusts an existing reward file (flag until confirmed)

Environment does not build · RNReference solution fails · GBFull reward for doing nothing · HKGrader always writes full reward (flag until confirmed)Passes when the required-test list is empty (flag until confirmed)

Missing instruction · RNMissing tests · RNMissing environment · RNNo reference solutionGrader needs vendor credentialsCompose file asks for host access

The grade is the lowest among the findings. Two or more findings drop it one more step.

A flag never sets the grade. It lowers the letter to C until a person confirms or clears it.

A container failure counts only when two runs agree. Runs that disagree give no grade.

Partial results

When an upload has no grade

A grade needs both container runs: the task with nothing done, and the task with its reference solution. They run in a sandbox with 4 CPUs, 8 GB of memory and 44 minutes. If a task asks for more, or the runs stop on our side, the result is partial: we report what the checks found and give no grade, because a task that passes every file check can still fail when it runs. Runs that stop on our side can be run again free.

The playground shows one. protein-assembly passes every file check, and its reference solution fails every run.

Open the playground