NeuroShard / Research evidence / 13 September 2026

One model, two GPUs, measured learning

Three fixed training arms start from the same 1.7B seed. Checked task targets, a deliberately damaged-target control, and a shared two-GPU run separate target quality from hardware cooperation. These are operated experiments; the public native serving model and ledger are separate.

Narrow learning contract: passed. Exact task accuracy changed by +57.03 percentage points. Paired one-sided p = 1.12e-44; public retention loss upper bound = +0.006656 nats. This concerns four generated task families and previously exposed retention probes. It does not establish broad assistant quality, permissionless GPU verification, model growth or independent ownership.
Measured task accuracy, training time and inference throughput
ModelExact answersValid JSON objectsMean retention change
Seed23 / 256 (8.98%)198
Clean · 1 GPU169 / 256 (66.02%)256-0.002195
Damaged · 1 GPU152 / 256 (59.38%)256-0.003081
Clean · 2 GPUs168 / 256 (65.62%)256-0.002346

Methods and interpretation · Recomputed metrics · Frozen inputs · Candidate commitments · Checksums · Clean single weights · Shared training weights

All 256 final-test tasks

Every answer is shown. The two unseen prompt variants are marked 2 and 3. “Correct” requires both the requested JSON schema and exact values; valid JSON alone earns no credit.

001 · lookup · wording 0 · gainb08b8fe8ce21d90e201b9e8913df369810e39e5462fdba363a9eb134213e9b2b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H635","city":"Oslo","units":2},{"id":"F873","city":"Bern","units":0},{"id":"E150","city":"Suva","units":1},{"id":"T980","city":"Suva","units":2},{"id":"Y205","city":"Kyoto","units":11}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Oslo","units":2}]
12 tokens · 2.211s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 1.340s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 1.211s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.880s
002 · filter · wording 0 · gainc4a1edf54109a7055389d975cbda5395d2d6b55388689bfb611c7e531ec1206f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"U984","status":"ready"},{"id":"R199","status":"pending"},{"id":"J556","status":"pending"},{"id":"X404","status":"closed"},{"id":"A572","status":"pending"}]
Checked answer
{"ids": ["U984"]}
Seed

wrong_schema_or_value

{"ids": ["A572", "J556", "U984", "R199", "X404"]}
35 tokens · 0.972s
Clean · 1 GPU

Correct

{"ids":["U984"]}
11 tokens · 0.316s
Damaged · 1 GPU

Correct

{"ids":["U984"]}
11 tokens · 0.441s
Clean · 2 GPUs

Correct

{"ids":["U984"]}
11 tokens · 0.323s
003 · total · wording 0 · same16f14dd7711869cf8cceff9d1a73f972220e76b1fdcf7fcbe2d55b3039713346
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Q857","units":5,"price":7},{"id":"S261","units":6,"price":4},{"id":"F625","units":4,"price":16},{"id":"F534","units":18,"price":9},{"id":"F501","units":10,"price":8}]
Checked answer
{"total": 365}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.330s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.225s
004 · sort · wording 0 · gaine87acf19d3511a5733f97021a4ea5a0ea61a4506fe1e70ebe5c9c6449f873e8b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U687","priority":2},{"id":"X278","priority":6},{"id":"T638","priority":8},{"id":"Q352","priority":6},{"id":"E277","priority":2}]
Checked answer
{"ids": ["T638", "Q352"]}
Seed

wrong_schema_or_value

{"ids": ["U687", "X278", "T638", "Q352", "E277"]}
35 tokens · 0.959s
Clean · 1 GPU

Correct

{"ids":["T638","Q352"]}
16 tokens · 0.441s
Damaged · 1 GPU

Correct

{"ids":["T638","Q352"]}
16 tokens · 0.676s
Clean · 2 GPUs

Correct

{"ids":["T638","Q352"]}
16 tokens · 0.448s
005 · lookup · wording 1 · samea3432c82d7df4a034cd5ea3d4b8c17269b6f56665858f8a99e19ee8b9cca48b7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id C398 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"C398","city":"Oslo","units":0},{"id":"Z327","city":"Perth","units":15},{"id":"V662","city":"Perth","units":19},{"id":"Z297","city":"Suva","units":12},{"id":"O756","city":"Suva","units":4}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

Correct

{"city":"Oslo","units":0}
11 tokens · 0.307s
Clean · 1 GPU

Correct

{"city":"Oslo","units":0}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"city":"Oslo","units":0}
11 tokens · 0.480s
Clean · 2 GPUs

Correct

{"city":"Oslo","units":0}
11 tokens · 0.310s
006 · filter · wording 1 · gainc91ed8f3302daf312a35075051a1b2b56f3c947cf6e0d6b85a1a54d7ba27d9ca
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"W186","status":"pending"},{"id":"X341","status":"closed"},{"id":"J338","status":"pending"},{"id":"T557","status":"pending"},{"id":"D680","status":"ready"}]
Checked answer
{"ids": ["D680"]}
Seed

wrong_schema_or_value

{
  "ids": ["D680", "J338", "T557", "W186", "X341"]
}
38 tokens · 1.038s
Clean · 1 GPU

Correct

{"ids":["D680"]}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"ids":["D680"]}
11 tokens · 0.436s
Clean · 2 GPUs

Correct

{"ids":["D680"]}
11 tokens · 0.310s
007 · total · wording 1 · same43549dda2f3acf35cad28cb9c36d29890318ff78766be48ac6605bc99340ea75
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"G781","units":14,"price":2},{"id":"Z845","units":13,"price":17},{"id":"Q815","units":9,"price":13},{"id":"O480","units":8,"price":5},{"id":"Y725","units":10,"price":14}]
Checked answer
{"total": 546}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.341s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.226s
008 · sort · wording 1 · gaind623855de43a8ace034e8d6dcd32fff075eac2207741871f80744ca32b88e26f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"W217","priority":2},{"id":"T168","priority":1},{"id":"W989","priority":9},{"id":"S415","priority":6},{"id":"F529","priority":7}]
Checked answer
{"ids": ["W989", "F529"]}
Seed

invalid_json_object

[{"id":"W217","priority":2},{"id":"T168","priority":1},{"id":"W989","priority":9},{"id":"S415","priority":6},{"id":"F529","priority":7}]
62 tokens · 1.692s
Clean · 1 GPU

Correct

{"ids":["W989","F529"]}
16 tokens · 0.440s
Damaged · 1 GPU

Correct

{"ids":["W989","F529"]}
16 tokens · 0.633s
Clean · 2 GPUs

Correct

{"ids":["W989","F529"]}
16 tokens · 0.447s
009 · lookup · wording 2 · gain6b28c12ee2e6ff586e7737141c7743e03be89f7811619be78019a3c116dba29a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z498? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"R504","city":"Oslo","units":10},{"id":"E636","city":"Riga","units":13},{"id":"Z873","city":"Lima","units":13},{"id":"Z498","city":"Kyoto","units":8},{"id":"N279","city":"Bern","units":17}]
Checked answer
{"city": "Kyoto", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.417s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.331s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.480s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.337s
010 · filter · wording 2 · gain6189e6455bea7b59054965242f1ac401f2cd1cfa43ee57845e9fea6c1d9a079c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"W906","status":"ready"},{"id":"A719","status":"ready"},{"id":"I561","status":"ready"},{"id":"B204","status":"closed"},{"id":"V610","status":"closed"}]
Checked answer
{"ids": ["A719", "I561", "W906"]}
Seed

invalid_json_object

[{"id":"A719","status":"ready"},{"id":"I561","status":"ready"},{"id":"W906","status":"ready"},{"id":"V610","status":"closed"}]
51 tokens · 1.396s
Clean · 1 GPU

Correct

{"ids":["A719","I561","W906"]}
21 tokens · 0.578s
Damaged · 1 GPU

Correct

{"ids":["A719","I561","W906"]}
21 tokens · 0.830s
Clean · 2 GPUs

Correct

{"ids":["A719","I561","W906"]}
21 tokens · 0.585s
011 · total · wording 2 · same51235c1b08df14cbeb9258167c1be23dd57c577eac26dca84599fd9dd53ecc58
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"W347","units":19,"price":6},{"id":"Z828","units":3,"price":10},{"id":"I926","units":2,"price":16},{"id":"C603","units":3,"price":12},{"id":"I552","units":2,"price":18}]
Checked answer
{"total": 248}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.347s
Clean · 2 GPUs

wrong_schema_or_value

{"total":333}
8 tokens · 0.226s
012 · sort · wording 2 · same3b98ccea1f8843d84a7a1107a52ebcc5608170110ed1c3b8013220e418861268
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"U540","priority":3},{"id":"I773","priority":6},{"id":"O299","priority":8},{"id":"H103","priority":7},{"id":"V453","priority":1}]
Checked answer
{"ids": ["O299", "H103"]}
Seed

wrong_schema_or_value

{"ids": ["O299", "U540"]}
17 tokens · 0.469s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["O299","I773"]}
16 tokens · 0.443s
Damaged · 1 GPU

Correct

{"ids":["O299","H103"]}
16 tokens · 0.471s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["O299","I773"]}
16 tokens · 0.445s
013 · lookup · wording 3 · gainbdb080cc4f8c28ef45679f041fe0946d99873555e2f934bbd5f3fee6a0f06aad
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=H177. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"M381","city":"Accra","units":4},{"id":"H177","city":"Accra","units":18},{"id":"U691","city":"Riga","units":18},{"id":"R308","city":"Perth","units":1},{"id":"E977","city":"Riga","units":7}]
Checked answer
{"city": "Accra", "units": 18}
Seed

wrong_schema_or_value

{"city":"Accra","units":4}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Accra","units":18}
12 tokens · 0.338s
Damaged · 1 GPU

Correct

{"city":"Accra","units":18}
12 tokens · 0.338s
Clean · 2 GPUs

Correct

{"city":"Accra","units":18}
12 tokens · 0.336s
014 · filter · wording 3 · gain08e886c1888db28affffb280ad5c367a120f33e8373dbdaa235ed525c726e872
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"K597","status":"closed"},{"id":"J765","status":"ready"},{"id":"N160","status":"closed"},{"id":"E299","status":"ready"},{"id":"E512","status":"closed"}]
Checked answer
{"ids": ["E299", "J765"]}
Seed

wrong_schema_or_value

{"ids": ["J765", "E299", "E512", "K597", "N160"]}
35 tokens · 0.958s
Clean · 1 GPU

Correct

{"ids":["E299","J765"]}
16 tokens · 0.440s
Damaged · 1 GPU

Correct

{"ids":["E299","J765"]}
16 tokens · 0.453s
Clean · 2 GPUs

Correct

{"ids":["E299","J765"]}
16 tokens · 0.447s
015 · total · wording 3 · same71c1a9fb67ea0e37948d23926f149ad2a97298820054edd6f142286a4f064632
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"A619","units":4,"price":8},{"id":"B858","units":6,"price":18},{"id":"A408","units":19,"price":13},{"id":"U852","units":13,"price":16},{"id":"M840","units":7,"price":15}]
Checked answer
{"total": 700}
Seed

wrong_schema_or_value

{
  "total": 40
}
11 tokens · 0.307s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.224s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.230s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
016 · sort · wording 3 · gainb4dc63b65179c4906f27e5522e813545d1fce47dfb78564f60aff61cb301a225
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"S442","priority":3},{"id":"P982","priority":6},{"id":"R780","priority":1},{"id":"E296","priority":3},{"id":"R286","priority":4}]
Checked answer
{"ids": ["P982", "R286"]}
Seed

invalid_json_object

[{"id":"S442","priority":3},{"id":"P982","priority":6},{"id":"R780","priority":1},{"id":"E296","priority":3},{"id":"R286","priority":4}]
62 tokens · 1.697s
Clean · 1 GPU

Correct

{"ids":["P982","R286"]}
16 tokens · 0.448s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["P982","S442"]}
16 tokens · 0.451s
Clean · 2 GPUs

Correct

{"ids":["P982","R286"]}
16 tokens · 0.446s
017 · lookup · wording 0 · samedf20f552f853e8edd96572af89b9a864db8b42adf834dc565713158bd1a29da2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z213. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W903","city":"Bern","units":12},{"id":"D753","city":"Lima","units":11},{"id":"F927","city":"Bern","units":8},{"id":"F655","city":"Perth","units":5},{"id":"Z213","city":"Suva","units":2}]
Checked answer
{"city": "Suva", "units": 2}
Seed

Correct

{
  "city": "Suva",
  "units": 2
}
18 tokens · 0.498s
Clean · 1 GPU

Correct

{"city":"Suva","units":2}
11 tokens · 0.306s
Damaged · 1 GPU

Correct

{"city":"Suva","units":2}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"city":"Suva","units":2}
11 tokens · 0.309s
018 · filter · wording 0 · gainafe8bb04574002069633f5af5b5fdc6c641134bde6ed94391a7daee94ca1a547
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Y872","status":"closed"},{"id":"H853","status":"closed"},{"id":"D836","status":"pending"},{"id":"T843","status":"ready"},{"id":"Z138","status":"closed"}]
Checked answer
{"ids": ["T843"]}
Seed

wrong_schema_or_value

{"ids": ["D836", "T843", "Z138"]}
23 tokens · 0.636s
Clean · 1 GPU

Correct

{"ids":["T843"]}
11 tokens · 0.305s
Damaged · 1 GPU

Correct

{"ids":["T843"]}
11 tokens · 0.314s
Clean · 2 GPUs

Correct

{"ids":["T843"]}
11 tokens · 0.308s
019 · total · wording 0 · same7cc06d21b1408dc924ab935fa087dc6b34c652aec5a3be5377ea73e725c67833
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"J191","units":15,"price":12},{"id":"D848","units":5,"price":2},{"id":"O230","units":6,"price":5},{"id":"D354","units":16,"price":19},{"id":"Q709","units":11,"price":14}]
Checked answer
{"total": 678}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.232s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.226s
020 · sort · wording 0 · gain11f3a4676cd1a1a774ff8d7c6d41a629b1c955347c418beaffc5952fc24ad385
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"M612","priority":8},{"id":"I511","priority":2},{"id":"W399","priority":1},{"id":"S895","priority":5},{"id":"S742","priority":1}]
Checked answer
{"ids": ["M612", "S895"]}
Seed

wrong_schema_or_value

{"ids": ["I511", "W399", "S895", "S742", "M612"]}
35 tokens · 0.963s
Clean · 1 GPU

Correct

{"ids":["M612","S895"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["M612","S895"]}
16 tokens · 0.460s
Clean · 2 GPUs

Correct

{"ids":["M612","S895"]}
16 tokens · 0.448s
021 · lookup · wording 1 · gainedbcc5f3202110b01cbdae959906fda442872626db2e28248f247fe8306a5382
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"N970","city":"Accra","units":13},{"id":"J610","city":"Kyoto","units":9},{"id":"E224","city":"Kyoto","units":4},{"id":"A249","city":"Accra","units":14},{"id":"K547","city":"Lima","units":9}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Accra","units":13}
12 tokens · 0.333s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.281s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.288s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.287s
022 · filter · wording 1 · gaine555dc45a37313a05043b67b9d9283f77afe9ca3da3e064f5260215df599a654
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"K668","status":"closed"},{"id":"G229","status":"ready"},{"id":"W427","status":"pending"},{"id":"N127","status":"ready"},{"id":"E656","status":"ready"}]
Checked answer
{"ids": ["E656", "G229", "N127"]}
Seed

wrong_schema_or_value

{
  "ids": ["G229", "N127", "E656", "K668"]
}
32 tokens · 0.876s
Clean · 1 GPU

Correct

{"ids":["E656","G229","N127"]}
21 tokens · 0.585s
Damaged · 1 GPU

Correct

{"ids":["E656","G229","N127"]}
21 tokens · 0.591s
Clean · 2 GPUs

Correct

{"ids":["E656","G229","N127"]}
21 tokens · 0.590s
023 · total · wording 1 · same42c89f858b49eb8fe62d522325862927b79cc2cf760c4ae3eac949371cee0162
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"O226","units":13,"price":18},{"id":"E697","units":6,"price":6},{"id":"B163","units":13,"price":16},{"id":"Y513","units":5,"price":6},{"id":"U101","units":14,"price":1}]
Checked answer
{"total": 522}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.335s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.229s
024 · sort · wording 1 · gain3e0756631ab29918b3e21e6183c8054d1566f20fe32675482c6b41e62574648f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"C894","priority":7},{"id":"R639","priority":4},{"id":"L693","priority":9},{"id":"X945","priority":2},{"id":"S346","priority":5}]
Checked answer
{"ids": ["L693", "C894"]}
Seed

invalid_json_object

[{"id":"C894","priority":7},{"id":"R639","priority":4},{"id":"L693","priority":9},{"id":"X945","priority":2},{"id":"S346","priority":5}]
62 tokens · 1.697s
Clean · 1 GPU

Correct

{"ids":["L693","C894"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["L693","C894"]}
16 tokens · 0.451s
Clean · 2 GPUs

Correct

{"ids":["L693","C894"]}
16 tokens · 0.451s
025 · lookup · wording 2 · gainb6d80a3722d56ec4e0b07ac313360b43d9308543477ea2e6e1ba62d7bab946c0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to W391? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T892","city":"Oslo","units":11},{"id":"T894","city":"Accra","units":0},{"id":"J355","city":"Suva","units":8},{"id":"O991","city":"Riga","units":8},{"id":"W391","city":"Kyoto","units":9}]
Checked answer
{"city": "Kyoto", "units": 9}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": 9
}
16 tokens · 0.446s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":9}
12 tokens · 0.333s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":9}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":9}
12 tokens · 0.340s
026 · filter · wording 2 · gainb60450663de1bf8f85c9736bb320bf3d3ba94b2978f07a929033396162ba3c26
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"X740","status":"ready"},{"id":"Z600","status":"ready"},{"id":"I226","status":"ready"},{"id":"G927","status":"pending"},{"id":"M607","status":"ready"}]
Checked answer
{"ids": ["I226", "M607", "X740", "Z600"]}
Seed

invalid_json_object

[{"id":"X740","status":"ready"},{"id":"Z600","status":"ready"},{"id":"I226","status":"ready"},{"id":"G927","status":"pending"},{"id":"M607","status":"ready"}]
63 tokens · 1.731s
Clean · 1 GPU

Correct

{"ids":["I226","M607","X740","Z600"]}
26 tokens · 0.723s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["I226","M607","X740"]}
21 tokens · 0.587s
Clean · 2 GPUs

Correct

{"ids":["I226","M607","X740","Z600"]}
26 tokens · 0.729s
027 · total · wording 2 · same8264843620917a78bcc7d78126066819f79f1eb0b99ac46c4104c692331f6b9b
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V860","units":7,"price":6},{"id":"O233","units":9,"price":16},{"id":"Q267","units":8,"price":19},{"id":"B717","units":17,"price":13},{"id":"B371","units":4,"price":2}]
Checked answer
{"total": 567}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":410}
8 tokens · 0.228s
028 · sort · wording 2 · gain067efc76f5b98229c2389180b806aff4395e7a1bef9c0fc1796beccf2f55b295
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"L672","priority":4},{"id":"I528","priority":1},{"id":"J434","priority":3},{"id":"T270","priority":2},{"id":"U382","priority":2}]
Checked answer
{"ids": ["L672", "J434"]}
Seed

wrong_schema_or_value

{"ids": ["L672", "U382", "T270", "I528", "J434", "J434"]}
41 tokens · 1.120s
Clean · 1 GPU

Correct

{"ids":["L672","J434"]}
16 tokens · 0.451s
Damaged · 1 GPU

Correct

{"ids":["L672","J434"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["L672","J434"]}
16 tokens · 0.450s
029 · lookup · wording 3 · gain77009b47bf6af0f9758e3355179f9127f3a12203d6e802559b797bc16a292406
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=G260. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T996","city":"Lima","units":8},{"id":"J873","city":"Riga","units":16},{"id":"G260","city":"Perth","units":3},{"id":"G466","city":"Perth","units":7},{"id":"F183","city":"Oslo","units":17}]
Checked answer
{"city": "Perth", "units": 3}
Seed

wrong_schema_or_value

{"city":"Lima","units":8}
11 tokens · 0.303s
Clean · 1 GPU

Correct

{"city":"Perth","units":3}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Perth","units":3}
11 tokens · 0.306s
Clean · 2 GPUs

Correct

{"city":"Perth","units":3}
11 tokens · 0.312s
030 · filter · wording 3 · gain79ea51db7518140b3a2c2a75a5e116c8ab476abe2183aec955005a22c299f781
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"C623","status":"ready"},{"id":"K561","status":"closed"},{"id":"G720","status":"closed"},{"id":"F411","status":"closed"},{"id":"G530","status":"closed"}]
Checked answer
{"ids": ["C623"]}
Seed

wrong_schema_or_value

{"ids": ["C623", "K561", "G720", "F411", "G530"]}
35 tokens · 0.954s
Clean · 1 GPU

Correct

{"ids":["C623"]}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"ids":["C623"]}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"ids":["C623"]}
11 tokens · 0.312s
031 · total · wording 3 · same79c2c51060a173282581c0986b9a983ead29e35863edec662bfdb4d98a9f701b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"F554","units":4,"price":13},{"id":"A549","units":10,"price":2},{"id":"Y424","units":18,"price":13},{"id":"U347","units":5,"price":17},{"id":"X634","units":10,"price":7}]
Checked answer
{"total": 461}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.332s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.230s
032 · sort · wording 3 · sameca098496fd1aa688225b193f9ef24684fab99c6dfb53c57baa817165b26f7d34
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"C315","priority":3},{"id":"S748","priority":3},{"id":"R133","priority":4},{"id":"D365","priority":8},{"id":"P746","priority":4}]
Checked answer
{"ids": ["D365", "P746"]}
Seed

invalid_json_object

[{"id":"C315"},{"id":"S748"},{"id":"R133"},{"id":"D365"},{"id":"P746"}]
43 tokens · 1.174s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["D365","R133"]}
16 tokens · 0.449s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D365","R133"]}
16 tokens · 0.443s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["D365","R133"]}
16 tokens · 0.452s
033 · lookup · wording 0 · gainc0bc11ad04484aad6bbe0e3c64cd61931507ba822c9d62ae4d70df9540d9097c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record E991. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"B667","city":"Accra","units":16},{"id":"E991","city":"Bern","units":17},{"id":"G987","city":"Riga","units":8},{"id":"V244","city":"Lima","units":19},{"id":"E341","city":"Accra","units":0}]
Checked answer
{"city": "Bern", "units": 17}
Seed

invalid_json_object

[{"city":"Accra","units":16}]
13 tokens · 0.361s
Clean · 1 GPU

Correct

{"city":"Bern","units":17}
11 tokens · 0.308s
Damaged · 1 GPU

Correct

{"city":"Bern","units":17}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"city":"Bern","units":17}
11 tokens · 0.313s
034 · filter · wording 0 · gain14eb8daee0a257b9ada3223acfd1c039dab5606b8f3e1b3f8e9bc416c38a7cfc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"D183","status":"pending"},{"id":"S596","status":"pending"},{"id":"K601","status":"pending"},{"id":"Q734","status":"ready"},{"id":"K843","status":"ready"}]
Checked answer
{"ids": ["K843", "Q734"]}
Seed

wrong_schema_or_value

{"ids": ["D183", "K601", "K843", "Q734"]}
29 tokens · 0.796s
Clean · 1 GPU

Correct

{"ids":["K843","Q734"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["K843","Q734"]}
16 tokens · 0.447s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["K601","K843","Q734"]}
21 tokens · 0.591s
035 · total · wording 0 · same844be322f4c4782e0ec3e49dac78c7131267b36bf38b2f67c6ed5fbc731a4c62
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"X514","units":3,"price":9},{"id":"V435","units":14,"price":19},{"id":"G770","units":16,"price":13},{"id":"M687","units":8,"price":18},{"id":"M471","units":8,"price":9}]
Checked answer
{"total": 717}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.230s
036 · sort · wording 0 · gain5f8cda65ab39760e09c19773f2aa5c7cae6ade2e8bd35a5ea3dc3cd258dce0de
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"S198","priority":5},{"id":"C993","priority":3},{"id":"E344","priority":4},{"id":"F735","priority":7},{"id":"V862","priority":4}]
Checked answer
{"ids": ["F735", "S198"]}
Seed

wrong_schema_or_value

{"ids": ["C993", "E344", "S198", "V862"]}
29 tokens · 0.798s
Clean · 1 GPU

Correct

{"ids":["F735","S198"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["F735","S198"]}
16 tokens · 0.444s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["F735","E344"]}
16 tokens · 0.449s
037 · lookup · wording 1 · gainbd83c49a57c6ea89a47996cd85f13e3ab70410890b18d95ab2066cc4f01f595c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id H531 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"F841","city":"Lima","units":0},{"id":"O373","city":"Oslo","units":12},{"id":"D951","city":"Suva","units":8},{"id":"H531","city":"Kyoto","units":19},{"id":"O798","city":"Oslo","units":19}]
Checked answer
{"city": "Kyoto", "units": 19}
Seed

wrong_schema_or_value

{"city":"Lima","units":0}
11 tokens · 0.307s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":19}
13 tokens · 0.362s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":19}
13 tokens · 0.368s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":19}
13 tokens · 0.366s
038 · filter · wording 1 · gain3fffeed7d3a4e15059aca9995ff907476acfad882574485633080a756d373b1e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"U110","status":"closed"},{"id":"J154","status":"closed"},{"id":"G875","status":"ready"},{"id":"H389","status":"closed"},{"id":"A251","status":"closed"}]
Checked answer
{"ids": ["G875"]}
Seed

wrong_schema_or_value

{
  "ids": ["A251", "G875", "H389", "J154", "U110"]
}
38 tokens · 1.050s
Clean · 1 GPU

Correct

{"ids":["G875"]}
11 tokens · 0.306s
Damaged · 1 GPU

Correct

{"ids":["G875"]}
11 tokens · 0.313s
Clean · 2 GPUs

Correct

{"ids":["G875"]}
11 tokens · 0.311s
039 · total · wording 1 · samefc5f49d430ac0b25a4fd7e7e48615d7cd9d84070cf63c8e68f8e9aceb9411f6e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"F650","units":5,"price":1},{"id":"A114","units":4,"price":9},{"id":"S490","units":13,"price":11},{"id":"O982","units":15,"price":12},{"id":"P978","units":6,"price":14}]
Checked answer
{"total": 448}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.261s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
040 · sort · wording 1 · gain860b62157303438c22ef6859ddb07c3a1f3beb47482c67bc2746b039ba1d3ce2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"F329","priority":8},{"id":"C681","priority":3},{"id":"B293","priority":9},{"id":"C478","priority":5},{"id":"P172","priority":6}]
Checked answer
{"ids": ["B293", "F329"]}
Seed

invalid_json_object

[{"id":"C478","priority":5},{"id":"P172","priority":6}]
26 tokens · 0.723s
Clean · 1 GPU

Correct

{"ids":["B293","F329"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["B293","F329"]}
16 tokens · 0.453s
Clean · 2 GPUs

Correct

{"ids":["B293","F329"]}
16 tokens · 0.450s
041 · lookup · wording 2 · samed722bed201e7624b8117597527ac6fadb4154106bc48b54116ae3b383ec8745e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Y759","city":"Oslo","units":6},{"id":"X316","city":"Perth","units":14},{"id":"P655","city":"Suva","units":6},{"id":"K585","city":"Kyoto","units":19},{"id":"Q123","city":"Accra","units":11}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.284s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.286s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.284s
042 · filter · wording 2 · gain8fffd5845528f4ca6489f27583fbd4798960855a7eff5d2dc69548e8ef215ff5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"V697","status":"pending"},{"id":"R249","status":"ready"},{"id":"S407","status":"closed"},{"id":"B343","status":"ready"},{"id":"U466","status":"closed"}]
Checked answer
{"ids": ["B343", "R249"]}
Seed

invalid_json_object

[{"id":"B343","status":"ready"},{"id":"U466","status":"closed"},{"id":"R249","status":"ready"},{"id":"V697","status":"pending"},{"id":"S407","status":"closed"}]
63 tokens · 1.715s
Clean · 1 GPU

Correct

{"ids":["B343","R249"]}
16 tokens · 0.443s
Damaged · 1 GPU

Correct

{"ids":["B343","R249"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["B343","R249"]}
16 tokens · 0.449s
043 · total · wording 2 · same92b9c718f59307279736980bc55373874b93018796cdf6457955b5c58bc52c3d
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"G876","units":10,"price":8},{"id":"G796","units":18,"price":8},{"id":"D935","units":3,"price":14},{"id":"Y826","units":16,"price":16},{"id":"D798","units":14,"price":1}]
Checked answer
{"total": 536}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.226s
044 · sort · wording 2 · same6620e0e5341412329ac46f18fbbfc1d7a9c24d98bf4f7c002200cc35514e31da
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"X620","priority":2},{"id":"M609","priority":2},{"id":"D420","priority":9},{"id":"P508","priority":7},{"id":"H223","priority":9}]
Checked answer
{"ids": ["D420", "H223"]}
Seed

wrong_schema_or_value

{"ids": ["X620", "M609", "D420", "P508", "H223"]}
35 tokens · 0.957s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["H223","D420"]}
16 tokens · 0.448s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["H223","D420"]}
16 tokens · 0.449s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["H223","D420"]}
16 tokens · 0.446s
045 · lookup · wording 3 · same3f15c938ee6736cc691060e376b5437510b5598c1c277bd4ef5a81eed64506da
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=J682. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"O768","city":"Kyoto","units":14},{"id":"Q485","city":"Riga","units":18},{"id":"J682","city":"Kyoto","units":10},{"id":"J920","city":"Kyoto","units":11},{"id":"C288","city":"Perth","units":19}]
Checked answer
{"city": "Kyoto", "units": 10}
Seed

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.359s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.364s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.366s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.365s
046 · filter · wording 3 · gain75cb94bfa4a9e73d75aa590576277f74b6a8c66d29a95ee3fcc0e13940031727
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"V190","status":"pending"},{"id":"R298","status":"pending"},{"id":"Z107","status":"ready"},{"id":"S426","status":"pending"},{"id":"R496","status":"closed"}]
Checked answer
{"ids": ["Z107"]}
Seed

wrong_schema_or_value

{"ids": ["Z107", "S426", "V190", "R298", "R496"]}
35 tokens · 0.957s
Clean · 1 GPU

Correct

{"ids":["Z107"]}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"ids":["Z107"]}
11 tokens · 0.310s
Clean · 2 GPUs

Correct

{"ids":["Z107"]}
11 tokens · 0.308s
047 · total · wording 3 · same565b97997aa09bcf97cb42bf5963eb30104d3fa1449f62ba8d416bbf67752c23
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Q749","units":1,"price":3},{"id":"V106","units":14,"price":14},{"id":"G505","units":9,"price":11},{"id":"A278","units":3,"price":15},{"id":"P540","units":18,"price":7}]
Checked answer
{"total": 469}
Seed

wrong_schema_or_value

{
  "total": 56
}
11 tokens · 0.305s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":331}
8 tokens · 0.226s
048 · sort · wording 3 · gainc7ee3ef3e3097ff8986aa86dc3e4a0483c832409616d7ee740e0092f0c1ac658
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"M177","priority":7},{"id":"N720","priority":9},{"id":"M432","priority":7},{"id":"H859","priority":4},{"id":"T422","priority":1}]
Checked answer
{"ids": ["N720", "M177"]}
Seed

invalid_json_object

[{"id":"M177","priority":7},{"id":"N720","priority":9},{"id":"M432","priority":7},{"id":"H859","priority":4},{"id":"T422","priority":1}]
62 tokens · 1.685s
Clean · 1 GPU

Correct

{"ids":["N720","M177"]}
16 tokens · 0.446s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["N720","M432"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["N720","M177"]}
16 tokens · 0.447s
049 · lookup · wording 0 · same9e4fb57ed4ffc78c1427ea47292db7f096aabac8bcc51f7a49f7b3412d2d596e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record N578. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"N578","city":"Riga","units":18},{"id":"B358","city":"Oslo","units":2},{"id":"N486","city":"Lima","units":9},{"id":"E602","city":"Bern","units":16},{"id":"F353","city":"Riga","units":0}]
Checked answer
{"city": "Riga", "units": 18}
Seed

Correct

{
  "city": "Riga",
  "units": 18
}
19 tokens · 0.519s
Clean · 1 GPU

Correct

{"city":"Riga","units":18}
12 tokens · 0.338s
Damaged · 1 GPU

Correct

{"city":"Riga","units":18}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Riga","units":18}
12 tokens · 0.337s
050 · filter · wording 0 · gain7b2dd5d7dfd78a83639e8de270ab1ebdfa2615728b73ded0793ef0ea91a3e6f0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"V819","status":"ready"},{"id":"Y982","status":"ready"},{"id":"N711","status":"pending"},{"id":"A888","status":"ready"},{"id":"Q232","status":"closed"}]
Checked answer
{"ids": ["A888", "V819", "Y982"]}
Seed

wrong_schema_or_value

{"ids": ["A888", "V819", "Y982", "N711", "Q232"]}
35 tokens · 0.952s
Clean · 1 GPU

Correct

{"ids":["A888","V819","Y982"]}
21 tokens · 0.585s
Damaged · 1 GPU

Correct

{"ids":["A888","V819","Y982"]}
21 tokens · 0.591s
Clean · 2 GPUs

Correct

{"ids":["A888","V819","Y982"]}
21 tokens · 0.585s
051 · total · wording 0 · samea9bab383cfeefd0bb37cd662bc0d5b593e9425a9a61490b790c56bfacf491b33
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"R402","units":7,"price":1},{"id":"R270","units":7,"price":12},{"id":"R203","units":18,"price":15},{"id":"Y381","units":10,"price":16},{"id":"O813","units":16,"price":8}]
Checked answer
{"total": 649}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.227s
052 · sort · wording 0 · gaine038b0e08284d1b1c0d249709229f656ac258e65225d065aa637ef1c7b9dafb8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"R942","priority":7},{"id":"R322","priority":9},{"id":"I351","priority":9},{"id":"E801","priority":4},{"id":"N417","priority":8}]
Checked answer
{"ids": ["I351", "R322"]}
Seed

wrong_schema_or_value

{"ids": ["R322", "R942", "I351", "E801", "N417"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["I351","R322"]}
16 tokens · 0.446s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["R322","N417"]}
16 tokens · 0.453s
Clean · 2 GPUs

Correct

{"ids":["I351","R322"]}
16 tokens · 0.444s
053 · lookup · wording 1 · same1ad53c052559418db0670967b89b0af6c6b5aa1a8b053ab4b690bee163aa959e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Y337 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"Y337","city":"Perth","units":8},{"id":"Q613","city":"Accra","units":1},{"id":"K475","city":"Perth","units":6},{"id":"I837","city":"Riga","units":4},{"id":"S810","city":"Accra","units":1}]
Checked answer
{"city": "Perth", "units": 8}
Seed

Correct

{"city":"Perth","units":8}
11 tokens · 0.312s
Clean · 1 GPU

Correct

{"city":"Perth","units":8}
11 tokens · 0.311s
Damaged · 1 GPU

Correct

{"city":"Perth","units":8}
11 tokens · 0.313s
Clean · 2 GPUs

Correct

{"city":"Perth","units":8}
11 tokens · 0.311s
054 · filter · wording 1 · gain9824225f04a53e465f85127e2e9c12f9ba2019416a8853eaf1ee9c60950bda61
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M929","status":"pending"},{"id":"G773","status":"closed"},{"id":"Q818","status":"pending"},{"id":"V120","status":"pending"},{"id":"L778","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["M929", "G773", "Q818", "V120", "L778"]
}
38 tokens · 1.043s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.172s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.315s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.173s
055 · total · wording 1 · same2bad7042a89db8a9728ff64e2e42543e966dffc911c060689c8e9e1250d7b860
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"B989","units":5,"price":12},{"id":"D774","units":0,"price":8},{"id":"Q197","units":12,"price":15},{"id":"W721","units":14,"price":19},{"id":"G535","units":17,"price":17}]
Checked answer
{"total": 795}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":441}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":442}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":711}
8 tokens · 0.229s
056 · sort · wording 1 · samecfc99c35c13a54643f2b77992a20203e6412746bd232752fe38683dbb77f5132
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Y194","priority":1},{"id":"R553","priority":1},{"id":"H693","priority":8},{"id":"J171","priority":7},{"id":"M311","priority":8}]
Checked answer
{"ids": ["H693", "M311"]}
Seed

invalid_json_object

[{"id":"Y194","priority":1},{"id":"R553","priority":1},{"id":"H693","priority":8},{"id":"J171","priority":7},{"id":"M311","priority":8}]
62 tokens · 1.694s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["M311","H693"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["H693","M311"]}
16 tokens · 0.453s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["M311","H693"]}
16 tokens · 0.450s
057 · lookup · wording 2 · gain54e4978b0132d7eb9364d33c1836d1690e923dddc0527ecb21531fd05c6d999f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to G838? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"F960","city":"Riga","units":9},{"id":"D144","city":"Kyoto","units":0},{"id":"Q181","city":"Perth","units":0},{"id":"X937","city":"Riga","units":18},{"id":"G838","city":"Accra","units":4}]
Checked answer
{"city": "Accra", "units": 4}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":"Accra","units":4}
11 tokens · 0.312s
Damaged · 1 GPU

Correct

{"city":"Accra","units":4}
11 tokens · 0.312s
Clean · 2 GPUs

Correct

{"city":"Accra","units":4}
11 tokens · 0.311s
058 · filter · wording 2 · same5d77ebacbcac439fd1f8c2714e4d17cdc3b0ffcfceb30fd6fc4d9a5507e5453e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Z252","status":"pending"},{"id":"N503","status":"ready"},{"id":"P890","status":"ready"},{"id":"U179","status":"ready"},{"id":"U218","status":"closed"}]
Checked answer
{"ids": ["N503", "P890", "U179"]}
Seed

invalid_json_object

[{"id":"N503","status":"ready"},{"id":"P890","status":"ready"},{"id":"U179","status":"ready"},{"id":"U218","status":"closed"}]
51 tokens · 1.400s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["P890","N503","U179"]}
21 tokens · 0.588s
Damaged · 1 GPU

Correct

{"ids":["N503","P890","U179"]}
21 tokens · 0.592s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["P890","N503","U179"]}
21 tokens · 0.589s
059 · total · wording 2 · samecf4bb726f73bffd42094775cd8b57b4ce725da3801b1b1711ba1bd8fc16b3d81
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I517","units":3,"price":13},{"id":"I200","units":12,"price":11},{"id":"B294","units":6,"price":14},{"id":"O308","units":14,"price":3},{"id":"R482","units":19,"price":16}]
Checked answer
{"total": 601}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.229s
060 · sort · wording 2 · samef5c00720484f9560c16587be7778e7a632feaff37665a5d0307113624f552abd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"A991","priority":6},{"id":"Q537","priority":9},{"id":"I377","priority":6},{"id":"J979","priority":1},{"id":"X738","priority":1}]
Checked answer
{"ids": ["Q537", "A991"]}
Seed

wrong_schema_or_value

{"ids": ["A991", "Q537", "I377", "J979", "X738"]}
35 tokens · 0.960s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["Q537","I377"]}
16 tokens · 0.450s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Q537","I377"]}
16 tokens · 0.447s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["Q537","I377"]}
16 tokens · 0.450s
061 · lookup · wording 3 · gain8aa9bd80c4de5d0920f96765dd546ec024d2154d291ac27cbec00a9a01e78be5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P550","city":"Bern","units":10},{"id":"K284","city":"Kyoto","units":16},{"id":"L945","city":"Lima","units":12},{"id":"P678","city":"Bern","units":15},{"id":"L144","city":"Riga","units":17}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Bern","units":10}
11 tokens · 0.307s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.284s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.282s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.285s
062 · filter · wording 3 · gaine5f0c5ec31dcb189853271eed4635767eea5d6862accdbde15897ed1bb46e1cb
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Q221","status":"pending"},{"id":"F645","status":"ready"},{"id":"E463","status":"ready"},{"id":"P383","status":"pending"},{"id":"W967","status":"closed"}]
Checked answer
{"ids": ["E463", "F645"]}
Seed

wrong_schema_or_value

{"ids": ["F645", "E463", "P383", "W967", "Q221"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["E463","F645"]}
16 tokens · 0.454s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["F645","E463"]}
16 tokens · 0.451s
Clean · 2 GPUs

Correct

{"ids":["E463","F645"]}
16 tokens · 0.450s
063 · total · wording 3 · same099cdca496e4610d48ad7051644881634a2042b7e882b68f76bfff97fa9a7872
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"N337","units":11,"price":2},{"id":"Q640","units":16,"price":10},{"id":"M913","units":4,"price":3},{"id":"P231","units":9,"price":3},{"id":"E428","units":2,"price":14}]
Checked answer
{"total": 249}
Seed

wrong_schema_or_value

{
  "total": 10
}
11 tokens · 0.305s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":231}
8 tokens · 0.229s
064 · sort · wording 3 · same8e41a2fa3a602632574b42860ea0142a79874a9c4c1db7efc7cba308ef3f4ae3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"R147","priority":3},{"id":"Z835","priority":8},{"id":"D312","priority":6},{"id":"C400","priority":8},{"id":"E680","priority":8}]
Checked answer
{"ids": ["C400", "E680"]}
Seed

invalid_json_object

[{"id":"R147"},{"id":"Z835"},{"id":"D312"},{"id":"C400"},{"id":"E680"}]
43 tokens · 1.177s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["E680","C400"]}
16 tokens · 0.451s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["E680","C400"]}
16 tokens · 0.449s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["E680","C400"]}
16 tokens · 0.451s
065 · lookup · wording 0 · gainf9c0f67393fa027d86e412fb58f5500ac4064ea089bd0f407012c065be09ebaf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record H498. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"X347","city":"Lima","units":13},{"id":"H498","city":"Kyoto","units":10},{"id":"B681","city":"Oslo","units":13},{"id":"U595","city":"Suva","units":10},{"id":"U188","city":"Lima","units":2}]
Checked answer
{"city": "Kyoto", "units": 10}
Seed

wrong_schema_or_value

{
  "city": "Lima",
  "units": 13
}
19 tokens · 0.523s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.368s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.367s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":10}
13 tokens · 0.368s
066 · filter · wording 0 · samee943a7f664d5d969123e23f5981398bf7681dbe0b32867f2db52924def7fe82c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L476","status":"closed"},{"id":"S638","status":"closed"},{"id":"E125","status":"ready"},{"id":"H131","status":"ready"},{"id":"G659","status":"ready"}]
Checked answer
{"ids": ["E125", "G659", "H131"]}
Seed

wrong_schema_or_value

{"ids": ["E125", "G659", "H131", "L476", "S638"]}
35 tokens · 0.959s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["G659","H131","E125"]}
21 tokens · 0.590s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["E125","H131","G659"]}
21 tokens · 0.590s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["G659","E125","H131"]}
21 tokens · 0.590s
067 · total · wording 0 · same851007a15ffc307322fc64267a9c9809d56230fb84e19005f5b9d38e4ca54d82
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"D785","units":17,"price":9},{"id":"K902","units":4,"price":14},{"id":"Y706","units":17,"price":5},{"id":"F297","units":5,"price":5},{"id":"J197","units":2,"price":7}]
Checked answer
{"total": 333}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":371}
8 tokens · 0.229s
068 · sort · wording 0 · gainde08d683475a9e2c99d455895cc23d62e5fd581212984dc051090bb659201a01
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"K401","priority":7},{"id":"I934","priority":1},{"id":"H149","priority":4},{"id":"V754","priority":4},{"id":"W832","priority":5}]
Checked answer
{"ids": ["K401", "W832"]}
Seed

wrong_schema_or_value

{"ids": ["I934", "H149", "V754", "W832", "K401"]}
35 tokens · 0.962s
Clean · 1 GPU

Correct

{"ids":["K401","W832"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["K401","W832"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["K401","W832"]}
16 tokens · 0.450s
069 · lookup · wording 1 · same51015293c1a08c1a36a4bf53761b6b81a24363643039d6de5ecd8c0cfd77068c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id O816 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"O816","city":"Suva","units":8},{"id":"A580","city":"Riga","units":10},{"id":"Y937","city":"Accra","units":18},{"id":"D668","city":"Oslo","units":17},{"id":"O706","city":"Perth","units":8}]
Checked answer
{"city": "Suva", "units": 8}
Seed

Correct

{"city":"Suva","units":8}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Suva","units":8}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Suva","units":8}
11 tokens · 0.310s
Clean · 2 GPUs

Correct

{"city":"Suva","units":8}
11 tokens · 0.312s
070 · filter · wording 1 · gain165beff364e91b4b6518b8451b784a94491d5dcd69b0372f1229e0ab04cc5069
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"F769","status":"pending"},{"id":"B648","status":"closed"},{"id":"A871","status":"ready"},{"id":"W845","status":"ready"},{"id":"G188","status":"pending"}]
Checked answer
{"ids": ["A871", "W845"]}
Seed

wrong_schema_or_value

{
  "ids": ["A871", "B648", "G188", "F769", "W845"]
}
38 tokens · 1.039s
Clean · 1 GPU

Correct

{"ids":["A871","W845"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["A871","W845"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["A871","W845"]}
16 tokens · 0.449s
071 · total · wording 1 · same4af41df396b6dcf7e6303cee419cbcf14a9e0d419a2a344e7e8ba0a1da32dc9b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"S550","units":7,"price":19},{"id":"Z102","units":1,"price":6},{"id":"B913","units":4,"price":11},{"id":"A445","units":0,"price":7},{"id":"P695","units":14,"price":17}]
Checked answer
{"total": 421}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.226s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
072 · sort · wording 1 · gain9ea859aaafeeafd9551669fe7157dbf5061d2735fd24b1209e98740d6a6656c7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"F658","priority":2},{"id":"V837","priority":7},{"id":"H912","priority":4},{"id":"O716","priority":3},{"id":"K763","priority":7}]
Checked answer
{"ids": ["K763", "V837"]}
Seed

invalid_json_object

[{"id":"F658","priority":2},{"id":"V837","priority":7},{"id":"H912","priority":4},{"id":"O716","priority":3},{"id":"K763","priority":7}]
62 tokens · 1.689s
Clean · 1 GPU

Correct

{"ids":["K763","V837"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["K763","V837"]}
16 tokens · 0.451s
Clean · 2 GPUs

Correct

{"ids":["K763","V837"]}
16 tokens · 0.450s
073 · lookup · wording 2 · gain12197d0e9f3a5c78dc88ce73584b9acf66f62259e37e777abf2ba902b05a2c66
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z273? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"D371","city":"Riga","units":5},{"id":"U563","city":"Oslo","units":9},{"id":"U198","city":"Riga","units":11},{"id":"N654","city":"Accra","units":16},{"id":"Z273","city":"Kyoto","units":17}]
Checked answer
{"city": "Kyoto", "units": 17}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":17}
13 tokens · 0.365s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":17}
13 tokens · 0.366s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":17}
13 tokens · 0.363s
074 · filter · wording 2 · gain89b036122317d26d4f3fec7bfe4ed1937b3b7776bb7e66517512419031a62abb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"H813","status":"closed"},{"id":"M685","status":"closed"},{"id":"G343","status":"pending"},{"id":"U264","status":"pending"},{"id":"Z979","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"H813"},{"id":"M685"},{"id":"G343"},{"id":"U264"},{"id":"Z979"}]
43 tokens · 1.173s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.170s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.170s
075 · total · wording 2 · samee0e37d6d0cc56c1945dd3fd8f811c579c4f9e6884f9e1de1f963e865f18824d6
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"C832","units":3,"price":4},{"id":"Z446","units":17,"price":19},{"id":"I625","units":13,"price":8},{"id":"U310","units":3,"price":18},{"id":"D267","units":8,"price":2}]
Checked answer
{"total": 509}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.230s
Clean · 2 GPUs

wrong_schema_or_value

{"total":333}
8 tokens · 0.223s
076 · sort · wording 2 · gain562d1da0ef5d55ef33b724515f0f4a2a5103556fa8956376d918352f0ff55763
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"C895","priority":1},{"id":"X136","priority":8},{"id":"U502","priority":8},{"id":"C788","priority":2},{"id":"M546","priority":1}]
Checked answer
{"ids": ["U502", "X136"]}
Seed

wrong_schema_or_value

{"ids": ["C895", "X136", "U502", "C788", "M546"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["U502","X136"]}
16 tokens · 0.451s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["U502","C895"]}
16 tokens · 0.450s
Clean · 2 GPUs

Correct

{"ids":["U502","X136"]}
16 tokens · 0.456s
077 · lookup · wording 3 · same1654de92749ce0ab3ed6f199661119f56c7427bd4f3d5c75fcc659dd3210afe1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=V469. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"H602","city":"Oslo","units":9},{"id":"N412","city":"Perth","units":4},{"id":"V469","city":"Oslo","units":11},{"id":"X898","city":"Accra","units":12},{"id":"E510","city":"Accra","units":3}]
Checked answer
{"city": "Oslo", "units": 11}
Seed

Correct

{"city":"Oslo","units":11}
12 tokens · 0.335s
Clean · 1 GPU

Correct

{"city":"Oslo","units":11}
12 tokens · 0.342s
Damaged · 1 GPU

Correct

{"city":"Oslo","units":11}
12 tokens · 0.341s
Clean · 2 GPUs

Correct

{"city":"Oslo","units":11}
12 tokens · 0.347s
078 · filter · wording 3 · gain9976f33c9494f641ad7b6200fce8ac930a4ef8916615afcad25f73b5abc3408e
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Z718","status":"ready"},{"id":"W426","status":"closed"},{"id":"F515","status":"closed"},{"id":"Q836","status":"ready"},{"id":"W840","status":"closed"}]
Checked answer
{"ids": ["Q836", "Z718"]}
Seed

wrong_schema_or_value

{"ids": ["Z718", "W426", "F515", "Q836", "W840"]}
35 tokens · 0.962s
Clean · 1 GPU

Correct

{"ids":["Q836","Z718"]}
16 tokens · 0.450s
Damaged · 1 GPU

Correct

{"ids":["Q836","Z718"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["Q836","Z718"]}
16 tokens · 0.443s
079 · total · wording 3 · samecf09489a3de4af31ee8426bb68396ccc092de60fcefed034e885a40b459be3cf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"C209","units":5,"price":4},{"id":"O687","units":10,"price":2},{"id":"V388","units":7,"price":13},{"id":"K923","units":14,"price":13},{"id":"V603","units":12,"price":11}]
Checked answer
{"total": 445}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.333s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
080 · sort · wording 3 · gaine611a779b151a200dca49657b9c44f8dae2555bb1bb83b92ad5d7ef88874736a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"B221","priority":7},{"id":"X581","priority":4},{"id":"P914","priority":2},{"id":"J474","priority":7},{"id":"H316","priority":4}]
Checked answer
{"ids": ["B221", "J474"]}
Seed

invalid_json_object

[{"id":"B221","priority":7},{"id":"P914","priority":2}]
26 tokens · 0.712s
Clean · 1 GPU

Correct

{"ids":["B221","J474"]}
16 tokens · 0.450s
Damaged · 1 GPU

Correct

{"ids":["B221","J474"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["B221","J474"]}
16 tokens · 0.447s
081 · lookup · wording 0 · gain30c497f3eba159b2f765ad32c7504553ec5560158fe53c8a7ed73ef0d269cb6c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"G461","city":"Bern","units":5},{"id":"E606","city":"Riga","units":8},{"id":"M733","city":"Oslo","units":18},{"id":"N216","city":"Bern","units":12},{"id":"N326","city":"Suva","units":12}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Bern","units":5},{"city":"Riga","units":8},{"city":"Oslo","units":18},{"city":"Bern","units":12},{"city":"Suva","units":12}]
53 tokens · 1.455s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.284s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.282s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.281s
082 · filter · wording 0 · gain530f578bc329a850d6e967b69e7e73f2ab8697fb7be41d666d5a778b0b9e812c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"H832","status":"closed"},{"id":"N179","status":"pending"},{"id":"O981","status":"ready"},{"id":"O537","status":"ready"},{"id":"P783","status":"pending"}]
Checked answer
{"ids": ["O537", "O981"]}
Seed

wrong_schema_or_value

{"ids": ["O537", "O981", "P783"]}
23 tokens · 0.632s
Clean · 1 GPU

Correct

{"ids":["O537","O981"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["O537","O981"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["O537","O981"]}
16 tokens · 0.446s
083 · total · wording 0 · same3f265274e53016882e8034b000eaa13e77d0ea559e17e2d84115290884efd454
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"U790","units":2,"price":17},{"id":"A196","units":19,"price":5},{"id":"Y162","units":13,"price":2},{"id":"T483","units":14,"price":17},{"id":"B708","units":4,"price":12}]
Checked answer
{"total": 441}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.226s
Clean · 2 GPUs

wrong_schema_or_value

{"total":371}
8 tokens · 0.226s
084 · sort · wording 0 · same139ce2ee4753742b4cc04a330b909413566338c1939f64adf9eff4dd657c8252
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"R716","priority":7},{"id":"L769","priority":7},{"id":"J227","priority":9},{"id":"Q873","priority":1},{"id":"A348","priority":8}]
Checked answer
{"ids": ["J227", "A348"]}
Seed

wrong_schema_or_value

{"ids": ["R716", "L769", "J227", "Q873", "A348"]}
35 tokens · 0.960s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["J227","R716"]}
16 tokens · 0.444s
Damaged · 1 GPU

Correct

{"ids":["J227","A348"]}
16 tokens · 0.452s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["J227","R716"]}
16 tokens · 0.446s
085 · lookup · wording 1 · same0e8028c0209f771134ffc5b027b7afc7457ac3625965092e3aeddae6dd51eb0f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id A813 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"A813","city":"Lima","units":8},{"id":"T753","city":"Perth","units":15},{"id":"K809","city":"Bern","units":19},{"id":"P758","city":"Riga","units":9},{"id":"N678","city":"Oslo","units":2}]
Checked answer
{"city": "Lima", "units": 8}
Seed

Correct

{"city":"Lima","units":8}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Lima","units":8}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"city":"Lima","units":8}
11 tokens · 0.311s
Clean · 2 GPUs

Correct

{"city":"Lima","units":8}
11 tokens · 0.308s
086 · filter · wording 1 · gain32752cd5e24e6ed7ff439ddc06605f713de6b7093a6f39fa68fab0308d09d5d8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"A141","status":"closed"},{"id":"B778","status":"closed"},{"id":"B122","status":"ready"},{"id":"Q592","status":"ready"},{"id":"G612","status":"pending"}]
Checked answer
{"ids": ["B122", "Q592"]}
Seed

wrong_schema_or_value

{
  "ids": ["A141", "B122", "B778", "G612"]
}
32 tokens · 0.876s
Clean · 1 GPU

Correct

{"ids":["B122","Q592"]}
16 tokens · 0.445s
Damaged · 1 GPU

Correct

{"ids":["B122","Q592"]}
16 tokens · 0.450s
Clean · 2 GPUs

Correct

{"ids":["B122","Q592"]}
16 tokens · 0.446s
087 · total · wording 1 · same9caac3d01b96ab2c5ba4c5b2bbe6cf9b884535cbaa72e64946a3cd8f482ef5e4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"V582","units":19,"price":11},{"id":"K460","units":19,"price":9},{"id":"J105","units":17,"price":6},{"id":"M704","units":4,"price":4},{"id":"S865","units":13,"price":9}]
Checked answer
{"total": 615}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":441}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.229s
Clean · 2 GPUs

wrong_schema_or_value

{"total":731}
8 tokens · 0.226s
088 · sort · wording 1 · same80e19423471fd69dc75558cca79c99b82b51f90627bc78bef591dfcaa6a57d7b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"M543","priority":7},{"id":"X773","priority":2},{"id":"L424","priority":7},{"id":"P121","priority":7},{"id":"Q414","priority":7}]
Checked answer
{"ids": ["L424", "M543"]}
Seed

invalid_json_object

[{"id":"M543","priority":7},{"id":"L424","priority":7},{"id":"P121","priority":7},{"id":"Q414","priority":7}]
50 tokens · 1.367s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["P121","M543","Q414"]}
21 tokens · 0.585s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["M543","Q414"]}
16 tokens · 0.444s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["P121","M543","Q414"]}
21 tokens · 0.581s
089 · lookup · wording 2 · gain6339bd519d8349b3e1f9523b91cd474457f7a6ef3ca33b009369f8ae0c1865c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M793? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"A622","city":"Bern","units":2},{"id":"M793","city":"Kyoto","units":8},{"id":"K226","city":"Bern","units":13},{"id":"J736","city":"Oslo","units":8},{"id":"V652","city":"Accra","units":15}]
Checked answer
{"city": "Kyoto", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.339s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":8}
12 tokens · 0.336s
090 · filter · wording 2 · same8102fcb9c4af46f8e4b9f28453db20aca11693a3161c3086156d8409c03406dd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"H523","status":"ready"},{"id":"A488","status":"ready"},{"id":"I235","status":"ready"},{"id":"L312","status":"ready"},{"id":"X625","status":"closed"}]
Checked answer
{"ids": ["A488", "H523", "I235", "L312"]}
Seed

invalid_json_object

[{"id":"A488","status":"ready"},{"id":"H523","status":"ready"},{"id":"I235","status":"ready"},{"id":"L312","status":"ready"},{"id":"X625","status":"closed"}]
63 tokens · 1.717s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["A488","I235","H523","L312"]}
26 tokens · 0.725s
Damaged · 1 GPU

Correct

{"ids":["A488","H523","I235","L312"]}
26 tokens · 0.730s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["A488","I235","H523","L312"]}
26 tokens · 0.724s
091 · total · wording 2 · same0db4cfb98d7938eca9acd3b0ddb69f64bb1855e1e935b17e0ad7895fdf4542a8
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y640","units":6,"price":5},{"id":"K843","units":3,"price":14},{"id":"L845","units":13,"price":14},{"id":"Z980","units":7,"price":14},{"id":"M431","units":3,"price":9}]
Checked answer
{"total": 379}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":232}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":333}
8 tokens · 0.227s
092 · sort · wording 2 · gainf80b86926a533bd3b754f26520f89337462bc62668d24cc94f1aaf9eff56402f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"L615","priority":6},{"id":"X685","priority":7},{"id":"F142","priority":7},{"id":"I751","priority":1},{"id":"R564","priority":4}]
Checked answer
{"ids": ["F142", "X685"]}
Seed

wrong_schema_or_value

{"ids": ["L615", "X685", "F142", "I751", "R564"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["F142","X685"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["F142","X685"]}
16 tokens · 0.450s
Clean · 2 GPUs

Correct

{"ids":["F142","X685"]}
16 tokens · 0.439s
093 · lookup · wording 3 · same9b453eb3dbc2686c1625e95cd99c40d88a88c712ed78ea9c282a8bc575c283e1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=P806. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E114","city":"Perth","units":13},{"id":"P806","city":"Lima","units":14},{"id":"S991","city":"Kyoto","units":5},{"id":"I430","city":"Riga","units":11},{"id":"Y836","city":"Suva","units":0}]
Checked answer
{"city": "Lima", "units": 14}
Seed

Correct

{"city":"Lima","units":14}
12 tokens · 0.332s
Clean · 1 GPU

Correct

{"city":"Lima","units":14}
12 tokens · 0.339s
Damaged · 1 GPU

Correct

{"city":"Lima","units":14}
12 tokens · 0.339s
Clean · 2 GPUs

Correct

{"city":"Lima","units":14}
12 tokens · 0.329s
094 · filter · wording 3 · gainc4239a81f25b89f2b7687d9e2e1ce5e9e313e541f34b5d8cd042f4b944a95fb5
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"I796","status":"ready"},{"id":"H706","status":"pending"},{"id":"X747","status":"pending"},{"id":"I420","status":"closed"},{"id":"H694","status":"closed"}]
Checked answer
{"ids": ["I796"]}
Seed

wrong_schema_or_value

{"ids": ["I796", "H706", "X747", "I420", "H694"]}
35 tokens · 0.961s
Clean · 1 GPU

Correct

{"ids":["I796"]}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"ids":["I796"]}
11 tokens · 0.312s
Clean · 2 GPUs

Correct

{"ids":["I796"]}
11 tokens · 0.303s
095 · total · wording 3 · same1674ef7dd664420b4fcc8372a0c19ba2422cc3b263f34f00253cdb1fc8a598ac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"N884","units":9,"price":12},{"id":"P317","units":15,"price":6},{"id":"A505","units":7,"price":3},{"id":"F465","units":10,"price":7},{"id":"H976","units":12,"price":12}]
Checked answer
{"total": 433}
Seed

invalid_json_object

{
  "total": 12 + 6 + 3 + 7 + 12 = 40
}
28 tokens · 0.773s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":311}
8 tokens · 0.223s
096 · sort · wording 3 · gain3ddee79734212122b3cc420c3f2160a3a1e06836dae1f8bf409ed1c2e9f3ad2e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"T141","priority":9},{"id":"N308","priority":5},{"id":"Z282","priority":2},{"id":"Q573","priority":7},{"id":"W649","priority":9}]
Checked answer
{"ids": ["T141", "W649"]}
Seed

invalid_json_object

[{"id":"T141","priority":9},{"id":"W649","priority":9}]
26 tokens · 0.716s
Clean · 1 GPU

Correct

{"ids":["T141","W649"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["T141","W649"]}
16 tokens · 0.444s
Clean · 2 GPUs

Correct

{"ids":["T141","W649"]}
16 tokens · 0.446s
097 · lookup · wording 0 · same809d5d167996e64aadae8b09527b40bf80867105803fa46e509e1af8be24f67a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Y925. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W710","city":"Bern","units":17},{"id":"V111","city":"Kyoto","units":7},{"id":"Z162","city":"Kyoto","units":9},{"id":"Y925","city":"Kyoto","units":15},{"id":"K799","city":"Bern","units":19}]
Checked answer
{"city": "Kyoto", "units": 15}
Seed

Correct

{
  "city": "Kyoto",
  "units": 15
}
20 tokens · 0.553s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.368s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.363s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.362s
098 · filter · wording 0 · gain22e8a394c04ca914c8388325c10b13b8557faaef867a7ec6d6bfe7fe29af38cd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"G267","status":"closed"},{"id":"M889","status":"closed"},{"id":"P892","status":"ready"},{"id":"T277","status":"closed"},{"id":"D467","status":"closed"}]
Checked answer
{"ids": ["P892"]}
Seed

wrong_schema_or_value

{"ids": ["P892", "T277", "D467", "G267", "M889"]}
35 tokens · 0.962s
Clean · 1 GPU

Correct

{"ids":["P892"]}
11 tokens · 0.312s
Damaged · 1 GPU

Correct

{"ids":["P892"]}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"ids":["P892"]}
11 tokens · 0.308s
099 · total · wording 0 · same4507d9d59cd35bd251c48a95a7dc14f1bac4ac3c819c6104b4fe605a771b8a80
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C159","units":7,"price":18},{"id":"C601","units":16,"price":13},{"id":"W752","units":9,"price":18},{"id":"I959","units":4,"price":4},{"id":"G723","units":2,"price":19}]
Checked answer
{"total": 550}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.226s
100 · sort · wording 0 · gain5138e66a0f7951f0203d6230b4db870da3e4ff025210b67e6127af867fab95db
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E626","priority":5},{"id":"B601","priority":8},{"id":"E494","priority":9},{"id":"F296","priority":2},{"id":"T368","priority":8}]
Checked answer
{"ids": ["E494", "B601"]}
Seed

wrong_schema_or_value

{"ids": ["E626", "B601", "E494", "F296", "T368"]}
35 tokens · 0.958s
Clean · 1 GPU

Correct

{"ids":["E494","B601"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["E494","B601"]}
16 tokens · 0.444s
Clean · 2 GPUs

Correct

{"ids":["E494","B601"]}
16 tokens · 0.446s
101 · lookup · wording 1 · same33549aaddae88eb29c3f7ef5368451cf0160f04c0e98e1e40fc792fb3751ff32
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D482","city":"Accra","units":6},{"id":"W234","city":"Lima","units":18},{"id":"Y911","city":"Riga","units":10},{"id":"Z586","city":"Kyoto","units":17},{"id":"W375","city":"Lima","units":14}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Accra","units":6}
11 tokens · 0.306s
Clean · 1 GPU

wrong_schema_or_value

{"city":"Kyoto","units":17}
13 tokens · 0.364s
Damaged · 1 GPU

wrong_schema_or_value

{"city":"Kyoto","units":17}
13 tokens · 0.365s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.282s
102 · filter · wording 1 · gaine5375a8b5b6d75cd9b4d0c7f9c3fcb253a3ae9c66142e1e85051b03eabfa8a0b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"R757","status":"pending"},{"id":"S792","status":"pending"},{"id":"A260","status":"closed"},{"id":"F451","status":"pending"},{"id":"I366","status":"ready"}]
Checked answer
{"ids": ["I366"]}
Seed

wrong_schema_or_value

{
  "ids": ["A260", "F451", "I366"]
}
26 tokens · 0.716s
Clean · 1 GPU

Correct

{"ids":["I366"]}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"ids":["I366"]}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"ids":["I366"]}
11 tokens · 0.307s
103 · total · wording 1 · same7c68ef371e43d6adff266241ba265c7ec095a54f10228cf76e34dd3b37396893
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"Z626","units":12,"price":6},{"id":"C153","units":10,"price":5},{"id":"L912","units":5,"price":10},{"id":"G980","units":3,"price":12},{"id":"L268","units":2,"price":17}]
Checked answer
{"total": 242}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":311}
8 tokens · 0.225s
104 · sort · wording 1 · gainba52309f04bddd1a5ecb61131f3e4bb138fbda4b0441e32242138ad3b822d2a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V414","priority":6},{"id":"E294","priority":6},{"id":"N965","priority":3},{"id":"Q893","priority":2},{"id":"V330","priority":1}]
Checked answer
{"ids": ["E294", "V414"]}
Seed

invalid_json_object

[{"id":"V414","priority":6},{"id":"E294","priority":6},{"id":"N965","priority":3},{"id":"Q893","priority":2},{"id":"V330","priority":1}]
62 tokens · 1.698s
Clean · 1 GPU

Correct

{"ids":["E294","V414"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["E294","V414"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["E294","V414"]}
16 tokens · 0.444s
105 · lookup · wording 2 · gainc13c3ccdef17bbe6bb9f3fd3c86c18458a93775a8a171d433141840b55c8c272
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to F933? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"L870","city":"Perth","units":5},{"id":"T792","city":"Suva","units":3},{"id":"F933","city":"Riga","units":9},{"id":"D151","city":"Riga","units":10},{"id":"I197","city":"Oslo","units":11}]
Checked answer
{"city": "Riga", "units": 9}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":"Riga","units":9}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Riga","units":9}
11 tokens · 0.310s
Clean · 2 GPUs

Correct

{"city":"Riga","units":9}
11 tokens · 0.309s
106 · filter · wording 2 · gain11dc36eb23241849368c41cbff8d2f93a131f851f0b4ad427dd5e32cdfd03855
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"A825","status":"pending"},{"id":"H420","status":"closed"},{"id":"T814","status":"pending"},{"id":"X170","status":"closed"},{"id":"S462","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"A825"},{"id":"H420"},{"id":"T814"},{"id":"X170"},{"id":"S462"}]
43 tokens · 1.173s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.172s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.179s
107 · total · wording 2 · same1ffb3f753b56de427dec4419215aa00642d6c5ed8ea3a4f64b2f1ebb6b17108b
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I817","units":19,"price":2},{"id":"Y912","units":2,"price":15},{"id":"O886","units":9,"price":16},{"id":"Q380","units":18,"price":12},{"id":"R125","units":5,"price":13}]
Checked answer
{"total": 493}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.252s
108 · sort · wording 2 · gainfeea9604c09f9bf0b94a6c77c7d2db995c37359847c6fefa2c28d71fc7e606ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S176","priority":9},{"id":"Y906","priority":6},{"id":"I397","priority":1},{"id":"Y583","priority":7},{"id":"U219","priority":8}]
Checked answer
{"ids": ["S176", "U219"]}
Seed

wrong_schema_or_value

{"ids": ["Y583", "Y906", "I397", "S176", "U219"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["S176","U219"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["S176","U219"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["S176","U219"]}
16 tokens · 0.464s
109 · lookup · wording 3 · gain37cddae74153a976021a79c99ea5959bbe35f87bef908660e52ec563dc35bfd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=C828. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T209","city":"Suva","units":15},{"id":"C828","city":"Riga","units":0},{"id":"I187","city":"Suva","units":10},{"id":"A627","city":"Lima","units":2},{"id":"U267","city":"Suva","units":4}]
Checked answer
{"city": "Riga", "units": 0}
Seed

wrong_schema_or_value

{"city": "Suva", "units": 15}
15 tokens · 0.413s
Clean · 1 GPU

Correct

{"city":"Riga","units":0}
11 tokens · 0.311s
Damaged · 1 GPU

wrong_schema_or_value

{"city":null,"units":null}
10 tokens · 0.279s
Clean · 2 GPUs

Correct

{"city":"Riga","units":0}
11 tokens · 0.311s
110 · filter · wording 3 · gained7460071a1f7859e7ba3ef536f4f6d417165c8e2e73dbd3fa98ee2a2dfce3c1
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Z212","status":"pending"},{"id":"P469","status":"pending"},{"id":"E459","status":"pending"},{"id":"D991","status":"closed"},{"id":"B265","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["B265", "D991", "E459", "P469", "Z212"]}
35 tokens · 0.953s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.173s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D991"]}
11 tokens · 0.310s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.173s
111 · total · wording 3 · samede1f65195ac75e6c75daf1be86f3c43d7824b90f2837a9bb406b9f31f2babd61
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"L534","units":3,"price":14},{"id":"K768","units":1,"price":14},{"id":"T471","units":14,"price":6},{"id":"M129","units":10,"price":7},{"id":"D197","units":6,"price":9}]
Checked answer
{"total": 264}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.332s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":232}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":241}
8 tokens · 0.228s
112 · sort · wording 3 · gain33fcbd6ec4b53d8c8bf8ef3f37d11034404591fa7e1ed1368336c00b3b4e966e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"D847","priority":7},{"id":"Y836","priority":4},{"id":"L699","priority":8},{"id":"P652","priority":8},{"id":"K182","priority":7}]
Checked answer
{"ids": ["L699", "P652"]}
Seed

invalid_json_object

[{"id":"D847","priority":7},{"id":"Y836","priority":4},{"id":"L699","priority":8},{"id":"P652","priority":8},{"id":"K182","priority":7}]
62 tokens · 1.689s
Clean · 1 GPU

Correct

{"ids":["L699","P652"]}
16 tokens · 0.448s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D847","L699"]}
16 tokens · 0.443s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["P652","L699"]}
16 tokens · 0.445s
113 · lookup · wording 0 · gainbe37b26a59e9b4ed663d9e6bdc18967ed6dc860971fd6b6a99f2ee36ef0d186f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record V488. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Z796","city":"Riga","units":6},{"id":"X289","city":"Kyoto","units":2},{"id":"T687","city":"Riga","units":2},{"id":"B320","city":"Suva","units":16},{"id":"V488","city":"Kyoto","units":15}]
Checked answer
{"city": "Kyoto", "units": 15}
Seed

wrong_schema_or_value

{
  "city": "Riga",
  "units": 6
}
18 tokens · 0.498s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.366s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.365s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.362s
114 · filter · wording 0 · gaind3bca1f484d3fa67a9905e4558e718684ea05897afd8ec0799d289f410d99b1b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"X422","status":"closed"},{"id":"W182","status":"pending"},{"id":"J757","status":"closed"},{"id":"I108","status":"closed"},{"id":"G265","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["I108", "G265", "J757", "W182", "X422"]}
35 tokens · 0.962s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.172s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.306s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.172s
115 · total · wording 0 · sameab89422fdc294cc290a6c44ab249ffe655357e5d24ed03012268df5024f9855d
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"G994","units":13,"price":14},{"id":"X676","units":17,"price":19},{"id":"S331","units":9,"price":3},{"id":"O614","units":15,"price":4},{"id":"G228","units":12,"price":9}]
Checked answer
{"total": 700}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.227s
116 · sort · wording 0 · same44cc42671be6e521094ed5111ff8257d9b162026f701a315643e7dad992e2296
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z831","priority":9},{"id":"X484","priority":4},{"id":"T414","priority":7},{"id":"P115","priority":9},{"id":"Z183","priority":9}]
Checked answer
{"ids": ["P115", "Z183"]}
Seed

wrong_schema_or_value

{"ids": ["X484", "Z831", "T414", "P115", "Z183"]}
35 tokens · 0.957s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["Z183","P115"]}
16 tokens · 0.450s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z831","Z183"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["P115","Z183"]}
16 tokens · 0.447s
117 · lookup · wording 1 · gainc62aac8a00cc91b884a72f31577f6fa2a27e81d10ae96a7307a3f216726bdaeb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id E898 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M638","city":"Oslo","units":1},{"id":"E742","city":"Suva","units":6},{"id":"M800","city":"Riga","units":11},{"id":"R542","city":"Suva","units":18},{"id":"E898","city":"Kyoto","units":15}]
Checked answer
{"city": "Kyoto", "units": 15}
Seed

wrong_schema_or_value

{"city":"Oslo","units":1}
11 tokens · 0.305s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.361s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.362s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":15}
13 tokens · 0.366s
118 · filter · wording 1 · gainad60e3db08a8136ddbc257d74e40b1940b29a8e9226882d204d10d09bf3f4d4a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T474","status":"closed"},{"id":"U534","status":"closed"},{"id":"I284","status":"ready"},{"id":"H883","status":"ready"},{"id":"D153","status":"closed"}]
Checked answer
{"ids": ["H883", "I284"]}
Seed

wrong_schema_or_value

{
  "ids": ["I284", "H883", "U534", "T474", "D153"]
}
38 tokens · 1.039s
Clean · 1 GPU

Correct

{"ids":["H883","I284"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["H883","I284"]}
16 tokens · 0.442s
Clean · 2 GPUs

Correct

{"ids":["H883","I284"]}
16 tokens · 0.448s
119 · total · wording 1 · same7614e95c472b3d9c415b0759c0706c4d3573129ccbfbe292aaac0dfa032a1bc1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"P425","units":9,"price":19},{"id":"B231","units":14,"price":6},{"id":"X156","units":15,"price":3},{"id":"P257","units":4,"price":17},{"id":"Y418","units":15,"price":13}]
Checked answer
{"total": 563}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.224s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
120 · sort · wording 1 · gaine096873c24910aff8a6f6d166f3ecdb5d51e19ebad1d150443774b158d913162
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"O691","priority":6},{"id":"S185","priority":1},{"id":"W678","priority":5},{"id":"T329","priority":2},{"id":"M123","priority":3}]
Checked answer
{"ids": ["O691", "W678"]}
Seed

invalid_json_object

[{"id":"O691","priority":6},{"id":"S185","priority":1},{"id":"W678","priority":5},{"id":"T329","priority":2},{"id":"M123","priority":3}]
62 tokens · 1.696s
Clean · 1 GPU

Correct

{"ids":["O691","W678"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["O691","W678"]}
16 tokens · 0.450s
Clean · 2 GPUs

Correct

{"ids":["O691","W678"]}
16 tokens · 0.448s
121 · lookup · wording 2 · samef895ee83d16d57144ba3160ab77d6079a74d88b08f088b01b27d799b2b3109b9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"G237","city":"Perth","units":13},{"id":"H795","city":"Riga","units":5},{"id":"N566","city":"Lima","units":13},{"id":"E450","city":"Bern","units":5},{"id":"C889","city":"Accra","units":5}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.283s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.280s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.283s
122 · filter · wording 2 · gain0b2a2a844cffd728e078ee33e19f4e0f0864647f09b4592226b5b700fa536119
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"V930","status":"ready"},{"id":"V671","status":"ready"},{"id":"K382","status":"closed"},{"id":"M290","status":"ready"},{"id":"F693","status":"pending"}]
Checked answer
{"ids": ["M290", "V671", "V930"]}
Seed

invalid_json_object

[{"id":"V930","status":"ready"},{"id":"V671","status":"ready"},{"id":"K382","status":"closed"},{"id":"M290","status":"ready"},{"id":"F693","status":"pending"}]
63 tokens · 1.726s
Clean · 1 GPU

Correct

{"ids":["M290","V671","V930"]}
21 tokens · 0.586s
Damaged · 1 GPU

Correct

{"ids":["M290","V671","V930"]}
21 tokens · 0.588s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["M290","V930","V671"]}
21 tokens · 0.587s
123 · total · wording 2 · same31bb4b8d8c5de4766855e61b0293032bce80c51552151baf71c3e59c93e808d4
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"O513","units":15,"price":4},{"id":"U258","units":9,"price":7},{"id":"W746","units":19,"price":16},{"id":"E323","units":17,"price":19},{"id":"M209","units":11,"price":3}]
Checked answer
{"total": 783}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":471}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":531}
8 tokens · 0.228s
124 · sort · wording 2 · gainf0cc115a54032f31cc28b8cd859ec15dc02320a8ea01a6e32b33fd3d49c2b6da
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"I608","priority":3},{"id":"D657","priority":7},{"id":"V322","priority":7},{"id":"W169","priority":7},{"id":"I211","priority":2}]
Checked answer
{"ids": ["D657", "V322"]}
Seed

wrong_schema_or_value

{"ids": ["I608", "V322", "I211", "D657", "W169", "I608"]}
41 tokens · 1.119s
Clean · 1 GPU

Correct

{"ids":["D657","V322"]}
16 tokens · 0.446s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D657","W169"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["D657","V322"]}
16 tokens · 0.448s
125 · lookup · wording 3 · gain813ced39b6e161948fcd3a5d37c5b9746ed0ab53a7d765c179a132f3079d0b19
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=V991. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"A564","city":"Kyoto","units":5},{"id":"U112","city":"Riga","units":13},{"id":"D436","city":"Bern","units":19},{"id":"V991","city":"Lima","units":2},{"id":"I635","city":"Kyoto","units":3}]
Checked answer
{"city": "Lima", "units": 2}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":5}
12 tokens · 0.335s
Clean · 1 GPU

Correct

{"city":"Lima","units":2}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Lima","units":2}
11 tokens · 0.306s
Clean · 2 GPUs

Correct

{"city":"Lima","units":2}
11 tokens · 0.310s
126 · filter · wording 3 · gain84f3b5a7f7b7a4eace5393ea6db87ef7756b085fff99d58d95ff387c2ccd3be3
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Y680","status":"closed"},{"id":"N522","status":"pending"},{"id":"M628","status":"closed"},{"id":"Y130","status":"pending"},{"id":"I473","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["M628", "Y680", "Y130", "N522", "I473"]}
35 tokens · 0.961s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.170s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.172s
127 · total · wording 3 · samea730a3d3ca12dce2ea64a654b4dbf2908c822b68d596b5dbe30b082cb23eba20
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"D498","units":19,"price":7},{"id":"O779","units":11,"price":17},{"id":"W260","units":18,"price":4},{"id":"O901","units":17,"price":2},{"id":"W744","units":9,"price":1}]
Checked answer
{"total": 435}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.333s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.226s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.228s
128 · sort · wording 3 · same52c7a957256feb6f837a911253481b45e50292a498e6e19887ebd1c11ea65752
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"C907","priority":8},{"id":"C254","priority":3},{"id":"I259","priority":2},{"id":"Z806","priority":2},{"id":"G483","priority":4}]
Checked answer
{"ids": ["C907", "G483"]}
Seed

invalid_json_object

[{"id":"C907","priority":8},{"id":"C254","priority":3},{"id":"I259","priority":2},{"id":"Z806","priority":2},{"id":"G483","priority":4}]
62 tokens · 1.696s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["C907","C254"]}
16 tokens · 0.445s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["C907","C254"]}
16 tokens · 0.446s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["C907","C254"]}
16 tokens · 0.447s
129 · lookup · wording 0 · gain8a6c212bb002623d5f51919cfaec3d74bcc4ccd895c2af95a8092569fffa572e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record E679. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"P182","city":"Bern","units":16},{"id":"G476","city":"Riga","units":1},{"id":"O407","city":"Bern","units":2},{"id":"E679","city":"Bern","units":14},{"id":"I647","city":"Kyoto","units":8}]
Checked answer
{"city": "Bern", "units": 14}
Seed

wrong_schema_or_value

{
  "city": "Bern",
  "units": 16
}
18 tokens · 0.497s
Clean · 1 GPU

Correct

{"city":"Bern","units":14}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Bern","units":14}
11 tokens · 0.313s
Clean · 2 GPUs

Correct

{"city":"Bern","units":14}
11 tokens · 0.307s
130 · filter · wording 0 · gainccff2a96c3be80b41ec20386b0d053903f91de596bf173a8cbd8a6174250c0de
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L848","status":"ready"},{"id":"L472","status":"closed"},{"id":"P127","status":"pending"},{"id":"Y674","status":"pending"},{"id":"T191","status":"ready"}]
Checked answer
{"ids": ["L848", "T191"]}
Seed

wrong_schema_or_value

{"ids": ["L472", "L848", "P127", "T191", "Y674"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["L848","T191"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["L848","T191"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["L848","T191"]}
16 tokens · 0.446s
131 · total · wording 0 · same642314bf5f356b42aade9e7ebf260cae0e8c923a58ea02687afd0e96ed50f735
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"B702","units":11,"price":14},{"id":"C617","units":13,"price":4},{"id":"W964","units":18,"price":5},{"id":"Y164","units":19,"price":11},{"id":"K530","units":8,"price":5}]
Checked answer
{"total": 545}
Seed

invalid_json_object

{"total": 11 * 14 + 13 * 4 + 18 * 5 + 19 * 11 + 8 * 5}
40 tokens · 1.094s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.228s
132 · sort · wording 0 · gain31ad744d33daaf6366819726bc93c6f797b47305f998419c094d2459b653628c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"C328","priority":1},{"id":"P886","priority":6},{"id":"Y951","priority":4},{"id":"H191","priority":8},{"id":"T605","priority":1}]
Checked answer
{"ids": ["H191", "P886"]}
Seed

wrong_schema_or_value

{"ids": ["C328", "P886", "Y951", "H191", "T605"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["H191","P886"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["H191","P886"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["H191","P886"]}
16 tokens · 0.448s
133 · lookup · wording 1 · same53bf96a517b1e03a97ba876828194f3d11e37dc36f6dbc3d436665429c268c2c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id J510 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"J510","city":"Oslo","units":6},{"id":"V992","city":"Accra","units":7},{"id":"L267","city":"Kyoto","units":10},{"id":"K208","city":"Riga","units":6},{"id":"F652","city":"Suva","units":14}]
Checked answer
{"city": "Oslo", "units": 6}
Seed

Correct

{"city":"Oslo","units":6}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Oslo","units":6}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"city":"Oslo","units":6}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"city":"Oslo","units":6}
11 tokens · 0.308s
134 · filter · wording 1 · gain0b97dd8760f2bee3a270cb8e4c0cc7ecd23cea085ef81331c3d66587aec26726
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"H596","status":"closed"},{"id":"G222","status":"pending"},{"id":"Z134","status":"closed"},{"id":"X850","status":"ready"},{"id":"D388","status":"ready"}]
Checked answer
{"ids": ["D388", "X850"]}
Seed

wrong_schema_or_value

{
  "ids": ["X850", "D388", "H596", "G222", "Z134"]
}
38 tokens · 1.037s
Clean · 1 GPU

Correct

{"ids":["D388","X850"]}
16 tokens · 0.441s
Damaged · 1 GPU

Correct

{"ids":["D388","X850"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["D388","X850"]}
16 tokens · 0.445s
135 · total · wording 1 · samefc8b544207f60e809d5d9d352f33d73237b867624f3d6a2ca4eb2953c89408d2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"N577","units":11,"price":6},{"id":"L703","units":15,"price":5},{"id":"X678","units":11,"price":6},{"id":"U939","units":0,"price":2},{"id":"R349","units":3,"price":5}]
Checked answer
{"total": 222}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":232}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":331}
8 tokens · 0.227s
136 · sort · wording 1 · gaind7b872139e314f20efd903747be57de76dc74d119ab2937ac6f3cc94999413e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"B225","priority":9},{"id":"L710","priority":4},{"id":"Q308","priority":8},{"id":"Z890","priority":4},{"id":"K818","priority":7}]
Checked answer
{"ids": ["B225", "Q308"]}
Seed

invalid_json_object

[{"id":"B225","priority":9},{"id":"L710","priority":4},{"id":"Q308","priority":8},{"id":"Z890","priority":4},{"id":"K818","priority":7}]
62 tokens · 1.699s
Clean · 1 GPU

Correct

{"ids":["B225","Q308"]}
16 tokens · 0.444s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["B225","K818"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["B225","Q308"]}
16 tokens · 0.447s
137 · lookup · wording 2 · gain1cea95ab1253997aaf8286011172ce48c66613d1c003986defd3f9ace017a899
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to F599? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Q466","city":"Oslo","units":4},{"id":"Z307","city":"Accra","units":6},{"id":"V864","city":"Accra","units":9},{"id":"C323","city":"Perth","units":17},{"id":"F599","city":"Bern","units":10}]
Checked answer
{"city": "Bern", "units": 10}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":"Bern","units":10}
11 tokens · 0.307s
Damaged · 1 GPU

Correct

{"city":"Bern","units":10}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"city":"Bern","units":10}
11 tokens · 0.324s
138 · filter · wording 2 · gain32f7a888ac35e4274464543ba3c28718f087812edbd38986fa5cdf7d6d04f488
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"J354","status":"ready"},{"id":"K529","status":"pending"},{"id":"P265","status":"closed"},{"id":"L396","status":"closed"},{"id":"U461","status":"pending"}]
Checked answer
{"ids": ["J354"]}
Seed

invalid_json_object

[{"id":"J354"},{"id":"K529"},{"id":"P265"},{"id":"L396"},{"id":"U461"}]
43 tokens · 1.176s
Clean · 1 GPU

Correct

{"ids":["J354"]}
11 tokens · 0.308s
Damaged · 1 GPU

Correct

{"ids":["J354"]}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"ids":["J354"]}
11 tokens · 0.313s
139 · total · wording 2 · sameccb5ece90bfacb5096127465718bcf8553f94a4016bb5f679eb377ba8d964fbf
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V607","units":0,"price":6},{"id":"W219","units":11,"price":5},{"id":"J430","units":18,"price":9},{"id":"Z693","units":7,"price":18},{"id":"B728","units":6,"price":2}]
Checked answer
{"total": 355}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":232}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.229s
140 · sort · wording 2 · gain87609a2cafc0f0231ecee6ea39c0db6effdcbacb485f500b40dc30fcc4d40538
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"N748","priority":2},{"id":"I457","priority":7},{"id":"O652","priority":3},{"id":"M306","priority":8},{"id":"M164","priority":1}]
Checked answer
{"ids": ["M306", "I457"]}
Seed

wrong_schema_or_value

{"ids": ["M306", "M164", "N748", "O652", "I457", "M306"]}
41 tokens · 1.118s
Clean · 1 GPU

Correct

{"ids":["M306","I457"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["M306","I457"]}
16 tokens · 0.441s
Clean · 2 GPUs

Correct

{"ids":["M306","I457"]}
16 tokens · 0.449s
141 · lookup · wording 3 · gain856423e315aa07ef02fab0378375b4374d46ff6b480c8fb8249f53d2af20c703
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"B704","city":"Riga","units":0},{"id":"Z171","city":"Kyoto","units":19},{"id":"G606","city":"Lima","units":9},{"id":"H711","city":"Suva","units":16},{"id":"U828","city":"Bern","units":19}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":0}
11 tokens · 0.305s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.280s
Damaged · 1 GPU

wrong_schema_or_value

{"city":"Kyoto","units":19}
13 tokens · 0.364s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.283s
142 · filter · wording 3 · gaind5fab07717a2fd478e92d44a49a14bc5d58555eddb74eff00cc3879b3424cf61
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"M517","status":"pending"},{"id":"W602","status":"pending"},{"id":"T805","status":"ready"},{"id":"M250","status":"ready"},{"id":"P530","status":"closed"}]
Checked answer
{"ids": ["M250", "T805"]}
Seed

wrong_schema_or_value

{"ids": ["M250", "M517", "P530", "T805", "W602"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["M250","T805"]}
16 tokens · 0.448s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["T805","M250"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["M250","T805"]}
16 tokens · 0.447s
143 · total · wording 3 · same57290b544d050a72fa728e68741e21b7f203049c3822d2d8c59af9114643d130
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"G193","units":14,"price":5},{"id":"E989","units":6,"price":9},{"id":"I519","units":19,"price":2},{"id":"A501","units":12,"price":18},{"id":"T116","units":4,"price":11}]
Checked answer
{"total": 422}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.331s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.226s
144 · sort · wording 3 · gain048d714706ba7cb2b8f48d4e0bb334da4e81ba008140f35ac392c6f1a30e5b13
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"I784","priority":7},{"id":"T782","priority":1},{"id":"O151","priority":9},{"id":"H847","priority":5},{"id":"A254","priority":2}]
Checked answer
{"ids": ["O151", "I784"]}
Seed

invalid_json_object

[{"id":"I784","priority":7},{"id":"T782","priority":1},{"id":"O151","priority":9},{"id":"H847","priority":5},{"id":"A254","priority":2}]
62 tokens · 1.689s
Clean · 1 GPU

Correct

{"ids":["O151","I784"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["O151","I784"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["O151","I784"]}
16 tokens · 0.448s
145 · lookup · wording 0 · same6d56705672eec2493a4451bd68c6faf4045c490d23957f8301c0f832d844cd72
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record H467. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H467","city":"Riga","units":15},{"id":"I606","city":"Riga","units":5},{"id":"X767","city":"Suva","units":15},{"id":"B662","city":"Perth","units":13},{"id":"Y375","city":"Bern","units":8}]
Checked answer
{"city": "Riga", "units": 15}
Seed

Correct

{
  "city": "Riga",
  "units": 15
}
19 tokens · 0.522s
Clean · 1 GPU

Correct

{"city":"Riga","units":15}
12 tokens · 0.338s
Damaged · 1 GPU

Correct

{"city":"Riga","units":15}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Riga","units":15}
12 tokens · 0.341s
146 · filter · wording 0 · gain3852ad5ea15e6213feb9b635dfcaf5504961507be394109eb0c6c52281cf42fe
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"J817","status":"pending"},{"id":"N779","status":"pending"},{"id":"R100","status":"closed"},{"id":"M770","status":"closed"},{"id":"M201","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["M201", "M770", "N779", "R100", "J817"]}
35 tokens · 0.952s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.171s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.173s
147 · total · wording 0 · samee6f51abeee8a8273965ca72771373d4f838983b82cef63e038286d7907d8534b
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"J897","units":18,"price":18},{"id":"Y772","units":15,"price":2},{"id":"D578","units":14,"price":4},{"id":"R955","units":0,"price":11},{"id":"R757","units":18,"price":9}]
Checked answer
{"total": 572}
Seed

invalid_json_object

{"total": 18 * 18 + 15 * 2 + 14 * 4 + 0 * 11 + 18 * 9}
40 tokens · 1.093s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":433}
8 tokens · 0.228s
148 · sort · wording 0 · same93c8ccf77ca5831ab0ff5bc2f5a63717e1d823f328001d7f12e34a72c031ee20
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Q225","priority":9},{"id":"K964","priority":7},{"id":"Q943","priority":7},{"id":"U541","priority":5},{"id":"H578","priority":7}]
Checked answer
{"ids": ["Q225", "H578"]}
Seed

wrong_schema_or_value

{"ids": ["Q225", "K964", "Q943", "U541", "H578"]}
35 tokens · 0.957s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["Q225","Q943"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["Q225","H578"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["Q225","H578"]}
16 tokens · 0.451s
149 · lookup · wording 1 · same7d46c7756978dc4b3f764ef8e47c236410eaea9acf278afcb6e2f9888d2a2c8b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id P288 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"L158","city":"Bern","units":5},{"id":"D571","city":"Perth","units":14},{"id":"G804","city":"Lima","units":7},{"id":"P288","city":"Perth","units":16},{"id":"J726","city":"Accra","units":17}]
Checked answer
{"city": "Perth", "units": 16}
Seed

Correct

{"city":"Perth","units":16}
12 tokens · 0.333s
Clean · 1 GPU

Correct

{"city":"Perth","units":16}
12 tokens · 0.336s
Damaged · 1 GPU

Correct

{"city":"Perth","units":16}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Perth","units":16}
12 tokens · 0.339s
150 · filter · wording 1 · gain140724eef54e7ab306b7465a1301b80eccdd133ccb0fa109d96b77fc20a1af90
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"D881","status":"closed"},{"id":"O535","status":"ready"},{"id":"V311","status":"ready"},{"id":"F692","status":"pending"},{"id":"K588","status":"pending"}]
Checked answer
{"ids": ["O535", "V311"]}
Seed

wrong_schema_or_value

{
  "ids": ["O535", "V311", "F692", "D881"]
}
32 tokens · 0.874s
Clean · 1 GPU

Correct

{"ids":["O535","V311"]}
16 tokens · 0.461s
Damaged · 1 GPU

Correct

{"ids":["O535","V311"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["O535","V311"]}
16 tokens · 0.448s
151 · total · wording 1 · same931b0967d8ca1b55c8f1b28861a38aaf5bdeeb4ab93a8fbaf2f2ce599f119713
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"X729","units":15,"price":8},{"id":"R813","units":0,"price":17},{"id":"L250","units":15,"price":3},{"id":"B953","units":11,"price":13},{"id":"B141","units":6,"price":3}]
Checked answer
{"total": 326}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.233s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
152 · sort · wording 1 · gain2e03ca7d6d97a50450546de6bea10985638f3c079a2e2b1b566e6b55a965efb1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"W838","priority":9},{"id":"V296","priority":2},{"id":"L682","priority":8},{"id":"Q158","priority":2},{"id":"Z359","priority":4}]
Checked answer
{"ids": ["W838", "L682"]}
Seed

invalid_json_object

[{"id":"W838","priority":9},{"id":"V296","priority":2},{"id":"L682","priority":8},{"id":"Q158","priority":2},{"id":"Z359","priority":4}]
62 tokens · 1.697s
Clean · 1 GPU

Correct

{"ids":["W838","L682"]}
16 tokens · 0.461s
Damaged · 1 GPU

Correct

{"ids":["W838","L682"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["W838","L682"]}
16 tokens · 0.447s
153 · lookup · wording 2 · gain9ee5efd2404d7a2babd8bfc15bdaf9a11beac879d10d743981cdd46e3e43b6ef
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z323? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"G549","city":"Kyoto","units":16},{"id":"F235","city":"Perth","units":15},{"id":"Q300","city":"Accra","units":3},{"id":"V687","city":"Suva","units":5},{"id":"Z323","city":"Perth","units":11}]
Checked answer
{"city": "Perth", "units": 11}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":"Perth","units":11}
12 tokens · 0.335s
Damaged · 1 GPU

Correct

{"city":"Perth","units":11}
12 tokens · 0.339s
Clean · 2 GPUs

Correct

{"city":"Perth","units":11}
12 tokens · 0.337s
154 · filter · wording 2 · gainfc005ae7a19cfc2d0e0f6b5b5b9ae04ff4f8dedffd4409f3766b2a3a758a34b3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S396","status":"pending"},{"id":"H236","status":"ready"},{"id":"Q470","status":"closed"},{"id":"V334","status":"ready"},{"id":"X795","status":"pending"}]
Checked answer
{"ids": ["H236", "V334"]}
Seed

invalid_json_object

[{"id":"H236","status":"ready"},{"id":"V334","status":"ready"},{"id":"X795","status":"pending"},{"id":"S396","status":"pending"},{"id":"Q470","status":"closed"}]
63 tokens · 1.726s
Clean · 1 GPU

Correct

{"ids":["H236","V334"]}
16 tokens · 0.445s
Damaged · 1 GPU

Correct

{"ids":["H236","V334"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["H236","V334"]}
16 tokens · 0.449s
155 · total · wording 2 · same590435f85a761d00670ba2bcf33d7cad41ef9c5355fb0b68d0b20c7278679fde
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Z104","units":13,"price":18},{"id":"S798","units":3,"price":9},{"id":"Z678","units":6,"price":1},{"id":"D289","units":18,"price":13},{"id":"Q653","units":13,"price":1}]
Checked answer
{"total": 514}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":371}
8 tokens · 0.228s
156 · sort · wording 2 · gain2f859217c4a53a4d60e2fc2ed6c5c3712f20ba997cd2c975b0c043033ed438ba
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J613","priority":8},{"id":"L946","priority":9},{"id":"F341","priority":2},{"id":"Z394","priority":5},{"id":"D522","priority":7}]
Checked answer
{"ids": ["L946", "J613"]}
Seed

wrong_schema_or_value

{"ids": ["J613", "L946", "F341", "Z394", "D522"]}
35 tokens · 0.963s
Clean · 1 GPU

Correct

{"ids":["L946","J613"]}
16 tokens · 0.448s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["L946","D522"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["L946","J613"]}
16 tokens · 0.449s
157 · lookup · wording 3 · gain762bdee6a2425c3e7e19b10c7e04fb88911fbc3f74889dd8b05f138ac6603404
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=G389. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"W911","city":"Riga","units":4},{"id":"L644","city":"Riga","units":13},{"id":"X330","city":"Kyoto","units":11},{"id":"S845","city":"Lima","units":13},{"id":"G389","city":"Riga","units":3}]
Checked answer
{"city": "Riga", "units": 3}
Seed

wrong_schema_or_value

{"city":"Riga","units":4}
11 tokens · 0.308s
Clean · 1 GPU

Correct

{"city":"Riga","units":3}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"city":"Riga","units":3}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"city":"Riga","units":3}
11 tokens · 0.311s
158 · filter · wording 3 · gain22e4fb2259b79d7e9d8e1b7b84ff969d1f9e7f8709f1385febbfd6689f45dfba
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"T144","status":"closed"},{"id":"F224","status":"closed"},{"id":"Q143","status":"closed"},{"id":"Y754","status":"closed"},{"id":"X259","status":"ready"}]
Checked answer
{"ids": ["X259"]}
Seed

wrong_schema_or_value

{"ids": ["X259", "F224", "Y754", "Q143", "T144"]}
35 tokens · 0.963s
Clean · 1 GPU

Correct

{"ids":["X259"]}
11 tokens · 0.308s
Damaged · 1 GPU

Correct

{"ids":["X259"]}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"ids":["X259"]}
11 tokens · 0.307s
159 · total · wording 3 · samebb4b191f771fbe752ac01e332c512cb2f739a8eec2f1844f3e90760a16e79f92
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"L302","units":7,"price":4},{"id":"R585","units":19,"price":17},{"id":"P644","units":14,"price":9},{"id":"H764","units":15,"price":11},{"id":"B472","units":2,"price":8}]
Checked answer
{"total": 658}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.336s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.228s
160 · sort · wording 3 · gain6476e4c75527b75b547ef8a6878f0687c7e5cef9c18acf32ee46da56c6b1c71e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"E556","priority":2},{"id":"Z217","priority":4},{"id":"L487","priority":6},{"id":"O954","priority":7},{"id":"V273","priority":3}]
Checked answer
{"ids": ["O954", "L487"]}
Seed

invalid_json_object

[{"id":"O954","priority":7},{"id":"V273","priority":3},{"id":"E556","priority":2},{"id":"Z217","priority":4},{"id":"L487","priority":6}]
62 tokens · 1.697s
Clean · 1 GPU

Correct

{"ids":["O954","L487"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["O954","L487"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["O954","L487"]}
16 tokens · 0.449s
161 · lookup · wording 0 · gaina2d0d010d5b9f97738f8fb2754d27be79f291da7ae752a882311483ff45c8d51
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"R340","city":"Lima","units":6},{"id":"X222","city":"Kyoto","units":13},{"id":"T494","city":"Perth","units":9},{"id":"P347","city":"Bern","units":2},{"id":"S277","city":"Lima","units":4}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Lima","units":6},{"city":"Kyoto","units":13},{"city":"Perth","units":9},{"city":"Bern","units":2},{"city":"Lima","units":4}]
53 tokens · 1.453s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.281s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.285s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.283s
162 · filter · wording 0 · gain261f28f8fafaea8cdc9e61e8c752cd084f217b9eac2cafcbfc6c0dfaba99b748
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"N577","status":"ready"},{"id":"C453","status":"pending"},{"id":"K576","status":"closed"},{"id":"V390","status":"ready"},{"id":"L496","status":"pending"}]
Checked answer
{"ids": ["N577", "V390"]}
Seed

wrong_schema_or_value

{"ids": ["C453", "K576", "L496", "N577", "V390"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["N577","V390"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["N577","V390"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["N577","V390"]}
16 tokens · 0.448s
163 · total · wording 0 · same2c638769a4844b72af83b6e910f1119fd1ba1fe2a4e6aaaf70890c4c640b77da
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"R943","units":16,"price":13},{"id":"F718","units":12,"price":11},{"id":"Q357","units":15,"price":8},{"id":"U549","units":12,"price":9},{"id":"P180","units":12,"price":9}]
Checked answer
{"total": 676}
Seed

wrong_schema_or_value

{"total": 130}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":471}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.228s
164 · sort · wording 0 · gaind587acf747b460c50f2d2a7b15db8d9c7bfa7967b44149d1bb038e148f9c8810
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"M883","priority":7},{"id":"C961","priority":6},{"id":"Y898","priority":9},{"id":"T528","priority":9},{"id":"N948","priority":9}]
Checked answer
{"ids": ["N948", "T528"]}
Seed

wrong_schema_or_value

{"ids": ["M883", "Y898", "T528", "N948", "C961", "P900"]}
41 tokens · 1.124s
Clean · 1 GPU

Correct

{"ids":["N948","T528"]}
16 tokens · 0.446s
Damaged · 1 GPU

Correct

{"ids":["N948","T528"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["N948","T528"]}
16 tokens · 0.449s
165 · lookup · wording 1 · gaindd99c3054e6fa338409c4f34d6a1664edba9d5a6ea2cb4f15b0da494beb56217
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id P787 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"F762","city":"Lima","units":12},{"id":"L621","city":"Kyoto","units":3},{"id":"P787","city":"Kyoto","units":18},{"id":"Z955","city":"Oslo","units":6},{"id":"T121","city":"Bern","units":10}]
Checked answer
{"city": "Kyoto", "units": 18}
Seed

wrong_schema_or_value

{"city":"Lima","units":12}
12 tokens · 0.333s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":18}
13 tokens · 0.367s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":18}
13 tokens · 0.364s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":18}
13 tokens · 0.369s
166 · filter · wording 1 · gainde3e4d98417b6660becea59317936f3c7e256fc507cbbdecd9fa46578d9fc38b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"E926","status":"closed"},{"id":"T526","status":"closed"},{"id":"T856","status":"ready"},{"id":"U675","status":"pending"},{"id":"A415","status":"ready"}]
Checked answer
{"ids": ["A415", "T856"]}
Seed

wrong_schema_or_value

{
  "ids": ["A415", "E926", "T856", "T526"]
}
32 tokens · 0.873s
Clean · 1 GPU

Correct

{"ids":["A415","T856"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["A415","T856"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["A415","T856"]}
16 tokens · 0.456s
167 · total · wording 1 · samedd066fe5ee3bbd1982e3a0be2bd547341e059139417ea93a4713db7199272754
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D336","units":5,"price":1},{"id":"K510","units":11,"price":11},{"id":"L663","units":19,"price":5},{"id":"A715","units":11,"price":5},{"id":"F438","units":11,"price":17}]
Checked answer
{"total": 463}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.226s
Clean · 2 GPUs

wrong_schema_or_value

{"total":441}
8 tokens · 0.229s
168 · sort · wording 1 · gaincd674abcefc229834e89d0cd7f2ac5d7cb4a6237b99a5bbbc0254f3832af6472
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"L492","priority":1},{"id":"B322","priority":2},{"id":"O528","priority":3},{"id":"Z222","priority":4},{"id":"S494","priority":7}]
Checked answer
{"ids": ["S494", "Z222"]}
Seed

invalid_json_object

[{"id":"L492","priority":1},{"id":"B322","priority":2},{"id":"O528","priority":3},{"id":"Z222","priority":4},{"id":"S494","priority":7}]
62 tokens · 1.693s
Clean · 1 GPU

Correct

{"ids":["S494","Z222"]}
16 tokens · 0.442s
Damaged · 1 GPU

Correct

{"ids":["S494","Z222"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["S494","Z222"]}
16 tokens · 0.450s
169 · lookup · wording 2 · gainaa4ee8ddece8d3f3ff44c65907beb1b57d38d448eb1bd40da4264ea09517fd79
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to K226? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"U175","city":"Lima","units":12},{"id":"K226","city":"Accra","units":15},{"id":"V879","city":"Riga","units":5},{"id":"K396","city":"Bern","units":5},{"id":"H410","city":"Perth","units":15}]
Checked answer
{"city": "Accra", "units": 15}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":"Accra","units":15}
12 tokens · 0.338s
Damaged · 1 GPU

Correct

{"city":"Accra","units":15}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Accra","units":15}
12 tokens · 0.341s
170 · filter · wording 2 · gain0fb0036d9846661c3a41b30fd591bfd047cec6794d151a3784334433a42c6ac2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S529","status":"ready"},{"id":"E209","status":"pending"},{"id":"Z535","status":"ready"},{"id":"F265","status":"ready"},{"id":"A593","status":"ready"}]
Checked answer
{"ids": ["A593", "F265", "S529", "Z535"]}
Seed

invalid_json_object

[{"id":"A593","status":"ready"},{"id":"F265","status":"ready"},{"id":"E209","status":"pending"},{"id":"S529","status":"ready"},{"id":"Z535","status":"ready"}]
63 tokens · 1.716s
Clean · 1 GPU

Correct

{"ids":["A593","F265","S529","Z535"]}
26 tokens · 0.723s
Damaged · 1 GPU

Correct

{"ids":["A593","F265","S529","Z535"]}
26 tokens · 0.725s
Clean · 2 GPUs

Correct

{"ids":["A593","F265","S529","Z535"]}
26 tokens · 0.745s
171 · total · wording 2 · sameeaf04d2c93b1c653507651aa8da0f1668b143d6be1e2a50d0407f5551976fa92
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"D976","units":12,"price":12},{"id":"K382","units":5,"price":16},{"id":"C933","units":13,"price":1},{"id":"N247","units":4,"price":5},{"id":"T482","units":9,"price":4}]
Checked answer
{"total": 293}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":311}
8 tokens · 0.228s
172 · sort · wording 2 · gainbf610344f129f0064515faf5f2e2af11342e43d1302579178abb2f58c0222fdc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Z804","priority":8},{"id":"Z554","priority":4},{"id":"M943","priority":3},{"id":"I527","priority":1},{"id":"O770","priority":8}]
Checked answer
{"ids": ["O770", "Z804"]}
Seed

wrong_schema_or_value

{"ids": ["M943", "O770"]}
17 tokens · 0.467s
Clean · 1 GPU

Correct

{"ids":["O770","Z804"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["O770","Z804"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["O770","Z804"]}
16 tokens · 0.448s
173 · lookup · wording 3 · gain1dc5aaaa25953ca590cd8ce83629e706d402fc37ddb7ece432ff48037a09a90c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z746. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P953","city":"Kyoto","units":5},{"id":"U301","city":"Kyoto","units":3},{"id":"C823","city":"Oslo","units":4},{"id":"C565","city":"Riga","units":18},{"id":"Z746","city":"Lima","units":13}]
Checked answer
{"city": "Lima", "units": 13}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":5}
12 tokens · 0.331s
Clean · 1 GPU

Correct

{"city":"Lima","units":13}
12 tokens · 0.339s
Damaged · 1 GPU

Correct

{"city":"Lima","units":13}
12 tokens · 0.337s
Clean · 2 GPUs

Correct

{"city":"Lima","units":13}
12 tokens · 0.338s
174 · filter · wording 3 · gainf1b90fb4facfb300d33c50e406ab91c881a00546bbbfd42508fd25103f051583
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"F889","status":"pending"},{"id":"Z138","status":"ready"},{"id":"R442","status":"pending"},{"id":"X883","status":"ready"},{"id":"J765","status":"ready"}]
Checked answer
{"ids": ["J765", "X883", "Z138"]}
Seed

wrong_schema_or_value

{"ids": ["F889", "Z138", "R442", "X883", "J765"]}
35 tokens · 0.953s
Clean · 1 GPU

Correct

{"ids":["J765","X883","Z138"]}
21 tokens · 0.580s
Damaged · 1 GPU

Correct

{"ids":["J765","X883","Z138"]}
21 tokens · 0.589s
Clean · 2 GPUs

Correct

{"ids":["J765","X883","Z138"]}
21 tokens · 0.585s
175 · total · wording 3 · same204b3c8b4fa5d38f945bd83756c55e2d0540209b452d0d75223bc904ff8d9153
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Q474","units":16,"price":16},{"id":"B467","units":13,"price":9},{"id":"S974","units":19,"price":15},{"id":"L605","units":4,"price":13},{"id":"M503","units":3,"price":12}]
Checked answer
{"total": 746}
Seed

wrong_schema_or_value

{
  "total": 438
}
12 tokens · 0.331s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":473}
8 tokens · 0.228s
176 · sort · wording 3 · gainc75c20979debb33cf86df75fe5d4fbce6c45aceaa5ea4b8e3f64fd4a7b34679c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"F479","priority":5},{"id":"U459","priority":7},{"id":"G610","priority":7},{"id":"D455","priority":4},{"id":"D981","priority":6}]
Checked answer
{"ids": ["G610", "U459"]}
Seed

invalid_json_object

[{"id":"U459","priority":7},{"id":"D455","priority":4},{"id":"D981","priority":6},{"id":"F479","priority":5},{"id":"G610","priority":7}]
62 tokens · 1.682s
Clean · 1 GPU

Correct

{"ids":["G610","U459"]}
16 tokens · 0.445s
Damaged · 1 GPU

Correct

{"ids":["G610","U459"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["G610","U459"]}
16 tokens · 0.450s
177 · lookup · wording 0 · gainda6fc5bd9b0baf67abbb225ea3d5a15f03d833dcac5526a435233f7b5ddcc6f4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Q273. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y160","city":"Suva","units":10},{"id":"Q273","city":"Bern","units":18},{"id":"O778","city":"Perth","units":1},{"id":"T379","city":"Bern","units":19},{"id":"W999","city":"Perth","units":15}]
Checked answer
{"city": "Bern", "units": 18}
Seed

wrong_schema_or_value

{
  "city": "Suva",
  "units": 10
}
19 tokens · 0.520s
Clean · 1 GPU

Correct

{"city":"Bern","units":18}
11 tokens · 0.308s
Damaged · 1 GPU

Correct

{"city":"Bern","units":18}
11 tokens · 0.310s
Clean · 2 GPUs

Correct

{"city":"Bern","units":18}
11 tokens · 0.306s
178 · filter · wording 0 · gain4d3936d12c0643cd791cb582bb26919f6016c1f2f294231f073b59389df5a971
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L188","status":"closed"},{"id":"Z871","status":"pending"},{"id":"Q833","status":"ready"},{"id":"F816","status":"pending"},{"id":"B924","status":"pending"}]
Checked answer
{"ids": ["Q833"]}
Seed

wrong_schema_or_value

{"ids": ["B924", "F816", "L188", "Q833", "Z871"]}
35 tokens · 0.952s
Clean · 1 GPU

Correct

{"ids":["Q833"]}
11 tokens · 0.307s
Damaged · 1 GPU

Correct

{"ids":["Q833"]}
11 tokens · 0.311s
Clean · 2 GPUs

Correct

{"ids":["Q833"]}
11 tokens · 0.304s
179 · total · wording 0 · samef8927f9f1ec2ae74cf3367bfad0305d1e0fd7cd6d16dba4d8c1a426535f2a693
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"G679","units":4,"price":19},{"id":"S918","units":3,"price":3},{"id":"E569","units":9,"price":4},{"id":"F598","units":0,"price":4},{"id":"G282","units":3,"price":19}]
Checked answer
{"total": 178}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.250s
Clean · 1 GPU

wrong_schema_or_value

{"total":223}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":422}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":222}
8 tokens · 0.223s
180 · sort · wording 0 · sameaea26aa3a41d1712bc2880dc05752600e52eb9735ad7ac8837d73965f282663b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z715","priority":2},{"id":"Y705","priority":6},{"id":"N328","priority":7},{"id":"Z335","priority":7},{"id":"N566","priority":6}]
Checked answer
{"ids": ["N328", "Z335"]}
Seed

wrong_schema_or_value

{"ids": ["N328", "N566", "Z715", "Z335", "Y705"]}
35 tokens · 0.959s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["N328","N566"]}
16 tokens · 0.454s
Damaged · 1 GPU

Correct

{"ids":["N328","Z335"]}
16 tokens · 0.449s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["N328","N566"]}
16 tokens · 0.446s
181 · lookup · wording 1 · gain4a004445b10076b20a22f53e64fbb130193e74b2ed7e44fc29ddc92074a70112
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"J491","city":"Suva","units":0},{"id":"E639","city":"Suva","units":6},{"id":"S649","city":"Riga","units":16},{"id":"D922","city":"Bern","units":5},{"id":"Y937","city":"Oslo","units":18}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Suva","units":0}
11 tokens · 0.305s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.283s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.281s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.280s
182 · filter · wording 1 · gain748dbf51d954652022bf28a76f90f04779ead3c9e9cf81096f41fb27d4c87590
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"K774","status":"pending"},{"id":"R869","status":"ready"},{"id":"Z235","status":"closed"},{"id":"Z232","status":"pending"},{"id":"R769","status":"pending"}]
Checked answer
{"ids": ["R869"]}
Seed

wrong_schema_or_value

{
  "ids": ["R769", "R869", "Z232", "Z235"]
}
32 tokens · 0.874s
Clean · 1 GPU

Correct

{"ids":["R869"]}
11 tokens · 0.307s
Damaged · 1 GPU

Correct

{"ids":["R869"]}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"ids":["R869"]}
11 tokens · 0.310s
183 · total · wording 1 · same622e8c4b8631b9ecbb810875a1a206d2dea73416f6a0f339e45a1f4b0e51dee3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"X194","units":19,"price":8},{"id":"E167","units":15,"price":2},{"id":"H200","units":14,"price":3},{"id":"M957","units":14,"price":14},{"id":"V311","units":17,"price":19}]
Checked answer
{"total": 743}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":711}
8 tokens · 0.229s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":731}
8 tokens · 0.228s
184 · sort · wording 1 · gainbfa2b8435f93fa3dea5eaa943182976f60cd1ba372d94191cb4ba24ff3033f41
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"E273","priority":3},{"id":"J604","priority":8},{"id":"S773","priority":9},{"id":"H860","priority":7},{"id":"B343","priority":6}]
Checked answer
{"ids": ["S773", "J604"]}
Seed

invalid_json_object

[{"id":"E273","priority":3},{"id":"J604","priority":8},{"id":"S773","priority":9},{"id":"H860","priority":7},{"id":"B343","priority":6}]
62 tokens · 1.689s
Clean · 1 GPU

Correct

{"ids":["S773","J604"]}
16 tokens · 0.450s
Damaged · 1 GPU

Correct

{"ids":["S773","J604"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["S773","J604"]}
16 tokens · 0.448s
185 · lookup · wording 2 · gain6717ec4887f8a1cc0d113a0c1840ea8339dec25fb3b749a06f0a718489c3f778
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to X963? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E407","city":"Perth","units":12},{"id":"N326","city":"Perth","units":17},{"id":"X963","city":"Accra","units":12},{"id":"O547","city":"Kyoto","units":16},{"id":"W405","city":"Oslo","units":7}]
Checked answer
{"city": "Accra", "units": 12}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

Correct

{"city":"Accra","units":12}
12 tokens · 0.339s
Damaged · 1 GPU

Correct

{"city":"Accra","units":12}
12 tokens · 0.333s
Clean · 2 GPUs

Correct

{"city":"Accra","units":12}
12 tokens · 0.339s
186 · filter · wording 2 · gainb77708e83661435a42e858d6e86428951954b14fb628e9247ca14e7062d74a80
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"B568","status":"closed"},{"id":"S199","status":"pending"},{"id":"N852","status":"closed"},{"id":"D377","status":"pending"},{"id":"Z917","status":"ready"}]
Checked answer
{"ids": ["Z917"]}
Seed

invalid_json_object

[{"id":"B568"},{"id":"D377"},{"id":"Z917"}]
27 tokens · 0.741s
Clean · 1 GPU

Correct

{"ids":["Z917"]}
11 tokens · 0.310s
Damaged · 1 GPU

Correct

{"ids":["Z917"]}
11 tokens · 0.302s
Clean · 2 GPUs

Correct

{"ids":["Z917"]}
11 tokens · 0.311s
187 · total · wording 2 · samebd6e6baa85723ca6e6c207a12525fa231167b1baee018c8cfa40d632b10a03f5
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"U468","units":2,"price":11},{"id":"X819","units":8,"price":5},{"id":"M327","units":9,"price":12},{"id":"Q719","units":4,"price":15},{"id":"P362","units":13,"price":6}]
Checked answer
{"total": 308}
Seed

wrong_schema_or_value

{"total": 110}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":310}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":310}
8 tokens · 0.228s
188 · sort · wording 2 · gaincb810edd211523ed8cb1cd82d8bd344597bcb37c2a04e836389f8e4711aff246
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S118","priority":2},{"id":"S398","priority":2},{"id":"U697","priority":7},{"id":"E461","priority":8},{"id":"T600","priority":1}]
Checked answer
{"ids": ["E461", "U697"]}
Seed

wrong_schema_or_value

{"ids": ["S118", "S398", "U697", "E461", "T600"]}
35 tokens · 0.971s
Clean · 1 GPU

Correct

{"ids":["E461","U697"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["E461","U697"]}
16 tokens · 0.444s
Clean · 2 GPUs

Correct

{"ids":["E461","U697"]}
16 tokens · 0.449s
189 · lookup · wording 3 · same27a795c5135557e68015e404d9f6a390d20aa4e2471a1fb756785982c1dc1b21
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=G660. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"J406","city":"Perth","units":13},{"id":"Y276","city":"Accra","units":19},{"id":"E194","city":"Perth","units":13},{"id":"G660","city":"Suva","units":14},{"id":"D532","city":"Suva","units":18}]
Checked answer
{"city": "Suva", "units": 14}
Seed

Correct

{"city":"Suva","units":14}
12 tokens · 0.345s
Clean · 1 GPU

Correct

{"city":"Suva","units":14}
12 tokens · 0.337s
Damaged · 1 GPU

Correct

{"city":"Suva","units":14}
12 tokens · 0.334s
Clean · 2 GPUs

Correct

{"city":"Suva","units":14}
12 tokens · 0.339s
190 · filter · wording 3 · gain3ea7a1d97b94b07d9223ad12acc23b86f5c6ff31d00b9f4246ce2a836af76805
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"W565","status":"closed"},{"id":"M982","status":"closed"},{"id":"W874","status":"pending"},{"id":"E498","status":"closed"},{"id":"M213","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["M213", "E498", "M982", "W565", "W874", "W874"]}
41 tokens · 1.126s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.172s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.304s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.173s
191 · total · wording 3 · samea7e8e207d42cc1f4f061d76b3194fa90cf3cdaa4318c1916c6f19d06fab1f39a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"T646","units":14,"price":14},{"id":"T944","units":16,"price":18},{"id":"P543","units":12,"price":13},{"id":"H735","units":18,"price":12},{"id":"N824","units":4,"price":6}]
Checked answer
{"total": 880}
Seed

wrong_schema_or_value

{
  "total": 43
}
11 tokens · 0.307s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.223s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.229s
192 · sort · wording 3 · gaina2027ed86c4736b5a78d0daa10201108c2f9673303a07819b90d16dfa82cece5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"D659","priority":8},{"id":"U848","priority":5},{"id":"G539","priority":4},{"id":"K553","priority":6},{"id":"V724","priority":1}]
Checked answer
{"ids": ["D659", "K553"]}
Seed

invalid_json_object

[{"id":"U848","priority":5},{"id":"G539","priority":4},{"id":"V724","priority":1},{"id":"D659","priority":8},{"id":"K553","priority":6}]
62 tokens · 1.685s
Clean · 1 GPU

Correct

{"ids":["D659","K553"]}
16 tokens · 0.443s
Damaged · 1 GPU

Correct

{"ids":["D659","K553"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["D659","K553"]}
16 tokens · 0.448s
193 · lookup · wording 0 · same00c365bb3f7faa3d126c00729952f15bf1fd1a2f233259dfebc879f775acbfd8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record U671. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"K828","city":"Suva","units":1},{"id":"K789","city":"Accra","units":8},{"id":"I930","city":"Kyoto","units":8},{"id":"U671","city":"Kyoto","units":5},{"id":"V740","city":"Riga","units":18}]
Checked answer
{"city": "Kyoto", "units": 5}
Seed

Correct

{
  "city": "Kyoto",
  "units": 5
}
19 tokens · 0.523s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":5}
12 tokens · 0.337s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":5}
12 tokens · 0.332s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":5}
12 tokens · 0.337s
194 · filter · wording 0 · gain81cd7f10835a55945cbe2a3f7db48572d35bd82200d0ac99a542e8e1860cc98b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"E676","status":"pending"},{"id":"P851","status":"closed"},{"id":"V690","status":"pending"},{"id":"C339","status":"closed"},{"id":"B503","status":"ready"}]
Checked answer
{"ids": ["B503"]}
Seed

wrong_schema_or_value

{"ids": ["B503", "C339", "E676", "P851", "V690"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["B503"]}
11 tokens · 0.305s
Damaged · 1 GPU

Correct

{"ids":["B503"]}
11 tokens · 0.305s
Clean · 2 GPUs

Correct

{"ids":["B503"]}
11 tokens · 0.311s
195 · total · wording 0 · samefb1fcf6ef33d2830dd4c8d0463beda19e44a5af88bdf16a0c935ce76fa1aa79f
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"V274","units":14,"price":11},{"id":"E908","units":1,"price":12},{"id":"Z906","units":10,"price":3},{"id":"Y264","units":15,"price":16},{"id":"P994","units":5,"price":7}]
Checked answer
{"total": 471}
Seed

invalid_json_object

{"total": 11 + 12 + 3 + 16 + 7 = 47}
26 tokens · 0.716s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.224s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":331}
8 tokens · 0.228s
196 · sort · wording 0 · gain35438acb3b4a57ef854b849bb99acd45109bdca56bade724fe107932a4b65033
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"T873","priority":2},{"id":"K859","priority":4},{"id":"E504","priority":9},{"id":"G751","priority":8},{"id":"A923","priority":7}]
Checked answer
{"ids": ["E504", "G751"]}
Seed

wrong_schema_or_value

{"ids": ["K859", "E504", "G751", "A923", "T873"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["E504","G751"]}
16 tokens · 0.440s
Damaged · 1 GPU

Correct

{"ids":["E504","G751"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["E504","G751"]}
16 tokens · 0.442s
197 · lookup · wording 1 · same90a864c754a88784b561b16d69197c3500be35ec385547a6b2924d33af8fcf67
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id X286 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"V946","city":"Kyoto","units":9},{"id":"L220","city":"Suva","units":4},{"id":"Q258","city":"Riga","units":14},{"id":"W955","city":"Bern","units":11},{"id":"X286","city":"Kyoto","units":13}]
Checked answer
{"city": "Kyoto", "units": 13}
Seed

Correct

{"city":"Kyoto","units":13}
13 tokens · 0.361s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":13}
13 tokens · 0.357s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":13}
13 tokens · 0.374s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":13}
13 tokens · 0.366s
198 · filter · wording 1 · gain8f7ead960c9946ac71e9530538bd6d691446f6d59ac31c42ba8fed83d13d46a6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"H965","status":"closed"},{"id":"S431","status":"pending"},{"id":"N312","status":"closed"},{"id":"Q506","status":"ready"},{"id":"A210","status":"pending"}]
Checked answer
{"ids": ["Q506"]}
Seed

wrong_schema_or_value

{
  "ids": ["A210", "H965", "N312", "Q506", "S431"]
}
38 tokens · 1.041s
Clean · 1 GPU

Correct

{"ids":["Q506"]}
11 tokens · 0.305s
Damaged · 1 GPU

Correct

{"ids":["Q506"]}
11 tokens · 0.315s
Clean · 2 GPUs

Correct

{"ids":["Q506"]}
11 tokens · 0.310s
199 · total · wording 1 · same5c9437011d0fb327c7b1269ae8ee4b76ea9b97abfbb693c5a2f667c5f94ded83
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"W808","units":5,"price":1},{"id":"Q528","units":7,"price":3},{"id":"U704","units":0,"price":12},{"id":"V292","units":8,"price":11},{"id":"N719","units":11,"price":11}]
Checked answer
{"total": 235}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":311}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.236s
Clean · 2 GPUs

wrong_schema_or_value

{"total":241}
8 tokens · 0.227s
200 · sort · wording 1 · gain523e14fee91aa8b63fce51dcccae3485704e8e8a9f182380e362a9350071d58a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"L244","priority":4},{"id":"S210","priority":6},{"id":"H721","priority":2},{"id":"E968","priority":4},{"id":"Z960","priority":5}]
Checked answer
{"ids": ["S210", "Z960"]}
Seed

invalid_json_object

[{"id":"L244","priority":4},{"id":"S210","priority":6},{"id":"H721","priority":2},{"id":"E968","priority":4},{"id":"Z960","priority":5}]
62 tokens · 1.697s
Clean · 1 GPU

Correct

{"ids":["S210","Z960"]}
16 tokens · 0.450s
Damaged · 1 GPU

Correct

{"ids":["S210","Z960"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["S210","Z960"]}
16 tokens · 0.454s
201 · lookup · wording 2 · same526652e17bb584fce0bcf23875c3e0f6726ef4abfe34803a8e9cd4af33cf7b11
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T839","city":"Suva","units":7},{"id":"Z919","city":"Accra","units":5},{"id":"Z176","city":"Perth","units":17},{"id":"Q236","city":"Perth","units":6},{"id":"M604","city":"Riga","units":19}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens · 0.420s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.282s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.281s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.283s
202 · filter · wording 2 · gain14ca55401a7f18bf968a26c033b00167c6a60552d86143b6bcb4f821d93124b0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"A727","status":"ready"},{"id":"J276","status":"ready"},{"id":"W160","status":"closed"},{"id":"G430","status":"closed"},{"id":"X981","status":"ready"}]
Checked answer
{"ids": ["A727", "J276", "X981"]}
Seed

invalid_json_object

[{"id":"A727","status":"ready"},{"id":"J276","status":"ready"},{"id":"W160","status":"closed"},{"id":"G430","status":"closed"},{"id":"X981","status":"ready"}]
63 tokens · 1.719s
Clean · 1 GPU

Correct

{"ids":["A727","J276","X981"]}
21 tokens · 0.585s
Damaged · 1 GPU

Correct

{"ids":["A727","J276","X981"]}
21 tokens · 0.585s
Clean · 2 GPUs

Correct

{"ids":["A727","J276","X981"]}
21 tokens · 0.586s
203 · total · wording 2 · samed8aa12508cd81419d7e33e4a2a7c486e069ee4398d8b6fb73cff61ad5a0fd0c6
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"T858","units":0,"price":10},{"id":"C525","units":1,"price":5},{"id":"A400","units":17,"price":10},{"id":"G425","units":11,"price":19},{"id":"E667","units":17,"price":2}]
Checked answer
{"total": 418}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.224s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
204 · sort · wording 2 · gainbe1b7478dd9818bb76078fc0d25ff19fc8eedce4c96e9d3d1b36a5cd9820b55e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"M498","priority":1},{"id":"F692","priority":9},{"id":"Z955","priority":2},{"id":"C793","priority":4},{"id":"R595","priority":8}]
Checked answer
{"ids": ["F692", "R595"]}
Seed

wrong_schema_or_value

{"ids": ["M498", "Z955"]}
17 tokens · 0.469s
Clean · 1 GPU

Correct

{"ids":["F692","R595"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["F692","R595"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["F692","R595"]}
16 tokens · 0.450s
205 · lookup · wording 3 · gain0644accf0007aded319f85f3b44252faf7a5bb84df63d29b6fec8253a5b04978
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=X239. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"W549","city":"Suva","units":1},{"id":"X239","city":"Kyoto","units":14},{"id":"N611","city":"Riga","units":14},{"id":"L210","city":"Lima","units":1},{"id":"O934","city":"Oslo","units":0}]
Checked answer
{"city": "Kyoto", "units": 14}
Seed

wrong_schema_or_value

{"city":"Suva","units":1}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.366s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.365s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.367s
206 · filter · wording 3 · gain05358285d6b938d8c8b0bb1d350be50aa4c382cab53cb7ae764783f6eef67983
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"G433","status":"pending"},{"id":"Q116","status":"closed"},{"id":"R724","status":"pending"},{"id":"U622","status":"ready"},{"id":"S193","status":"pending"}]
Checked answer
{"ids": ["U622"]}
Seed

wrong_schema_or_value

{"ids": ["U622", "S193", "G433", "R724", "Q116"]}
35 tokens · 0.959s
Clean · 1 GPU

Correct

{"ids":["U622"]}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"ids":["U622"]}
11 tokens · 0.308s
Clean · 2 GPUs

Correct

{"ids":["U622"]}
11 tokens · 0.312s
207 · total · wording 3 · samece9a07ae9c3a6a158080ab057f8034e82395bc42db8bb3affe978299be7b7c52
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"W969","units":16,"price":1},{"id":"J812","units":1,"price":1},{"id":"Y477","units":10,"price":3},{"id":"U316","units":8,"price":12},{"id":"U978","units":16,"price":16}]
Checked answer
{"total": 399}
Seed

wrong_schema_or_value

{
  "total": 31
}
11 tokens · 0.306s
Clean · 1 GPU

wrong_schema_or_value

{"total":312}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.225s
208 · sort · wording 3 · gain642b8658cb62781cc160090f788fcb812b16b9b994b054c1456de479a73de858
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"D371","priority":7},{"id":"W427","priority":9},{"id":"R134","priority":2},{"id":"Q856","priority":5},{"id":"G641","priority":1}]
Checked answer
{"ids": ["W427", "D371"]}
Seed

invalid_json_object

[{"id":"W427","priority":9},{"id":"R134","priority":2},{"id":"Q856","priority":5},{"id":"D371","priority":7},{"id":"G641","priority":1}]
62 tokens · 1.698s
Clean · 1 GPU

Correct

{"ids":["W427","D371"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["W427","D371"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["W427","D371"]}
16 tokens · 0.446s
209 · lookup · wording 0 · gain1b6a7ce5a8443a87e2bad889944a204b0801946161e3fa6d36722861ea29790f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record R562. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"B409","city":"Kyoto","units":5},{"id":"J614","city":"Kyoto","units":8},{"id":"M625","city":"Perth","units":10},{"id":"X301","city":"Perth","units":3},{"id":"R562","city":"Oslo","units":14}]
Checked answer
{"city": "Oslo", "units": 14}
Seed

wrong_schema_or_value

{
  "city": "Kyoto",
  "units": 5
}
19 tokens · 0.525s
Clean · 1 GPU

Correct

{"city":"Oslo","units":14}
12 tokens · 0.337s
Damaged · 1 GPU

Correct

{"city":"Oslo","units":14}
12 tokens · 0.339s
Clean · 2 GPUs

Correct

{"city":"Oslo","units":14}
12 tokens · 0.339s
210 · filter · wording 0 · gain3e5a381fcbe3b546997aff7d9a5ccc8ab121019977381b9f6d883b10830798b4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"A277","status":"closed"},{"id":"L175","status":"pending"},{"id":"I398","status":"closed"},{"id":"M384","status":"pending"},{"id":"Q198","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A277", "L175", "I398", "M384", "Q198"]}
35 tokens · 0.955s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.171s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.170s
211 · total · wording 0 · same64026c63c93c30ca6e5c3e0e5f1a9d6059c60146d38062ec54cbf27ab546b3ba
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"M205","units":17,"price":2},{"id":"K770","units":3,"price":4},{"id":"W794","units":7,"price":8},{"id":"S614","units":13,"price":6},{"id":"N412","units":6,"price":2}]
Checked answer
{"total": 192}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":211}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":232}
8 tokens · 0.226s
Clean · 2 GPUs

wrong_schema_or_value

{"total":210}
8 tokens · 0.223s
212 · sort · wording 0 · gain0f7deca751f9c5c43a2b99c6349cfbae325bd784a70e91e600aac9fa30386008
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"M795","priority":6},{"id":"U245","priority":3},{"id":"E146","priority":2},{"id":"C652","priority":9},{"id":"N907","priority":5}]
Checked answer
{"ids": ["C652", "M795"]}
Seed

wrong_schema_or_value

{"ids": ["U245", "M795", "N907", "E146", "C652"]}
35 tokens · 0.957s
Clean · 1 GPU

Correct

{"ids":["C652","M795"]}
16 tokens · 0.444s
Damaged · 1 GPU

Correct

{"ids":["C652","M795"]}
16 tokens · 0.446s
Clean · 2 GPUs

Correct

{"ids":["C652","M795"]}
16 tokens · 0.442s
213 · lookup · wording 1 · gain38e48489d4efe4d7b3922c09cec5bba368444a785f0851114b574340f210da5d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id J906 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"J569","city":"Suva","units":6},{"id":"O394","city":"Oslo","units":9},{"id":"B502","city":"Kyoto","units":2},{"id":"G295","city":"Oslo","units":12},{"id":"J906","city":"Kyoto","units":14}]
Checked answer
{"city": "Kyoto", "units": 14}
Seed

wrong_schema_or_value

{"city":"Suva","units":6}
11 tokens · 0.306s
Clean · 1 GPU

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.366s
Damaged · 1 GPU

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.368s
Clean · 2 GPUs

Correct

{"city":"Kyoto","units":14}
13 tokens · 0.367s
214 · filter · wording 1 · gainca497c138fc9f0cbc9151aacef7f80fa5a94c1ee1e39a484db7ff025070db0e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"U655","status":"pending"},{"id":"L771","status":"pending"},{"id":"G401","status":"pending"},{"id":"W747","status":"closed"},{"id":"A333","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["U655", "L771", "G401", "W747", "A333"]
}
38 tokens · 1.041s
Clean · 1 GPU

Correct

{"ids":[]}
6 tokens · 0.172s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Z000"]}
11 tokens · 0.312s
Clean · 2 GPUs

Correct

{"ids":[]}
6 tokens · 0.170s
215 · total · wording 1 · samebb72a4934e9b13ee2931869ec697c6473be52a1a76a0d956b93f68f376694a23
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"O149","units":11,"price":11},{"id":"E241","units":19,"price":9},{"id":"Q809","units":9,"price":9},{"id":"N856","units":17,"price":3},{"id":"J890","units":8,"price":18}]
Checked answer
{"total": 568}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.224s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.228s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.225s
216 · sort · wording 1 · gainf67bbd24d84b3f5866c12e911303a6900cf612a1c142f186bcbc225dcb0c6a01
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J625","priority":3},{"id":"D924","priority":5},{"id":"M254","priority":3},{"id":"Y263","priority":4},{"id":"B351","priority":5}]
Checked answer
{"ids": ["B351", "D924"]}
Seed

invalid_json_object

[{"id":"M254","priority":3},{"id":"J625","priority":3},{"id":"D924","priority":5},{"id":"Y263","priority":4},{"id":"B351","priority":5}]
62 tokens · 1.694s
Clean · 1 GPU

Correct

{"ids":["B351","D924"]}
16 tokens · 0.447s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D924","B351"]}
16 tokens · 0.449s
Clean · 2 GPUs

Correct

{"ids":["B351","D924"]}
16 tokens · 0.445s
217 · lookup · wording 2 · gaine690df5037528c7413a408aa14b4ab7c86e2515803462adfaa385801e694dcab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to P554? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E112","city":"Kyoto","units":5},{"id":"I531","city":"Suva","units":3},{"id":"P554","city":"Lima","units":9},{"id":"P704","city":"Perth","units":7},{"id":"W171","city":"Accra","units":16}]
Checked answer
{"city": "Lima", "units": 9}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":"Lima","units":9}
11 tokens · 0.311s
Damaged · 1 GPU

Correct

{"city":"Lima","units":9}
11 tokens · 0.307s
Clean · 2 GPUs

Correct

{"city":"Lima","units":9}
11 tokens · 0.313s
218 · filter · wording 2 · gain53dc2c89cff9b405241f77214863da07b058e0efadf494f7a0462547a75c8836
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Z257","status":"closed"},{"id":"H299","status":"pending"},{"id":"W415","status":"pending"},{"id":"Y973","status":"closed"},{"id":"B547","status":"ready"}]
Checked answer
{"ids": ["B547"]}
Seed

invalid_json_object

[{"id":"B547","status":"ready"}]
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"ids":["B547"]}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"ids":["B547"]}
11 tokens · 0.311s
Clean · 2 GPUs

Correct

{"ids":["B547"]}
11 tokens · 0.312s
219 · total · wording 2 · same730ad59f92a594849944518b0e8f2cb67a3eab09ba10fd9e1573418d1726e4fa
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"C421","units":19,"price":1},{"id":"U251","units":15,"price":18},{"id":"U706","units":1,"price":7},{"id":"A471","units":12,"price":4},{"id":"O370","units":14,"price":13}]
Checked answer
{"total": 526}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":371}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":332}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.224s
220 · sort · wording 2 · gainc411e5523f5b5eddb0cfdbe0501df2bb001897150ea4291e26e9b8335202aca0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K925","priority":1},{"id":"R227","priority":8},{"id":"B116","priority":7},{"id":"P415","priority":2},{"id":"N973","priority":3}]
Checked answer
{"ids": ["R227", "B116"]}
Seed

wrong_schema_or_value

{"ids": ["K925", "R227", "B116", "P415", "N973"]}
35 tokens · 0.962s
Clean · 1 GPU

Correct

{"ids":["R227","B116"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["R227","B116"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["R227","B116"]}
16 tokens · 0.442s
221 · lookup · wording 3 · gain825e20b271b1b0e613113fc684bebfc00ea440c2691339a55f4baf87fe3159b5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"A256","city":"Riga","units":16},{"id":"G882","city":"Oslo","units":6},{"id":"N629","city":"Accra","units":6},{"id":"T573","city":"Oslo","units":13},{"id":"R903","city":"Lima","units":19}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":16}
12 tokens · 0.335s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.281s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.282s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.282s
222 · filter · wording 3 · gaine87f5e064dfadf671025339d256aa52c904ea9c0529cc8e73fec39a4bb411e85
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"N855","status":"closed"},{"id":"E271","status":"closed"},{"id":"T876","status":"closed"},{"id":"Q863","status":"ready"},{"id":"N960","status":"ready"}]
Checked answer
{"ids": ["N960", "Q863"]}
Seed

wrong_schema_or_value

{"ids": ["Q863", "N855", "E271", "T876", "N960"]}
35 tokens · 0.963s
Clean · 1 GPU

Correct

{"ids":["N960","Q863"]}
16 tokens · 0.448s
Damaged · 1 GPU

Correct

{"ids":["N960","Q863"]}
16 tokens · 0.443s
Clean · 2 GPUs

Correct

{"ids":["N960","Q863"]}
16 tokens · 0.442s
223 · total · wording 3 · same1ed7bc036596ffb4930dea8ef3112924fa5b1c9ef62c29f2ae72c06d79a305c0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"G132","units":13,"price":9},{"id":"Z475","units":5,"price":18},{"id":"Y505","units":14,"price":6},{"id":"J279","units":4,"price":15},{"id":"M597","units":13,"price":15}]
Checked answer
{"total": 546}
Seed

wrong_schema_or_value

{
  "total": 338
}
12 tokens · 0.335s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.227s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.229s
224 · sort · wording 3 · same2769fefc45dce9dbb2dc725474448cf6aba161cd757823c084481c534ff0accf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"K805","priority":1},{"id":"U211","priority":1},{"id":"L107","priority":2},{"id":"I523","priority":3},{"id":"G470","priority":4}]
Checked answer
{"ids": ["G470", "I523"]}
Seed

invalid_json_object

[{"id":"K805"},{"id":"U211"},{"id":"L107"},{"id":"I523"},{"id":"G470"}]
43 tokens · 1.179s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["G470","L107"]}
16 tokens · 0.444s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["G470","L107"]}
16 tokens · 0.447s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["G470","L107"]}
16 tokens · 0.452s
225 · lookup · wording 0 · gain36fb8fc8982ff576f4939a59c432d23bff07a4f72d9989bdf511b9563e1c154a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record B652. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Z986","city":"Lima","units":18},{"id":"B652","city":"Riga","units":4},{"id":"T337","city":"Suva","units":12},{"id":"L480","city":"Accra","units":8},{"id":"I703","city":"Oslo","units":12}]
Checked answer
{"city": "Riga", "units": 4}
Seed

invalid_json_object

[{"city":"Riga","units":4}]
12 tokens · 0.335s
Clean · 1 GPU

Correct

{"city":"Riga","units":4}
11 tokens · 0.313s
Damaged · 1 GPU

Correct

{"city":"Riga","units":4}
11 tokens · 0.311s
Clean · 2 GPUs

Correct

{"city":"Riga","units":4}
11 tokens · 0.312s
226 · filter · wording 0 · gain73b5bfc002e9009f15fe37d777d0c8003994c1001abc23186f18780c02a11b2b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L393","status":"pending"},{"id":"J319","status":"ready"},{"id":"Y240","status":"pending"},{"id":"M843","status":"pending"},{"id":"I161","status":"ready"}]
Checked answer
{"ids": ["I161", "J319"]}
Seed

wrong_schema_or_value

{"ids": ["I161", "J319", "M843", "L393"]}
29 tokens · 0.801s
Clean · 1 GPU

Correct

{"ids":["I161","J319"]}
16 tokens · 0.450s
Damaged · 1 GPU

Correct

{"ids":["I161","J319"]}
16 tokens · 0.448s
Clean · 2 GPUs

Correct

{"ids":["I161","J319"]}
16 tokens · 0.451s
227 · total · wording 0 · samed867d6ccaf2dc788c21f108bb03e45c097ab2993d1416cae7839a9ea1eb91215
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y128","units":7,"price":1},{"id":"C757","units":18,"price":5},{"id":"F440","units":14,"price":15},{"id":"B585","units":18,"price":4},{"id":"L919","units":16,"price":16}]
Checked answer
{"total": 635}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":441}
8 tokens · 0.225s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.223s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.226s
228 · sort · wording 0 · gain4a5bef3239c65fb3d41318c59816b5350f73e50cd2c2434d1f6a733126653029
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z383","priority":1},{"id":"N849","priority":8},{"id":"D362","priority":4},{"id":"B186","priority":7},{"id":"F590","priority":1}]
Checked answer
{"ids": ["N849", "B186"]}
Seed

wrong_schema_or_value

{"ids": ["N849", "D362", "B186", "F590", "Z383"]}
35 tokens · 0.957s
Clean · 1 GPU

Correct

{"ids":["N849","B186"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["N849","B186"]}
16 tokens · 0.451s
Clean · 2 GPUs

Correct

{"ids":["N849","B186"]}
16 tokens · 0.446s
229 · lookup · wording 1 · same10702ab41abb1aad7f337c2365a9a9cfd862d8ce7250ca806bd1c5d7a748c843
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Q639 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"I598","city":"Lima","units":16},{"id":"O649","city":"Suva","units":6},{"id":"Q639","city":"Riga","units":11},{"id":"Z225","city":"Oslo","units":14},{"id":"S232","city":"Oslo","units":18}]
Checked answer
{"city": "Riga", "units": 11}
Seed

Correct

{"city":"Riga","units":11}
12 tokens · 0.332s
Clean · 1 GPU

Correct

{"city":"Riga","units":11}
12 tokens · 0.339s
Damaged · 1 GPU

Correct

{"city":"Riga","units":11}
12 tokens · 0.336s
Clean · 2 GPUs

Correct

{"city":"Riga","units":11}
12 tokens · 0.334s
230 · filter · wording 1 · gaincb9b6b4ae01a167f71d119af8aa4fd469ab53996a5012206ef1ddb6c95d5f89e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"F396","status":"pending"},{"id":"Z208","status":"ready"},{"id":"P860","status":"pending"},{"id":"K674","status":"ready"},{"id":"U383","status":"ready"}]
Checked answer
{"ids": ["K674", "U383", "Z208"]}
Seed

wrong_schema_or_value

{
  "ids": ["F396", "Z208", "P860", "K674", "U383"]
}
38 tokens · 1.040s
Clean · 1 GPU

Correct

{"ids":["K674","U383","Z208"]}
21 tokens · 0.586s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["K674","F396","U383"]}
21 tokens · 0.580s
Clean · 2 GPUs

Correct

{"ids":["K674","U383","Z208"]}
21 tokens · 0.577s
231 · total · wording 1 · same71e828e75d43cf732f7cac66401586a94cb1140bcdfd8a3a44a2fd952f83d64e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"J903","units":6,"price":11},{"id":"P320","units":11,"price":19},{"id":"V832","units":15,"price":6},{"id":"P774","units":7,"price":15},{"id":"J279","units":9,"price":12}]
Checked answer
{"total": 578}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":351}
8 tokens · 0.226s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":371}
8 tokens · 0.224s
232 · sort · wording 1 · same7f6ed0b24b4a54a50e5fe468b0605275013d517c64dac1e7ab52dc9c701043a3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"A661","priority":3},{"id":"N349","priority":5},{"id":"Y802","priority":7},{"id":"N689","priority":6},{"id":"J873","priority":2}]
Checked answer
{"ids": ["Y802", "N689"]}
Seed

invalid_json_object

[{"id":"A661","priority":3},{"id":"N689","priority":6},{"id":"Y802","priority":7}]
38 tokens · 1.039s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["Y802","N349"]}
16 tokens · 0.446s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["Y802","N349"]}
16 tokens · 0.441s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["Y802","N349"]}
16 tokens · 0.444s
233 · lookup · wording 2 · same53e10d168d90c5156ed0b4ea146e6039fe89a12be2289f9eb19b4a00f2295e59
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to N442? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"N442","city":"Lima","units":0},{"id":"T545","city":"Bern","units":0},{"id":"N222","city":"Lima","units":13},{"id":"O454","city":"Riga","units":14},{"id":"I395","city":"Bern","units":9}]
Checked answer
{"city": "Lima", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.415s
Clean · 1 GPU

wrong_schema_or_value

{"city":null,"units":null}
10 tokens · 0.282s
Damaged · 1 GPU

wrong_schema_or_value

{"city":null,"units":null}
10 tokens · 0.281s
Clean · 2 GPUs

wrong_schema_or_value

{"city":null,"units":null}
10 tokens · 0.283s
234 · filter · wording 2 · gainac64040c02d1341f0721d10aafaa4faec5e620fc59d77cd68bfd23cde728bbc3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"P528","status":"ready"},{"id":"I154","status":"closed"},{"id":"T972","status":"closed"},{"id":"E978","status":"closed"},{"id":"O324","status":"closed"}]
Checked answer
{"ids": ["P528"]}
Seed

invalid_json_object

[{"id":"P528"},{"id":"I154"},{"id":"T972"},{"id":"E978"},{"id":"O324"}]
43 tokens · 1.179s
Clean · 1 GPU

Correct

{"ids":["P528"]}
11 tokens · 0.306s
Damaged · 1 GPU

Correct

{"ids":["P528"]}
11 tokens · 0.305s
Clean · 2 GPUs

Correct

{"ids":["P528"]}
11 tokens · 0.310s
235 · total · wording 2 · sameb3ef923b64f95fc8c7fe73a910ac50bbac92ea22f9f15bcc82c27e328c243dd0
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y498","units":12,"price":14},{"id":"N361","units":2,"price":16},{"id":"B407","units":15,"price":17},{"id":"F599","units":14,"price":3},{"id":"G422","units":18,"price":5}]
Checked answer
{"total": 587}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.253s
Clean · 1 GPU

wrong_schema_or_value

{"total":431}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.224s
236 · sort · wording 2 · gaindc54792c5c50a26570fb52f3900da555c7e8d890c8c4dc8939182ccb2b255dfc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"M598","priority":4},{"id":"T597","priority":4},{"id":"E473","priority":1},{"id":"U978","priority":2},{"id":"J233","priority":1}]
Checked answer
{"ids": ["M598", "T597"]}
Seed

wrong_schema_or_value

{"ids": ["M598", "T597", "E473", "U978", "J233"]}
35 tokens · 0.960s
Clean · 1 GPU

Correct

{"ids":["M598","T597"]}
16 tokens · 0.442s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["T597","M598"]}
16 tokens · 0.444s
Clean · 2 GPUs

Correct

{"ids":["M598","T597"]}
16 tokens · 0.447s
237 · lookup · wording 3 · same07cb2078f89f2e2c1233834cffd2ec5f0fade188b10b965fd23746da98472dfb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T987. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T987","city":"Perth","units":7},{"id":"G787","city":"Oslo","units":6},{"id":"Y276","city":"Accra","units":5},{"id":"D153","city":"Riga","units":17},{"id":"Y564","city":"Accra","units":1}]
Checked answer
{"city": "Perth", "units": 7}
Seed

Correct

{"city": "Perth", "units": 7}
14 tokens · 0.390s
Clean · 1 GPU

Correct

{"city":"Perth","units":7}
11 tokens · 0.311s
Damaged · 1 GPU

Correct

{"city":"Perth","units":7}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"city":"Perth","units":7}
11 tokens · 0.305s
238 · filter · wording 3 · gainadbebb48edcadfff15737241ae78abeb0d4a2b59f1213bd51c5346cd1ca46bc8
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"X215","status":"pending"},{"id":"A743","status":"closed"},{"id":"I329","status":"closed"},{"id":"A901","status":"ready"},{"id":"N600","status":"pending"}]
Checked answer
{"ids": ["A901"]}
Seed

wrong_schema_or_value

{"ids": ["A743", "A901", "I329", "N600", "X215"]}
35 tokens · 0.959s
Clean · 1 GPU

Correct

{"ids":["A901"]}
11 tokens · 0.309s
Damaged · 1 GPU

Correct

{"ids":["A901"]}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"ids":["A901"]}
11 tokens · 0.305s
239 · total · wording 3 · same192f781a76a3a05ce28ac31f57533c1a870a48f567ceec8b56f33077716890d4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"U599","units":6,"price":15},{"id":"V597","units":9,"price":7},{"id":"G303","units":4,"price":9},{"id":"P497","units":8,"price":7},{"id":"G404","units":12,"price":7}]
Checked answer
{"total": 329}
Seed

wrong_schema_or_value

{
  "total": 108
}
12 tokens · 0.331s
Clean · 1 GPU

wrong_schema_or_value

{"total":310}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.229s
Clean · 2 GPUs

wrong_schema_or_value

{"total":241}
8 tokens · 0.223s
240 · sort · wording 3 · same9ae673f5a58d60df371af041d321e0f63e7c610f3e7d2001d47abd185cb17dcb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"N929","priority":6},{"id":"F236","priority":3},{"id":"F146","priority":9},{"id":"P238","priority":5},{"id":"N844","priority":4}]
Checked answer
{"ids": ["F146", "N929"]}
Seed

invalid_json_object

[{"id":"N929","priority":6},{"id":"F236","priority":3},{"id":"F146","priority":9},{"id":"P238","priority":5},{"id":"N844","priority":4}]
62 tokens · 1.694s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["F146","N844"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["F146","N929"]}
16 tokens · 0.446s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["F146","N844"]}
16 tokens · 0.439s
241 · lookup · wording 0 · gain36340265978b50e3369fd1cf4e06cb3540a855f271921e9a1aefedd5449c4530
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"I405","city":"Lima","units":15},{"id":"A276","city":"Oslo","units":5},{"id":"B480","city":"Perth","units":10},{"id":"K853","city":"Bern","units":11},{"id":"T371","city":"Oslo","units":17}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Lima",
  "units": 15
}
19 tokens · 0.525s
Clean · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.283s
Damaged · 1 GPU

Correct

{"city":null,"units":null}
10 tokens · 0.282s
Clean · 2 GPUs

Correct

{"city":null,"units":null}
10 tokens · 0.277s
242 · filter · wording 0 · gain59ba398e1a5b5131d4f8214714453d899d8bbab88ccefbddf2e9700e2406fcf6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"R502","status":"ready"},{"id":"O411","status":"pending"},{"id":"M868","status":"pending"},{"id":"V659","status":"ready"},{"id":"B963","status":"closed"}]
Checked answer
{"ids": ["R502", "V659"]}
Seed

wrong_schema_or_value

{"ids": ["O411", "M868", "R502", "V659", "B963"]}
35 tokens · 0.956s
Clean · 1 GPU

Correct

{"ids":["R502","V659"]}
16 tokens · 0.440s
Damaged · 1 GPU

Correct

{"ids":["R502","V659"]}
16 tokens · 0.445s
Clean · 2 GPUs

Correct

{"ids":["R502","V659"]}
16 tokens · 0.439s
243 · total · wording 0 · same222dd4c8f95dbb270c37dbdd0652a2c67e939e67d1e7019a95a388731fe45b8f
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y132","units":1,"price":16},{"id":"K147","units":5,"price":7},{"id":"T950","units":18,"price":13},{"id":"G567","units":12,"price":15},{"id":"Q798","units":16,"price":17}]
Checked answer
{"total": 737}
Seed

invalid_json_object

{"total": 16 + 7 + 13 + 15 + 17 = 69}
27 tokens · 0.743s
Clean · 1 GPU

wrong_schema_or_value

{"total":471}
8 tokens · 0.223s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.224s
Clean · 2 GPUs

wrong_schema_or_value

{"total":471}
8 tokens · 0.224s
244 · sort · wording 0 · samebbc901163c4c5b4b0ac6dcdbe25a3d78a5e1a2cdd36f3b1161262d6dc694aae5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E972","priority":1},{"id":"B525","priority":3},{"id":"F536","priority":4},{"id":"J276","priority":3},{"id":"Y324","priority":4}]
Checked answer
{"ids": ["F536", "Y324"]}
Seed

wrong_schema_or_value

{"ids": ["B525", "F536", "J276", "Y324", "E972"]}
35 tokens · 0.960s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["F536","B525"]}
16 tokens · 0.444s
Damaged · 1 GPU

Correct

{"ids":["F536","Y324"]}
16 tokens · 0.447s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["F536","B525"]}
16 tokens · 0.445s
245 · lookup · wording 1 · same258961907856111b2ca1a2c061ad1d856f2a79cba0afd83540b5c058d708f4d3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id K954 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"Y723","city":"Bern","units":0},{"id":"W107","city":"Lima","units":18},{"id":"O318","city":"Kyoto","units":4},{"id":"M719","city":"Riga","units":1},{"id":"K954","city":"Perth","units":0}]
Checked answer
{"city": "Perth", "units": 0}
Seed

Correct

{"city":"Perth","units":0}
11 tokens · 0.307s
Clean · 1 GPU

Correct

{"city":"Perth","units":0}
11 tokens · 0.308s
Damaged · 1 GPU

Correct

{"city":"Perth","units":0}
11 tokens · 0.312s
Clean · 2 GPUs

Correct

{"city":"Perth","units":0}
11 tokens · 0.311s
246 · filter · wording 1 · gain468a09ed2e76091bd37be501abaab65886f7089e803bdac8ac629a9f8b1a0d2e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"R114","status":"pending"},{"id":"C431","status":"pending"},{"id":"V835","status":"pending"},{"id":"H469","status":"pending"},{"id":"E964","status":"ready"}]
Checked answer
{"ids": ["E964"]}
Seed

wrong_schema_or_value

{
  "ids": ["E964", "H469", "R114", "V835", "C431"]
}
38 tokens · 1.044s
Clean · 1 GPU

Correct

{"ids":["E964"]}
11 tokens · 0.311s
Damaged · 1 GPU

Correct

{"ids":["E964"]}
11 tokens · 0.312s
Clean · 2 GPUs

Correct

{"ids":["E964"]}
11 tokens · 0.306s
247 · total · wording 1 · same99a94e5099e4ce61a970945ba5f17c87af680306cd92689711198ab0d453cc0b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"B844","units":8,"price":10},{"id":"B439","units":8,"price":11},{"id":"T402","units":12,"price":15},{"id":"G494","units":3,"price":14},{"id":"P260","units":7,"price":11}]
Checked answer
{"total": 467}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens · 0.252s
Clean · 1 GPU

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.229s
Clean · 2 GPUs

wrong_schema_or_value

{"total":341}
8 tokens · 0.227s
248 · sort · wording 1 · gainc98786c35beb946de940956476cf05f07529a569505803687e8e07fe375662c9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"F270","priority":3},{"id":"I528","priority":7},{"id":"L682","priority":3},{"id":"E942","priority":3},{"id":"B496","priority":5}]
Checked answer
{"ids": ["I528", "B496"]}
Seed

invalid_json_object

[{"id":"F270","priority":3},{"id":"L682","priority":3},{"id":"E942","priority":3},{"id":"B496","priority":5}]
50 tokens · 1.371s
Clean · 1 GPU

Correct

{"ids":["I528","B496"]}
16 tokens · 0.447s
Damaged · 1 GPU

Correct

{"ids":["I528","B496"]}
16 tokens · 0.450s
Clean · 2 GPUs

Correct

{"ids":["I528","B496"]}
16 tokens · 0.441s
249 · lookup · wording 2 · gain0216df24c617e5217cbd0c00e3de959964b067599f1fc35fb4a0e8914b755443
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M688? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Y501","city":"Perth","units":1},{"id":"X352","city":"Kyoto","units":12},{"id":"U279","city":"Oslo","units":13},{"id":"M688","city":"Perth","units":8},{"id":"G212","city":"Suva","units":1}]
Checked answer
{"city": "Perth", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens · 0.416s
Clean · 1 GPU

Correct

{"city":"Perth","units":8}
11 tokens · 0.307s
Damaged · 1 GPU

Correct

{"city":"Perth","units":8}
11 tokens · 0.309s
Clean · 2 GPUs

Correct

{"city":"Perth","units":8}
11 tokens · 0.305s
250 · filter · wording 2 · gainc100588d0769712e8fd7cb32179acfe1e3d844bc83f8ec6469bcf8551c324599
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S900","status":"pending"},{"id":"S359","status":"ready"},{"id":"I490","status":"closed"},{"id":"D654","status":"pending"},{"id":"Z860","status":"pending"}]
Checked answer
{"ids": ["S359"]}
Seed

invalid_json_object

[{"id":"S359","status":"ready"},{"id":"I490","status":"closed"},{"id":"D654","status":"pending"},{"id":"Z860","status":"pending"}]
51 tokens · 1.396s
Clean · 1 GPU

Correct

{"ids":["S359"]}
11 tokens · 0.310s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["D654","S359"]}
16 tokens · 0.447s
Clean · 2 GPUs

Correct

{"ids":["S359"]}
11 tokens · 0.306s
251 · total · wording 2 · same33f86289b7f0236da44480671ce7f28777099151f6e1f6056f712677de4669c9
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I134","units":3,"price":11},{"id":"W560","units":15,"price":15},{"id":"U101","units":2,"price":8},{"id":"X430","units":8,"price":10},{"id":"I360","units":14,"price":17}]
Checked answer
{"total": 592}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens · 0.251s
Clean · 1 GPU

wrong_schema_or_value

{"total":331}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":432}
8 tokens · 0.225s
Clean · 2 GPUs

wrong_schema_or_value

{"total":431}
8 tokens · 0.225s
252 · sort · wording 2 · same7b3407cca579634ba7a57268524fa3ce6ac7d5702538a4137f5aa62bcdb272bc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"A213","priority":5},{"id":"T122","priority":7},{"id":"E628","priority":9},{"id":"J981","priority":7},{"id":"B341","priority":5}]
Checked answer
{"ids": ["E628", "J981"]}
Seed

wrong_schema_or_value

{"ids": ["A213", "T122", "E628", "J981", "B341"]}
35 tokens · 0.956s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["E628","T122"]}
16 tokens · 0.449s
Damaged · 1 GPU

Correct

{"ids":["E628","J981"]}
16 tokens · 0.448s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["E628","T122"]}
16 tokens · 0.442s
253 · lookup · wording 3 · samef46c4a7b7f29aa62e33358f48fefd79ab732f0081fd388951f9f8e003a07d3f7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=H187. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"H187","city":"Bern","units":1},{"id":"T805","city":"Lima","units":16},{"id":"N350","city":"Lima","units":8},{"id":"X625","city":"Perth","units":13},{"id":"Z263","city":"Kyoto","units":8}]
Checked answer
{"city": "Bern", "units": 1}
Seed

Correct

{"city": "Bern", "units": 1}
13 tokens · 0.360s
Clean · 1 GPU

Correct

{"city":"Bern","units":1}
10 tokens · 0.284s
Damaged · 1 GPU

Correct

{"city":"Bern","units":1}
10 tokens · 0.279s
Clean · 2 GPUs

Correct

{"city":"Bern","units":1}
10 tokens · 0.283s
254 · filter · wording 3 · same2ee53388bcfa81658d87bae281f5c7cf0a0b6860cb1b3bbbc384610e1f5972a9
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"M560","status":"pending"},{"id":"L509","status":"pending"},{"id":"R446","status":"pending"},{"id":"M596","status":"ready"},{"id":"N314","status":"ready"}]
Checked answer
{"ids": ["M596", "N314"]}
Seed

wrong_schema_or_value

{"ids": ["M560", "L509", "R446", "M596", "N314"]}
35 tokens · 0.960s
Clean · 1 GPU

wrong_schema_or_value

{"ids":["N314","M596"]}
16 tokens · 0.447s
Damaged · 1 GPU

wrong_schema_or_value

{"ids":["N314","M596"]}
16 tokens · 0.440s
Clean · 2 GPUs

wrong_schema_or_value

{"ids":["N314","M596"]}
16 tokens · 0.447s
255 · total · wording 3 · samefdc78f724d8efc910f6c8395905dd255091dedef5653fa1f3f312bfd929f4be3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"U270","units":7,"price":4},{"id":"S189","units":18,"price":7},{"id":"H792","units":19,"price":19},{"id":"T290","units":13,"price":19},{"id":"J625","units":2,"price":15}]
Checked answer
{"total": 792}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens · 0.332s
Clean · 1 GPU

wrong_schema_or_value

{"total":471}
8 tokens · 0.228s
Damaged · 1 GPU

wrong_schema_or_value

{"total":342}
8 tokens · 0.222s
Clean · 2 GPUs

wrong_schema_or_value

{"total":531}
8 tokens · 0.229s
256 · sort · wording 3 · gainb46692c26747684a6b4e5d72df11b9db6d54bb2ec5a8370bb74c93b758d58dc4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"L799","priority":6},{"id":"G601","priority":2},{"id":"Z184","priority":3},{"id":"V411","priority":6},{"id":"D563","priority":8}]
Checked answer
{"ids": ["D563", "L799"]}
Seed

invalid_json_object

[{"id":"L799"},{"id":"G601"},{"id":"Z184"},{"id":"V411"},{"id":"D563"}]
43 tokens · 1.172s
Clean · 1 GPU

Correct

{"ids":["D563","L799"]}
16 tokens · 0.440s
Damaged · 1 GPU

Correct

{"ids":["D563","L799"]}
16 tokens · 0.440s
Clean · 2 GPUs

Correct

{"ids":["D563","L799"]}
16 tokens · 0.445s

LZ · Apache-2.0 · Full raw evaluations, predetermined final candidates, no serving promotion.