NeuroShard / Research evidence / 13 September 2026

One model, four GPU workers

Can workers communicate less while preserving learning? Three predetermined training arms use the same 1.7B seed, global document schedule and 128 updates. Local windows exchange endpoint deltas every 16 updates and keep each worker’s Adam state. The model remains fixed in size.

Quality contract: failed. Efficiency contract: passed. The primary local-window candidate changes exact accuracy by +44.14 percentage points against the seed. Its approximate paired 99% lower change bound against DDP is -9.05 points. The exposed retention upper loss-change bound is +0.108140 nats.

This is one operated experiment on four generated task families and previously exposed public retention probes. It does not establish broad assistant quality, independent ownership, permissionless GPU verification, continual improvement or model growth. It issues no NEURO and does not promote a native serving checkpoint.

Measured task accuracy, active training time, transmitted bytes and serving throughput
ModelExact answersValid JSONMean retention change
Seed105 / 1024 (10.25%)776
1 GPU · AdamW641 / 1024 (62.60%)1009-0.002428
4 GPUs · DDP613 / 1024 (59.86%)1024-0.002885
4 GPUs · local windows557 / 1024 (54.39%)965+0.083294

Training and recovery

MethodActive timeLoop with checkpointsWhole-host bytes sent
1 GPU · AdamW208.09 s266.12 s0.000 GB
4 GPUs · DDP1979.26 s2048.37 s5330.663 GB
4 GPUs · local windows250.08 s494.51 s331.778 GB

Active time includes local updates and synchronization; startup, checkpoints and whole-instance allocation are separate. The 2.5 Gbit/s baseline and burst bandwidth of these AWS instances affected the timings. The methods ran in a fixed order, and checkpoint pauses can replenish network credits. This is an observed workload comparison, not an isolated estimate of algorithmic speedup.

After the rank-one process was killed following step 75, the group restarted at step 64. Exact final model, outer momentum, every rank’s Adam/RNG contents and the numerical trajectories agree with the uninterrupted run: True. This is coordinated recovery of the same operated membership.

Compressed-gradient feasibility

A separate training-only probe completes eight dense warmup updates and four rank-32 PowerSGD updates on two GPUs, with both workers agreeing. The first attempt runs out of GPU memory; moving dense error feedback to CPU between completed bucket reductions fixes that failure. Peak allocated GPU memory is 36.89 GiB. Small-model CPU and CUDA checks preserve the resident control’s numerical state exactly.

The four compressed updates show 45.14× logical compression, excluding dense warmup and transport overhead. This is not a full-length quality or sustained-throughput result, and no complete compression-state recovery is implemented. Probe measurements · Failed resident attempt · PowerSGD paper.

Serving the predetermined local-window model

ProvidersCompletedRequests/secondMedian latency
single64 / 643.0081.285 s
four64 / 649.0250.321 s
failure64 / 646.7130.450 s

Each phase uses the same 64 development tasks and client concurrency four. The outage phase advertises four endpoints with one stopped. This private fixed-task fixture has no native billing or general chat API.

Methods and interpretation · Recomputed metrics · Frozen inputs · Candidate commitments · Checksums · Small evidence archive

Predetermined weights: single GPU · four-worker DDP · four-worker local windows. All outcomes are retained, including failed comparisons. Downloads contain model/tokenizer files and attribution; full optimizer/outer states are separately backed up by the operator.

All 1,024 final-test tasks

0001 · lookup · wording 0 · gain70a9e42fda88d27fee26462ed4a84fc7f9e87d395fc6804a7b495d6bfae182bb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H142","city":"Perth","units":6},{"id":"C241","city":"Lima","units":17},{"id":"A538","city":"Bern","units":11},{"id":"D771","city":"Riga","units":3},{"id":"A655","city":"Riga","units":12}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Perth","units":6},{"city":"Lima","units":17},{"city":"Bern","units":11},{"city":"Riga","units":3},{"city":"Riga","units":12}]
54 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0002 · filter · wording 0 · gaineaf5043449b7f2bc2430a3ff481a28ca78b84e4c08d57b8e7014c4e89ffb2b4b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P980","status":"closed"},{"id":"N228","status":"pending"},{"id":"V371","status":"closed"},{"id":"H680","status":"ready"},{"id":"S551","status":"closed"}]
Checked answer
{"ids": ["H680"]}
Seed

wrong_schema_or_value

{"ids": ["H680", "S551", "P980", "V371", "N228"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["H680"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["H680"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["H680"]}
11 tokens
0003 · total · wording 0 · samee11bf8b7eae04f49abb9db20d8cb3e7c89d95a485974e76ba4f5661d8841b340
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"N657","units":7,"price":5},{"id":"P766","units":0,"price":10},{"id":"F259","units":19,"price":9},{"id":"V629","units":7,"price":3},{"id":"A709","units":14,"price":9}]
Checked answer
{"total": 353}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0004 · sort · wording 0 · gainee94fd87544b71cd9da51f57fd9b464c4fe25a860607a50bc4eafbff7787b516
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"D699","priority":8},{"id":"Q971","priority":4},{"id":"W300","priority":5},{"id":"F657","priority":7},{"id":"Q559","priority":1}]
Checked answer
{"ids": ["D699", "F657"]}
Seed

wrong_schema_or_value

{"ids": ["Q559", "W300", "D699", "Q971", "F657"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D699","F657"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D699","F657"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D699","F657"]}
16 tokens
0005 · lookup · wording 1 · gainb0b79edcdcf522b48d6bd701c5b5c0d5f3793f3baf57af391c8642264e224e41
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M852 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M534","city":"Lima","units":9},{"id":"M852","city":"Riga","units":11},{"id":"G527","city":"Oslo","units":18},{"id":"C961","city":"Lima","units":7},{"id":"N447","city":"Riga","units":19}]
Checked answer
{"city": "Riga", "units": 11}
Seed

wrong_schema_or_value

{"city":"Lima","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":11}
12 tokens
0006 · filter · wording 1 · gainc8c2e7d9d2eb534a8d9a27de557170a6f2f8bb39b0ddd70bc72675d4d3a6a64b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"Y439","status":"pending"},{"id":"H889","status":"pending"},{"id":"S402","status":"closed"},{"id":"N641","status":"ready"},{"id":"N993","status":"pending"}]
Checked answer
{"ids": ["N641"]}
Seed

wrong_schema_or_value

{
  "ids": ["N641", "N993", "S402"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":["N641"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["N641"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["N641"]}
11 tokens
0007 · total · wording 1 · sameee37a53f2810d311e8bdc34731da358cc75a218d303f16d0116beb7b9122fc73
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"X440","units":10,"price":17},{"id":"G152","units":1,"price":5},{"id":"Y376","units":3,"price":1},{"id":"T175","units":17,"price":10},{"id":"M868","units":13,"price":5}]
Checked answer
{"total": 413}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0008 · sort · wording 1 · gain3bc62a5652fc638b1dc7ca0a863816debdb65a26793ee65858432c243a156ba4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Q486","priority":1},{"id":"Q910","priority":6},{"id":"N223","priority":3},{"id":"E772","priority":9},{"id":"R407","priority":2}]
Checked answer
{"ids": ["E772", "Q910"]}
Seed

invalid_json_object

[{"id":"Q486","priority":1},{"id":"Q910","priority":6},{"id":"N223","priority":3},{"id":"E772","priority":9},{"id":"R407","priority":2}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["E772","Q910"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E772","N223"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E772","Q910"]}
16 tokens
0009 · lookup · wording 2 · gaina3edd368f6b9d9e3503cb7249ccc39f9b5754df47d9d1c80809fed528bc6b8c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to L421? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"X495","city":"Perth","units":17},{"id":"W559","city":"Suva","units":7},{"id":"L828","city":"Oslo","units":7},{"id":"L421","city":"Accra","units":9},{"id":"G770","city":"Suva","units":12}]
Checked answer
{"city": "Accra", "units": 9}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":9}
11 tokens
0010 · filter · wording 2 · samebd986df90ffd1d05f0a899692619d48739cf62a869d29960f268411b74d59464
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"M856","status":"ready"},{"id":"V540","status":"ready"},{"id":"R519","status":"pending"},{"id":"G858","status":"closed"},{"id":"L747","status":"ready"}]
Checked answer
{"ids": ["L747", "M856", "V540"]}
Seed

invalid_json_object

[{"id":"M856","status":"ready"},{"id":"V540","status":"ready"},{"id":"R519","status":"pending"},{"id":"G858","status":"closed"},{"id":"L747","status":"ready"}]
63 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["M856","L747","V540"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["L747","M856","V540"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["M856","L747","V540"]]
21 tokens
0011 · total · wording 2 · samea2f1ee093dd74d0f751b54fbba3d5e6571ee26a5ba623fbde7eb70d8c6594dd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V967","units":15,"price":10},{"id":"K750","units":12,"price":17},{"id":"G558","units":14,"price":4},{"id":"R162","units":0,"price":15},{"id":"C130","units":5,"price":18}]
Checked answer
{"total": 500}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0012 · sort · wording 2 · gainb19f828571321721628b5ca00c2220d4f8ce00d320187af166c64d90d4fd68a6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"R250","priority":2},{"id":"I161","priority":4},{"id":"S243","priority":5},{"id":"K361","priority":6},{"id":"Z965","priority":2}]
Checked answer
{"ids": ["K361", "S243"]}
Seed

wrong_schema_or_value

{"ids": ["R250", "I161", "S243", "K361", "Z965"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["K361","S243"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K361","S243"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K361","S243"]}
16 tokens
0013 · lookup · wording 3 · samef8cf3e6b3041f523c7aed9ba307f7a2066711b38a2ba94f7838bdc8fdd11a1e1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=P975. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P975","city":"Riga","units":9},{"id":"A231","city":"Suva","units":2},{"id":"X246","city":"Oslo","units":1},{"id":"X349","city":"Bern","units":18},{"id":"Q313","city":"Perth","units":12}]
Checked answer
{"city": "Riga", "units": 9}
Seed

Correct

{"city": "Riga", "units": 9}
14 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":9}
11 tokens
0014 · filter · wording 3 · same44bec707fec6c8515587eaf3a1b208820372798c2a4f3de0f3f409b4eecf3633
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"T936","status":"ready"},{"id":"N656","status":"pending"},{"id":"Y463","status":"pending"},{"id":"N888","status":"ready"},{"id":"Q934","status":"closed"}]
Checked answer
{"ids": ["N888", "T936"]}
Seed

wrong_schema_or_value

{"ids": ["N656", "N888", "T936", "Q934"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["T936","N888"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N656","T936"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N656","T936"]}
16 tokens
0015 · total · wording 3 · same13d2f13a7d3f1e1ff99a62c103d7d07f3037b605d346c19f7993b7e4cff497b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"G937","units":9,"price":11},{"id":"P173","units":4,"price":5},{"id":"V258","units":9,"price":11},{"id":"X904","units":1,"price":3},{"id":"O497","units":11,"price":3}]
Checked answer
{"total": 254}
Seed

wrong_schema_or_value

{
  "total": 110
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0016 · sort · wording 3 · same8194351c6f04092b975fac288f2236f56ea00d57a2dbd5ada6220f76c81e12f3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"A517","priority":4},{"id":"S453","priority":2},{"id":"W209","priority":8},{"id":"X601","priority":4},{"id":"K545","priority":1}]
Checked answer
{"ids": ["W209", "A517"]}
Seed

invalid_json_object

[{"id":"W209","priority":8},{"id":"X601","priority":4},{"id":"A517","priority":4},{"id":"S453","priority":2},{"id":"K545","priority":1}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["W209","X601"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["W209","A517"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["W209","X601"]}
16 tokens
0017 · lookup · wording 0 · same1b38dc230a557d9afad471054e50e17426df543432aa0a886eddb7350ab623eb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record F749. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"I822","city":"Lima","units":0},{"id":"C390","city":"Perth","units":4},{"id":"R752","city":"Accra","units":4},{"id":"F749","city":"Lima","units":8},{"id":"Z493","city":"Kyoto","units":8}]
Checked answer
{"city": "Lima", "units": 8}
Seed

Correct

{
  "city": "Lima",
  "units": 8
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":8}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":8}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":8}
11 tokens
0018 · filter · wording 0 · same10562fecfdc5e960cf7677b65d0f62643242d5a4747d9f58a4f54e80aa5f66f3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"I200","status":"pending"},{"id":"E734","status":"ready"},{"id":"M591","status":"ready"},{"id":"S308","status":"ready"},{"id":"B937","status":"pending"}]
Checked answer
{"ids": ["E734", "M591", "S308"]}
Seed

wrong_schema_or_value

{"ids": ["E734", "M591", "S308", "B937", "I200"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E734","M591","S308"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["E734","M591","S308"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E734","S308","M591"]}
21 tokens
0019 · total · wording 0 · same321291f7ceae82a28d04ac4af4304e86ed2afd5918f2683afb93ed1d6e65b1e6
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y939","units":5,"price":8},{"id":"D555","units":0,"price":2},{"id":"L796","units":12,"price":16},{"id":"Y751","units":17,"price":3},{"id":"I847","units":4,"price":13}]
Checked answer
{"total": 335}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0020 · sort · wording 0 · gain9f292ea9844a522e9d0bc95e65b8e1291ae627afcafb0b22d98e6e1110a8227f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"N789","priority":1},{"id":"O537","priority":4},{"id":"J807","priority":9},{"id":"T576","priority":9},{"id":"N763","priority":5}]
Checked answer
{"ids": ["J807", "T576"]}
Seed

wrong_schema_or_value

{"ids": ["N789", "O537", "J807", "T576", "N763"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J807","T576"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J807","T576"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J807","T576"]}
16 tokens
0021 · lookup · wording 1 · gainc3d2a3903f4fa8a9a51f11652b4794d69480d11055c830f7e2652b77a7b3415d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"L947","city":"Lima","units":3},{"id":"G117","city":"Bern","units":4},{"id":"J482","city":"Bern","units":15},{"id":"J286","city":"Accra","units":13},{"id":"R766","city":"Oslo","units":5}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Lima","units":3}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0022 · filter · wording 1 · gain890b786387625988d3433ea8389bc589d50f83b2d497d51613a48908dd7a262c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"J289","status":"ready"},{"id":"T577","status":"closed"},{"id":"Y892","status":"ready"},{"id":"E794","status":"closed"},{"id":"G838","status":"pending"}]
Checked answer
{"ids": ["J289", "Y892"]}
Seed

wrong_schema_or_value

{
  "ids": ["J289", "Y892", "G838"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":["J289","Y892"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J289","Y892"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J289","Y892"]}
16 tokens
0023 · total · wording 1 · samef4535f849ecce46fc0bcc057671e3f8f6895d320399fe69a281292345e668f3c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"P890","units":10,"price":6},{"id":"K773","units":7,"price":7},{"id":"T762","units":2,"price":12},{"id":"D645","units":3,"price":10},{"id":"B658","units":0,"price":7}]
Checked answer
{"total": 163}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":268}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":268}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0024 · sort · wording 1 · gain83192386847499ccffade2fa679a3260698ec81886948198d5de5423fbbf8bcd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"X824","priority":1},{"id":"D952","priority":6},{"id":"E938","priority":5},{"id":"A238","priority":1},{"id":"L597","priority":6}]
Checked answer
{"ids": ["D952", "L597"]}
Seed

invalid_json_object

[{"id":"X824","priority":1},{"id":"A238","priority":1},{"id":"L597","priority":6},{"id":"D952","priority":6},{"id":"E938","priority":5}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["D952","L597"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D952","L597"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D952","L597"]}
16 tokens
0025 · lookup · wording 2 · gain7df7421d676f0f06cdae8b685f93180a259c295dbbc33d76f8dc09f261efa8d2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to L130? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"A304","city":"Riga","units":11},{"id":"I689","city":"Bern","units":2},{"id":"T384","city":"Riga","units":14},{"id":"L130","city":"Bern","units":14},{"id":"H902","city":"Riga","units":6}]
Checked answer
{"city": "Bern", "units": 14}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":14}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":14}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":14}
11 tokens
0026 · filter · wording 2 · same3baa7915880390e8564c1ad8d98408b1ce9b97279a2420ffd440c3de65f1581d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"X614","status":"pending"},{"id":"U671","status":"closed"},{"id":"X289","status":"ready"},{"id":"G526","status":"ready"},{"id":"H926","status":"closed"}]
Checked answer
{"ids": ["G526", "X289"]}
Seed

invalid_json_object

[{"id":"X289","status":"ready"},{"id":"X614","status":"pending"},{"id":"U671","status":"closed"},{"id":"G526","status":"ready"},{"id":"H926","status":"closed"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["G526","X289"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G526","X289"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G526","X289"]]
16 tokens
0027 · total · wording 2 · samecd8d02d9fe9aaeb064d5c54c8cbd4af6b6493854d7548151a2ef076e839c92ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"F529","units":16,"price":17},{"id":"W196","units":18,"price":8},{"id":"U561","units":4,"price":13},{"id":"L688","units":13,"price":7},{"id":"P900","units":9,"price":12}]
Checked answer
{"total": 667}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0028 · sort · wording 2 · gain588ff8061f6ca9b8d45cf505fd06daf6d5b66f8df060736b4bdd26584b1b951e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"A573","priority":6},{"id":"Y639","priority":1},{"id":"J161","priority":7},{"id":"Z593","priority":6},{"id":"C580","priority":7}]
Checked answer
{"ids": ["C580", "J161"]}
Seed

wrong_schema_or_value

{"ids": ["A573", "Y639", "J161", "Z593", "C580"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C580","J161"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C580","J161"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C580","J161"]}
16 tokens
0029 · lookup · wording 3 · samef7453d30818a608800833971901cce1e5c86915ef9258211e1ae9612dcd429ec
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=X493. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"K997","city":"Bern","units":5},{"id":"T796","city":"Accra","units":14},{"id":"G206","city":"Oslo","units":7},{"id":"U450","city":"Perth","units":4},{"id":"X493","city":"Oslo","units":5}]
Checked answer
{"city": "Oslo", "units": 5}
Seed

Correct

{"city":"Oslo","units":5}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":5}
11 tokens
0030 · filter · wording 3 · same10145e4795693c2043068118c74d78e7687d9ade9c4fd8a09909ea65e7e89e7d
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Q640","status":"ready"},{"id":"G534","status":"ready"},{"id":"K827","status":"closed"},{"id":"B616","status":"pending"},{"id":"F940","status":"ready"}]
Checked answer
{"ids": ["F940", "G534", "Q640"]}
Seed

wrong_schema_or_value

{"ids": ["G534", "K827", "F940", "Q640", "B616"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F940","G534","Q640"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["B616","F940","G534","Q640"]}
26 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F940","Q640","G534"]}
21 tokens
0031 · total · wording 3 · samea0b461e4f5ee9e57ed25e88e9172e2fe46e8344baf30eeff8ae6d9e94a919904
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"D370","units":11,"price":13},{"id":"Z578","units":0,"price":14},{"id":"N604","units":4,"price":7},{"id":"L243","units":5,"price":4},{"id":"I427","units":8,"price":19}]
Checked answer
{"total": 343}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0032 · sort · wording 3 · gainb304732e9282bfcf7acf0e1e1f290a4f76d2834121cadf0f9dc202547e6b7ee0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"V724","priority":8},{"id":"V182","priority":9},{"id":"T781","priority":6},{"id":"M269","priority":1},{"id":"U939","priority":8}]
Checked answer
{"ids": ["V182", "U939"]}
Seed

invalid_json_object

[{"id":"V724","priority":8},{"id":"V182","priority":9},{"id":"T781","priority":6},{"id":"M269","priority":1},{"id":"U939","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["V182","U939"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["V182","U939"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["V182","U939"]}
16 tokens
0033 · lookup · wording 0 · same519ac8d78254af4fbd13766d80e59a65bd642f207b643ad265971d6eb9a067d5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record L635. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"U885","city":"Perth","units":0},{"id":"L635","city":"Accra","units":6},{"id":"F732","city":"Lima","units":10},{"id":"E321","city":"Perth","units":12},{"id":"Q110","city":"Lima","units":17}]
Checked answer
{"city": "Accra", "units": 6}
Seed

Correct

{
  "city": "Accra",
  "units": 6
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":6}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":6}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":6}
11 tokens
0034 · filter · wording 0 · gainbcca856a77e1de6c8ec61866fe353ad053f91adc7e328336065003df4f09ace0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"D780","status":"pending"},{"id":"H204","status":"pending"},{"id":"A647","status":"ready"},{"id":"O363","status":"pending"},{"id":"B673","status":"pending"}]
Checked answer
{"ids": ["A647"]}
Seed

wrong_schema_or_value

{"ids": ["A647", "B673", "H204", "O363", "D780"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A647"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["A647"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["A647"]}
11 tokens
0035 · total · wording 0 · same1510789c9bb90b0bd88889a39b4cb0ba71c7d7c9cd89746ad8169df8bb3d7195
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"L268","units":19,"price":7},{"id":"O539","units":4,"price":18},{"id":"Z355","units":12,"price":13},{"id":"E231","units":6,"price":13},{"id":"W361","units":11,"price":5}]
Checked answer
{"total": 494}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0036 · sort · wording 0 · samead6f7406e7410a2625090eb5b28ded3c574a9015dec941f3638feee52c052265
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"B937","priority":8},{"id":"B676","priority":9},{"id":"N933","priority":9},{"id":"A582","priority":4},{"id":"N139","priority":5}]
Checked answer
{"ids": ["B676", "N933"]}
Seed

wrong_schema_or_value

{"ids": ["B937", "N933"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["B676","N933"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B676","N933"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N933","B676"]}
16 tokens
0037 · lookup · wording 1 · same726119baefb32c6be8b58bcc27a38bbc587dcc30ee0c46845e7b0632ad9f1b8f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id N703 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"N703","city":"Oslo","units":0},{"id":"L245","city":"Bern","units":10},{"id":"Q661","city":"Suva","units":19},{"id":"H199","city":"Lima","units":17},{"id":"B679","city":"Lima","units":3}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

Correct

{"city":"Oslo","units":0}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":0}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":0}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":0}
11 tokens
0038 · filter · wording 1 · same2208ad541c43474e90a0512c242a6ee2523ef40a9be106d57553f7781e88fa98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"C808","status":"pending"},{"id":"K238","status":"ready"},{"id":"Y737","status":"closed"},{"id":"E926","status":"closed"},{"id":"W909","status":"ready"}]
Checked answer
{"ids": ["K238", "W909"]}
Seed

wrong_schema_or_value

{
  "ids": ["C808", "K238", "Y737", "E926", "W909"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["W909","K238"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K238","W909"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C808","W909","K238"]}
21 tokens
0039 · total · wording 1 · same6fb22ce50fd8796feb890227d62598dd8ad19a32c5e46ba5f75b7c17a8b24187
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"B474","units":17,"price":6},{"id":"O658","units":1,"price":3},{"id":"L770","units":13,"price":3},{"id":"K392","units":13,"price":1},{"id":"Y564","units":17,"price":7}]
Checked answer
{"total": 276}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0040 · sort · wording 1 · same6f75f36a6c1d272ae9d7ccfd5e32f23a843aa59530d5d908223157337598f998
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"D278","priority":6},{"id":"F714","priority":4},{"id":"Z597","priority":9},{"id":"F427","priority":2},{"id":"B625","priority":8}]
Checked answer
{"ids": ["Z597", "B625"]}
Seed

invalid_json_object

[{"id":"F714","priority":4},{"id":"F427","priority":2},{"id":"B625","priority":8},{"id":"D278","priority":6},{"id":"Z597","priority":9}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D278","B625"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D278","B625"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z597","D278"]}
16 tokens
0041 · lookup · wording 2 · same79133cd59f5044049625c048972152672c5809d08946d74af18320fef013c8d9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"K345","city":"Perth","units":14},{"id":"K659","city":"Riga","units":5},{"id":"O284","city":"Perth","units":14},{"id":"R999","city":"Suva","units":7},{"id":"K864","city":"Kyoto","units":14}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0042 · filter · wording 2 · sameabe327ff85a9149d2d443229141f9c3a02f3756d5d2353a5e8c9460053412589
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Z936","status":"closed"},{"id":"H675","status":"ready"},{"id":"S991","status":"ready"},{"id":"B889","status":"pending"},{"id":"K626","status":"pending"}]
Checked answer
{"ids": ["H675", "S991"]}
Seed

invalid_json_object

[{"id":"H675","status":"ready"},{"id":"S991","status":"ready"},{"id":"B889","status":"pending"},{"id":"Z936","status":"closed"}]
51 tokens
1 GPU · AdamW

invalid_json_object

["H675","S991"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["H675","S991"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["H675","S991"]]
16 tokens
0043 · total · wording 2 · samefc1155196c1c35a6ed3b8a46c2974bb26c1d795555e1067d3086795a3d7ebc53
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"E257","units":10,"price":19},{"id":"A193","units":16,"price":15},{"id":"E145","units":9,"price":19},{"id":"T757","units":19,"price":17},{"id":"H747","units":9,"price":6}]
Checked answer
{"total": 978}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0044 · sort · wording 2 · same6c9fd06cb0943a8985607d7f3d429e59ebb362902d3141666e7b495540ba7459
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"P946","priority":5},{"id":"T203","priority":1},{"id":"S823","priority":1},{"id":"W691","priority":3},{"id":"K869","priority":1}]
Checked answer
{"ids": ["P946", "W691"]}
Seed

wrong_schema_or_value

{"ids": ["P946", "T203", "S823", "W691", "K869"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P946","W691"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P946","W691"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["P946","S823"]}
16 tokens
0045 · lookup · wording 3 · samed2c42fdc07bb3015d0f30875d2d6dd8b24f9c57d1207fea3fa62bdd753612a36
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=F670. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"F670","city":"Accra","units":2},{"id":"S610","city":"Oslo","units":13},{"id":"S771","city":"Riga","units":2},{"id":"O172","city":"Oslo","units":7},{"id":"T997","city":"Oslo","units":19}]
Checked answer
{"city": "Accra", "units": 2}
Seed

Correct

{"city": "Accra", "units": 2}
14 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":2}
11 tokens
0046 · filter · wording 3 · same2470e076f5c99adae8aa22ecbe72e2aaab539a60a4d959bea3a1e236e8931a06
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"M273","status":"ready"},{"id":"X836","status":"ready"},{"id":"U594","status":"pending"},{"id":"C239","status":"closed"},{"id":"G615","status":"ready"}]
Checked answer
{"ids": ["G615", "M273", "X836"]}
Seed

wrong_schema_or_value

{"ids": ["M273", "X836", "U594", "C239", "G615"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["C239","G615","M273","X836"]}
26 tokens
4 GPUs · DDP

Correct

{"ids":["G615","M273","X836"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["M273","G615","X836"]}
21 tokens
0047 · total · wording 3 · samed939d0d46463cabc9c85f05dedbcf897aa774293f3c507ec9bcee172ac847df3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"R320","units":4,"price":4},{"id":"A955","units":18,"price":19},{"id":"L701","units":19,"price":7},{"id":"V462","units":9,"price":13},{"id":"F367","units":11,"price":6}]
Checked answer
{"total": 674}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0048 · sort · wording 3 · gain4b9390172952cd5ac662718f8fa4b5e065e4fd2a935cd5dc1bad738cc0502e25
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"N355","priority":5},{"id":"J553","priority":6},{"id":"G765","priority":3},{"id":"D329","priority":4},{"id":"W961","priority":3}]
Checked answer
{"ids": ["J553", "N355"]}
Seed

invalid_json_object

[{"id":"N355","priority":5},{"id":"J553","priority":6},{"id":"G765","priority":3},{"id":"D329","priority":4},{"id":"W961","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J553","N355"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J553","N355"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J553","N355"]}
16 tokens
0049 · lookup · wording 0 · gain1957889d0b6dd7dd9cb371c50ebade6280601ff3f9cb1e1c7ffe95cffd99cc67
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record B255. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"I918","city":"Riga","units":10},{"id":"U267","city":"Kyoto","units":9},{"id":"B255","city":"Bern","units":11},{"id":"H419","city":"Suva","units":12},{"id":"S134","city":"Lima","units":2}]
Checked answer
{"city": "Bern", "units": 11}
Seed

invalid_json_object

[{"city":"Riga","units":10},{"city":"Bern","units":11},{"city":"Suva","units":12},{"city":"Lima","units":2}]
44 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":11}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":11}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":11}
11 tokens
0050 · filter · wording 0 · gain30c81ee4c6e53d4a82724f6de0fa9bfe3d67fd12979eaaaef9b0b6fb54dabc95
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P718","status":"pending"},{"id":"K307","status":"pending"},{"id":"R189","status":"pending"},{"id":"K556","status":"closed"},{"id":"X432","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["K307", "K556", "P718", "R189", "X432"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0051 · total · wording 0 · samed6a4e8700786feef49f852144e4cbed0f6a69ef0c5d74b279abe2e91d26e3222
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C644","units":18,"price":9},{"id":"O112","units":0,"price":7},{"id":"Q439","units":4,"price":18},{"id":"R930","units":5,"price":17},{"id":"S443","units":18,"price":17}]
Checked answer
{"total": 625}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0052 · sort · wording 0 · gain9c43024076c9c2c1bbcf7202fd286d3c478d695834ff9ce76ed1a2df241e83e5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J293","priority":7},{"id":"L837","priority":5},{"id":"V906","priority":9},{"id":"C473","priority":6},{"id":"Q477","priority":4}]
Checked answer
{"ids": ["V906", "J293"]}
Seed

wrong_schema_or_value

{"ids": ["J293", "V906", "C473", "Q477", "L837"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["V906","J293"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["V906","J293"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["V906","J293"]}
16 tokens
0053 · lookup · wording 1 · gain52fd34b1aae9e26b0bc947d34402e0333858785fde0c5e1f7de25decaac7d5f5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id E410 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"C389","city":"Lima","units":0},{"id":"W573","city":"Riga","units":2},{"id":"E410","city":"Accra","units":3},{"id":"J972","city":"Kyoto","units":15},{"id":"L927","city":"Riga","units":18}]
Checked answer
{"city": "Accra", "units": 3}
Seed

wrong_schema_or_value

{"city":"Lima","units":0}
11 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":3}
11 tokens
0054 · filter · wording 1 · gain284bd65ba8b7950e45e290b04ca0583fea9a5d9509aef0a1d981ce7011bd54a7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"O840","status":"pending"},{"id":"Y647","status":"ready"},{"id":"D360","status":"pending"},{"id":"W694","status":"closed"},{"id":"E837","status":"closed"}]
Checked answer
{"ids": ["Y647"]}
Seed

wrong_schema_or_value

{
  "ids": ["O840", "Y647", "D360", "W694", "E837"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["Y647"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Y647"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Y647"]}
11 tokens
0055 · total · wording 1 · same9d513f2361445c7d70aa96aa5bd937d80ea8f87079928868785125360b74ae3e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"U451","units":3,"price":12},{"id":"N442","units":18,"price":1},{"id":"H599","units":7,"price":12},{"id":"C185","units":13,"price":2},{"id":"Z179","units":11,"price":6}]
Checked answer
{"total": 230}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0056 · sort · wording 1 · gain15f6807ce73614fb653f0770d37400763a93d906fb4c9275dc5e54f52e40f427
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Q388","priority":9},{"id":"C266","priority":1},{"id":"L525","priority":1},{"id":"V702","priority":7},{"id":"K380","priority":1}]
Checked answer
{"ids": ["Q388", "V702"]}
Seed

invalid_json_object

[{"id":"Q388","priority":9},{"id":"C266","priority":1},{"id":"L525","priority":1},{"id":"V702","priority":7},{"id":"K380","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["Q388","V702"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Q388","C266"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q388","V702"]}
16 tokens
0057 · lookup · wording 2 · gainc075ea867766ce8f78ede8ffaaf5acb98f8e3b38686bf4e3529988471883fab6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to S367? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"S367","city":"Bern","units":13},{"id":"N742","city":"Bern","units":12},{"id":"S105","city":"Perth","units":14},{"id":"A324","city":"Kyoto","units":7},{"id":"W715","city":"Oslo","units":4}]
Checked answer
{"city": "Bern", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":13}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":13}
11 tokens
0058 · filter · wording 2 · gain4abb528e03df44289b36ea5dc39c666239037712b65d11e684430023ad1ca360
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"N979","status":"pending"},{"id":"W631","status":"pending"},{"id":"C140","status":"closed"},{"id":"M328","status":"pending"},{"id":"F618","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"C140","status":"closed"},{"id":"F618","status":"closed"},{"id":"M328","status":"pending"},{"id":"N979","status":"pending"},{"id":"W631","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0059 · total · wording 2 · same33ca5fb2ad151c79d2c737e30af998dc6a2366533ace8e3e4fc9fe06d1d106ec
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"S976","units":19,"price":4},{"id":"N622","units":5,"price":8},{"id":"T354","units":4,"price":5},{"id":"A229","units":6,"price":14},{"id":"Q191","units":13,"price":19}]
Checked answer
{"total": 467}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0060 · sort · wording 2 · same2fa4c20677427240a51abd912fc84757aa80c35fda96df619cbc650016cd6d2c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K506","priority":1},{"id":"A709","priority":9},{"id":"G256","priority":8},{"id":"R345","priority":4},{"id":"R751","priority":9}]
Checked answer
{"ids": ["A709", "R751"]}
Seed

wrong_schema_or_value

{"ids": ["A709", "G256", "R345", "R751", "K506"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A709","R751"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A709","R751"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["A709","R345"]}
16 tokens
0061 · lookup · wording 3 · gainc786b77371904fb96351b8d3f0bd45b70c4deb1bde31c7b9f8d70737046be08b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"O779","city":"Suva","units":7},{"id":"I493","city":"Bern","units":8},{"id":"B767","city":"Bern","units":1},{"id":"H839","city":"Kyoto","units":11},{"id":"O743","city":"Kyoto","units":7}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Suva","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0062 · filter · wording 3 · gain3f99656f6e796b9455f013bbebca8f81ed1526cb99f07ad728b70ef2d4f7522d
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Q791","status":"pending"},{"id":"T683","status":"closed"},{"id":"E631","status":"pending"},{"id":"A499","status":"closed"},{"id":"Q390","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A499", "E631", "Q390", "Q791", "T683"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0063 · total · wording 3 · same24d4e93f9532555557ae1f72b3f12620ba7329828147db1aca6e49861d0ec371
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"H785","units":3,"price":18},{"id":"Q846","units":17,"price":13},{"id":"C568","units":17,"price":19},{"id":"O627","units":17,"price":10},{"id":"V530","units":16,"price":2}]
Checked answer
{"total": 800}
Seed

invalid_json_object

{
  "total": 18 + 13 + 19 + 10 + 2 = 68
}
30 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0064 · sort · wording 3 · same3f7686e60efa5bc21d6dfed142d9b571aed6ba926f1149c7d5d4688c1f524ece
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"V879","priority":5},{"id":"A180","priority":9},{"id":"J850","priority":9},{"id":"K643","priority":9},{"id":"D676","priority":9}]
Checked answer
{"ids": ["A180", "D676"]}
Seed

invalid_json_object

[{"id":"J850","priority":9},{"id":"K643","priority":9},{"id":"V879","priority":5},{"id":"A180","priority":9},{"id":"D676","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A180","D676"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A180","K643"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J850","D676"]}
16 tokens
0065 · lookup · wording 0 · gainf9866c562640b4661949fb70b81a187ce501c8b0f6cd8b1646cf716258335617
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Y336. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"P670","city":"Oslo","units":11},{"id":"Y336","city":"Riga","units":5},{"id":"Y490","city":"Accra","units":0},{"id":"Z164","city":"Suva","units":13},{"id":"U459","city":"Riga","units":7}]
Checked answer
{"city": "Riga", "units": 5}
Seed

wrong_schema_or_value

{
  "city": "Oslo",
  "units": 11
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":5}
11 tokens
0066 · filter · wording 0 · gain81d9e5e54ec95976d3374e0b03867ad0d0ec15101a4e4e7d52d9cb00866b92c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P903","status":"pending"},{"id":"T908","status":"pending"},{"id":"D803","status":"closed"},{"id":"M161","status":"pending"},{"id":"I782","status":"ready"}]
Checked answer
{"ids": ["I782"]}
Seed

wrong_schema_or_value

{"ids": ["D803", "I782", "M161", "P903", "T908"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["I782"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["I782"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["I782"]}
11 tokens
0067 · total · wording 0 · same421770a1a705cbcad9686d8582e29abf498de8cedade5f9c1a77b6e95ce4d121
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C268","units":9,"price":3},{"id":"W661","units":5,"price":17},{"id":"X534","units":5,"price":13},{"id":"R204","units":15,"price":13},{"id":"E214","units":2,"price":15}]
Checked answer
{"total": 402}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0068 · sort · wording 0 · gain3824f67fca5cd340367a341f9ab35afc7f285853f49866ab71489fe4a0876309
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"C972","priority":9},{"id":"E607","priority":2},{"id":"B210","priority":9},{"id":"T345","priority":6},{"id":"W585","priority":5}]
Checked answer
{"ids": ["B210", "C972"]}
Seed

wrong_schema_or_value

{"ids": ["C972", "B210", "T345", "W585"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["C972","B210"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["C972","B210"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B210","C972"]}
16 tokens
0069 · lookup · wording 1 · gain8ec904e068dcef9c472eb35e63843fe52be19bdcd6af1127ff5777ac4d27889b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M498 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"L674","city":"Kyoto","units":8},{"id":"M498","city":"Perth","units":9},{"id":"L274","city":"Accra","units":9},{"id":"P477","city":"Lima","units":1},{"id":"B678","city":"Riga","units":7}]
Checked answer
{"city": "Perth", "units": 9}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":8}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":9}
11 tokens
0070 · filter · wording 1 · gaina71683421adf1f5b6639878a6361e89e40cfaf20c8a73b67c2878c4a31e7bc96
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"P357","status":"pending"},{"id":"D573","status":"closed"},{"id":"I125","status":"pending"},{"id":"J328","status":"closed"},{"id":"M594","status":"ready"}]
Checked answer
{"ids": ["M594"]}
Seed

wrong_schema_or_value

{
  "ids": ["M594", "P357", "I125", "J328", "D573"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["M594"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M594"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M594"]}
11 tokens
0071 · total · wording 1 · same3df0aa6b9d4492c314342f9ac2531d0714939d4ca7a91fb0bfb3274702dd01bb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"Y187","units":1,"price":18},{"id":"B451","units":16,"price":19},{"id":"D697","units":4,"price":15},{"id":"A208","units":0,"price":19},{"id":"J644","units":11,"price":8}]
Checked answer
{"total": 470}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0072 · sort · wording 1 · same853f16f958f4a022ef80e15504dd6ae88c56f5d6c522b1e60f6512a667eaea1c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"P311","priority":6},{"id":"I472","priority":2},{"id":"Z255","priority":3},{"id":"U743","priority":8},{"id":"U675","priority":4}]
Checked answer
{"ids": ["U743", "P311"]}
Seed

invalid_json_object

[{"id":"P311","priority":6},{"id":"U675","priority":4}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["U743","P311"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P311","U743"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["P311","Z255"]}
16 tokens
0073 · lookup · wording 2 · gain854113ec40e981bcb1f6b059bba82fdb30885b8b8f79315924219fd6837957db
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Y760? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Z440","city":"Perth","units":7},{"id":"V427","city":"Kyoto","units":15},{"id":"T922","city":"Lima","units":13},{"id":"J993","city":"Bern","units":9},{"id":"Y760","city":"Lima","units":12}]
Checked answer
{"city": "Lima", "units": 12}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":12}
12 tokens
0074 · filter · wording 2 · gain56809404c429fc7a63281df2db09fffdf9babb68394febb07f0d9cf5c41cd044
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"V557","status":"closed"},{"id":"S397","status":"pending"},{"id":"Q122","status":"closed"},{"id":"V117","status":"pending"},{"id":"T326","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"V557"},{"id":"S397"},{"id":"Q122"},{"id":"V117"},{"id":"T326"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0075 · total · wording 2 · same0a0c54a3a0a356a780a9ef2115124a49d2f2f6432ca21d0dc687bed7502aa916
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"G775","units":17,"price":1},{"id":"X765","units":6,"price":4},{"id":"D718","units":8,"price":18},{"id":"P933","units":8,"price":7},{"id":"Q952","units":13,"price":17}]
Checked answer
{"total": 462}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0076 · sort · wording 2 · gain3e565adccd15ffad59faa205af26ca8ce5d6289ca60ce80ee57d4c00d7421117
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"H533","priority":1},{"id":"Q520","priority":7},{"id":"D189","priority":7},{"id":"H539","priority":1},{"id":"R958","priority":6}]
Checked answer
{"ids": ["D189", "Q520"]}
Seed

wrong_schema_or_value

{"ids": ["H533", "H539"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["D189","Q520"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D189","Q520"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D189","Q520"]}
16 tokens
0077 · lookup · wording 3 · sameab213253c7030fcee6531e165ea54746170409683f63646c4c297c955e6998a3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=H971. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"C751","city":"Suva","units":9},{"id":"H971","city":"Suva","units":9},{"id":"T494","city":"Bern","units":18},{"id":"W232","city":"Oslo","units":15},{"id":"O319","city":"Oslo","units":19}]
Checked answer
{"city": "Suva", "units": 9}
Seed

Correct

{"city":"Suva","units":9}
11 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":9}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":9}
11 tokens
0078 · filter · wording 3 · same99240f12b5c77ab89327fb1c3486499e5ea3098ffa4f4d48b1199ab0e17b3d6a
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"P429","status":"pending"},{"id":"A267","status":"pending"},{"id":"O521","status":"pending"},{"id":"D136","status":"ready"},{"id":"I461","status":"ready"}]
Checked answer
{"ids": ["D136", "I461"]}
Seed

wrong_schema_or_value

{"ids": ["A267", "D136", "I461", "P429", "O521"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I461","D136"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D136","I461"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I461","D136"]}
16 tokens
0079 · total · wording 3 · same726f512a8f07ce11d407186fb6d0f8ab385dd9e4e9574463789a93e9e1a75644
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Y698","units":7,"price":15},{"id":"J774","units":7,"price":4},{"id":"F814","units":17,"price":7},{"id":"H340","units":13,"price":17},{"id":"J883","units":15,"price":14}]
Checked answer
{"total": 683}
Seed

invalid_json_object

{
  "total": 15 + 4 + 7 + 17 + 14 + 15 + 17 + 13 + 15 + 14 + 17 + 13 + 15 + 17 + 13 + 15 + 14 + 17 + 13 + 15 + 14 + 17 + 13 + 
96 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0080 · sort · wording 3 · gain617a289dc33637318798841c57bd8187ce01636be1bb8bcf643de70bafad2e9a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"H256","priority":3},{"id":"Y156","priority":8},{"id":"P202","priority":8},{"id":"F870","priority":9},{"id":"R336","priority":9}]
Checked answer
{"ids": ["F870", "R336"]}
Seed

invalid_json_object

[{"id":"H256","priority":3},{"id":"P202","priority":8},{"id":"F870","priority":9},{"id":"R336","priority":9}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["F870","R336"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["R336","F870"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F870","R336"]}
16 tokens
0081 · lookup · wording 0 · gainbafa6e5a378e13f8f7cebbec16e18dffce1ee987004345b45f0d01fe6ae6169b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"V913","city":"Oslo","units":17},{"id":"L140","city":"Accra","units":15},{"id":"D396","city":"Lima","units":9},{"id":"D340","city":"Perth","units":12},{"id":"P377","city":"Oslo","units":13}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Oslo","units":17}]
13 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0082 · filter · wording 0 · gain839d791c1439e8eafd1a9ad239906c313a394fb76b0dfdd93a6bc4733679870a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"C730","status":"pending"},{"id":"H909","status":"closed"},{"id":"X156","status":"closed"},{"id":"L652","status":"closed"},{"id":"Y179","status":"ready"}]
Checked answer
{"ids": ["Y179"]}
Seed

wrong_schema_or_value

{"ids": ["C730", "H909", "X156", "L652", "Y179"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Y179"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Y179"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Y179"]}
11 tokens
0083 · total · wording 0 · same4f118d60ab68968b03f523e4e5df71f9abdf06104ecb68df782493f3976ebfa2
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"B268","units":15,"price":13},{"id":"L180","units":12,"price":6},{"id":"R259","units":6,"price":12},{"id":"O424","units":19,"price":2},{"id":"A344","units":4,"price":1}]
Checked answer
{"total": 381}
Seed

wrong_schema_or_value

{"total": 130}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0084 · sort · wording 0 · same42ac8e79aec4a8fd4326e1e0518430443e8d9774e528460ad630808124bf6d0f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"O853","priority":2},{"id":"L629","priority":6},{"id":"H245","priority":7},{"id":"Z130","priority":8},{"id":"R443","priority":4}]
Checked answer
{"ids": ["Z130", "H245"]}
Seed

wrong_schema_or_value

{"ids": ["O853", "L629", "H245", "Z130", "R443"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H245","Z130"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H245","Z130"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["H245","Z130"]}
16 tokens
0085 · lookup · wording 1 · gain32639603fb9c861cf191e8a8250831d8a6d911dba878f41fac75deb9179ec398
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id J801 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"J801","city":"Oslo","units":5},{"id":"H326","city":"Lima","units":11},{"id":"J650","city":"Oslo","units":13},{"id":"T265","city":"Riga","units":18},{"id":"X983","city":"Oslo","units":0}]
Checked answer
{"city": "Oslo", "units": 5}
Seed

wrong_schema_or_value

{"city":"Oslo","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":5}
11 tokens
0086 · filter · wording 1 · gain966b0a9bd79f2f6bc364ab2a6a848f8127a5e07817af68dd066e25582402bb0b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"C362","status":"ready"},{"id":"P333","status":"closed"},{"id":"G441","status":"closed"},{"id":"M934","status":"closed"},{"id":"F390","status":"closed"}]
Checked answer
{"ids": ["C362"]}
Seed

wrong_schema_or_value

{
  "ids": ["C362", "F390", "M934", "P333", "G441"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["C362"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["C362"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["C362"]}
11 tokens
0087 · total · wording 1 · samefc63ec9999c52d51b91e3fbcd1352d1656800ec29d6ea7ff1c12faa2011cdfef
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"K523","units":2,"price":2},{"id":"U939","units":7,"price":16},{"id":"S460","units":11,"price":19},{"id":"W894","units":2,"price":2},{"id":"H930","units":16,"price":16}]
Checked answer
{"total": 585}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0088 · sort · wording 1 · samec156983367f9845f0d33d34b3b39b0b673e63974532ecd152d5d18bd5f825f81
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"S808","priority":6},{"id":"M561","priority":7},{"id":"A569","priority":4},{"id":"C295","priority":2},{"id":"F292","priority":8}]
Checked answer
{"ids": ["F292", "M561"]}
Seed

invalid_json_object

[{"id":"S808","priority":6},{"id":"M561","priority":7},{"id":"A569","priority":4},{"id":"C295","priority":2},{"id":"F292","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["F292","M561"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F292","M561"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F292","S808"]}
16 tokens
0089 · lookup · wording 2 · gainb175048b4c7e6080a87b29f524ab4d1fc4f85ae6aff1cd3df3ce8f8b2cf2e1dc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to R789? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"R789","city":"Accra","units":11},{"id":"N243","city":"Oslo","units":6},{"id":"N876","city":"Riga","units":2},{"id":"B215","city":"Bern","units":3},{"id":"H124","city":"Lima","units":7}]
Checked answer
{"city": "Accra", "units": 11}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":11}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":11}
12 tokens
0090 · filter · wording 2 · samec6c8b67e0e79160970abe3aeb0b0e4e3962c3503cb6d3428917b788ff2aae24b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"M893","status":"ready"},{"id":"S221","status":"pending"},{"id":"E106","status":"closed"},{"id":"D406","status":"ready"},{"id":"I171","status":"closed"}]
Checked answer
{"ids": ["D406", "M893"]}
Seed

invalid_json_object

[{"id":"D406","status":"ready"},{"id":"I171","status":"closed"},{"id":"M893","status":"ready"},{"id":"S221","status":"pending"},{"id":"E106","status":"closed"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["D406","M893"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D406","M893"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["D406","M893"]]
16 tokens
0091 · total · wording 2 · same1acf05e04bbf658cdb3226fba1fa004d97d75a636a9e52b058441f2165d05d72
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"J443","units":3,"price":14},{"id":"C490","units":2,"price":8},{"id":"G761","units":14,"price":18},{"id":"J821","units":0,"price":16},{"id":"F769","units":0,"price":1}]
Checked answer
{"total": 310}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0092 · sort · wording 2 · sameae496ac92759c583fd24f5426156f80d6a3f6c83b8a8af233382168f22f336c8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"L807","priority":9},{"id":"C310","priority":4},{"id":"Q623","priority":8},{"id":"K554","priority":3},{"id":"O150","priority":6}]
Checked answer
{"ids": ["L807", "Q623"]}
Seed

Correct

{"ids": ["L807", "Q623"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["L807","Q623"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L807","Q623"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L807","Q623"]}
16 tokens
0093 · lookup · wording 3 · gaincfee516369d4ea44dd5b3920c6c7ad544da4d11986897ad63665ec849dc2033d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=I114. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"U814","city":"Perth","units":9},{"id":"I114","city":"Bern","units":19},{"id":"N374","city":"Lima","units":12},{"id":"W298","city":"Perth","units":17},{"id":"Q987","city":"Perth","units":12}]
Checked answer
{"city": "Bern", "units": 19}
Seed

wrong_schema_or_value

{"city":"Perth","units":9}
11 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":19}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":19}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":19}
11 tokens
0094 · filter · wording 3 · sameb02341480224e5fb2e0444e87a426ccc9473eab9963399065c7e849e46cc0848
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"K908","status":"ready"},{"id":"U498","status":"ready"},{"id":"T684","status":"pending"},{"id":"E516","status":"closed"},{"id":"Y735","status":"ready"}]
Checked answer
{"ids": ["K908", "U498", "Y735"]}
Seed

wrong_schema_or_value

{"ids": ["K908", "U498", "T684", "E516", "Y735"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["E516","K908","Y735","U498"]}
26 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["K908","Y735"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K908","E516","Y735"]}
21 tokens
0095 · total · wording 3 · sameccd8359a6862dc553fbd993ee2d3219472d4ab6554d10296acd82d598b606059
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Y767","units":16,"price":8},{"id":"S696","units":0,"price":4},{"id":"N964","units":14,"price":6},{"id":"Q989","units":7,"price":10},{"id":"M107","units":6,"price":1}]
Checked answer
{"total": 288}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0096 · sort · wording 3 · samef8308633ba99fb7fbe6f0d536a31198a5af2e3850142efa20eaa65de02d887a6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"N581","priority":4},{"id":"H776","priority":4},{"id":"J601","priority":9},{"id":"H644","priority":5},{"id":"M930","priority":2}]
Checked answer
{"ids": ["J601", "H644"]}
Seed

invalid_json_object

[{"id":"H776","priority":4},{"id":"H644","priority":5},{"id":"M930","priority":2},{"id":"N581","priority":4},{"id":"J601","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J601","H644"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J601","H644"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J601","N581"]}
16 tokens
0097 · lookup · wording 0 · samecf02121c4bad39cc19e40242a3eaf3b7f19e1ba26e6b9310da40466fc71b5d87
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record D316. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y922","city":"Suva","units":1},{"id":"D316","city":"Kyoto","units":17},{"id":"G878","city":"Perth","units":0},{"id":"V585","city":"Riga","units":17},{"id":"T785","city":"Bern","units":3}]
Checked answer
{"city": "Kyoto", "units": 17}
Seed

Correct

{
  "city": "Kyoto",
  "units": 17
}
20 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":17}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":17}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":17}
13 tokens
0098 · filter · wording 0 · gain3e292bbefbd6b2f38c8ada173741f2e3cea0c2a13ba2d0a165bda5fffec66973
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"X686","status":"ready"},{"id":"Q605","status":"closed"},{"id":"T909","status":"pending"},{"id":"L896","status":"pending"},{"id":"D922","status":"closed"}]
Checked answer
{"ids": ["X686"]}
Seed

wrong_schema_or_value

{"ids": ["X686", "Q605", "T909", "L896", "D922"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["X686"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["X686"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["X686"]}
11 tokens
0099 · total · wording 0 · same657a003ba333940b9d976f1afac38ea8b1d8ae083d413afc71e5907e5ace3666
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"A671","units":12,"price":9},{"id":"P459","units":5,"price":7},{"id":"F143","units":8,"price":11},{"id":"E527","units":18,"price":2},{"id":"Y483","units":7,"price":11}]
Checked answer
{"total": 344}
Seed

invalid_json_object

{"total": 12*9 + 5*7 + 8*11 + 18*2 + 7*11}
33 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0100 · sort · wording 0 · gain84e66e3bbb7338f80fb88520411d437aab25dcb31a121d1fe7441c417628886b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E718","priority":7},{"id":"H579","priority":4},{"id":"Q637","priority":9},{"id":"X943","priority":1},{"id":"T560","priority":5}]
Checked answer
{"ids": ["Q637", "E718"]}
Seed

wrong_schema_or_value

{"ids": ["H579", "Q637", "X943", "T560", "E718"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["E718","Q637"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E718","Q637"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q637","E718"]}
16 tokens
0101 · lookup · wording 1 · gaina014ff06fb7cb0bdd32d76b873000e6459965ce7b227195cc88118dbfde7c191
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"I834","city":"Oslo","units":18},{"id":"T940","city":"Perth","units":10},{"id":"C670","city":"Bern","units":14},{"id":"W954","city":"Kyoto","units":7},{"id":"U530","city":"Accra","units":2}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":18}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0102 · filter · wording 1 · same72cad94cc5461c31f005886962aab548d04129f75ecfbd702d3205e43b0d8e6c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"R372","status":"pending"},{"id":"W679","status":"ready"},{"id":"X576","status":"closed"},{"id":"V492","status":"closed"},{"id":"Q280","status":"ready"}]
Checked answer
{"ids": ["Q280", "W679"]}
Seed

wrong_schema_or_value

{
  "ids": ["R372", "W679", "X576", "V492", "Q280"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["Q280","W679"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Q280","W679"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["W679","Q280"]}
16 tokens
0103 · total · wording 1 · same15875f171551a310432f1913ea89a5a50098fa6e3371551004aa4bbcd4d6608d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"V918","units":8,"price":4},{"id":"K273","units":3,"price":7},{"id":"U916","units":11,"price":17},{"id":"K396","units":13,"price":11},{"id":"D744","units":14,"price":11}]
Checked answer
{"total": 537}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0104 · sort · wording 1 · gainb656628590383c2f8a1283f134e2add6584736b130cd70512842d63800177fb2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J801","priority":5},{"id":"W649","priority":2},{"id":"S241","priority":7},{"id":"E474","priority":6},{"id":"A484","priority":1}]
Checked answer
{"ids": ["S241", "E474"]}
Seed

invalid_json_object

[{"id":"J801","priority":5},{"id":"W649","priority":2},{"id":"S241","priority":7},{"id":"E474","priority":6},{"id":"A484","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["S241","E474"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S241","E474"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S241","E474"]}
16 tokens
0105 · lookup · wording 2 · gainef43824407b7f6dbcd70415c9b23e69f3f222cc7ff5178a76b43224fc49a534e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to E560? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"V724","city":"Suva","units":19},{"id":"B143","city":"Perth","units":9},{"id":"T185","city":"Accra","units":0},{"id":"E560","city":"Lima","units":13},{"id":"J901","city":"Kyoto","units":17}]
Checked answer
{"city": "Lima", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":13}
12 tokens
0106 · filter · wording 2 · same6d86e7657bd589e3a0a7199c633b1afcd906f1ebd877d481e4803e60da47624b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"H990","status":"closed"},{"id":"W595","status":"ready"},{"id":"J848","status":"ready"},{"id":"Z947","status":"closed"},{"id":"Q446","status":"closed"}]
Checked answer
{"ids": ["J848", "W595"]}
Seed

invalid_json_object

[{"id":"W595","status":"ready"},{"id":"J848","status":"ready"},{"id":"Z947","status":"closed"},{"id":"Q446","status":"closed"}]
51 tokens
1 GPU · AdamW

invalid_json_object

["J848","W595"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["J848","W595"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["J848","W595"]]
16 tokens
0107 · total · wording 2 · same2fde852f8008cd708f79e3ac8bd9306058f04188e0788be5f69fb0a7039dce91
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"W309","units":16,"price":14},{"id":"M107","units":9,"price":7},{"id":"D117","units":1,"price":5},{"id":"G606","units":9,"price":13},{"id":"U558","units":1,"price":12}]
Checked answer
{"total": 421}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0108 · sort · wording 2 · same721e3aff45786836966698155108e9f86c202ddfbe2075ddefe7f56bece9d5c0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Y679","priority":5},{"id":"B249","priority":5},{"id":"K535","priority":5},{"id":"C902","priority":1},{"id":"V659","priority":8}]
Checked answer
{"ids": ["V659", "B249"]}
Seed

wrong_schema_or_value

{"ids": ["Y679", "B249", "K535", "C902", "V659"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["V659","B249"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V659","Y679"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["V659","K535"]}
16 tokens
0109 · lookup · wording 3 · same03b3c0e905726455fa4d5545a1247e436149e4041376858d87112650d1db0f80
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=B636. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Q794","city":"Accra","units":0},{"id":"B402","city":"Accra","units":9},{"id":"B822","city":"Oslo","units":3},{"id":"B636","city":"Perth","units":19},{"id":"H580","city":"Suva","units":17}]
Checked answer
{"city": "Perth", "units": 19}
Seed

Correct

{"city":"Perth","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":19}
12 tokens
0110 · filter · wording 3 · gain3f9e57d9846e03676ced9229001394cb213b2360539763833651a680fe4e01e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"U972","status":"pending"},{"id":"K659","status":"closed"},{"id":"C741","status":"pending"},{"id":"W216","status":"ready"},{"id":"L816","status":"ready"}]
Checked answer
{"ids": ["L816", "W216"]}
Seed

wrong_schema_or_value

{"ids": ["L816", "U972", "C741", "K659", "W216"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L816","W216"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L816","W216"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L816","W216"]}
16 tokens
0111 · total · wording 3 · same7e2a2a162c61b61913eef434935af7d1c945fab7bfca5b52aea1f1a05135fca0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"S249","units":5,"price":2},{"id":"D389","units":10,"price":5},{"id":"A698","units":0,"price":19},{"id":"Y820","units":9,"price":6},{"id":"H852","units":15,"price":14}]
Checked answer
{"total": 324}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0112 · sort · wording 3 · gain8efbcb5085f8e84bd2ad3c440f69c78d58cbb1ace775dff8cff2b5d254dd972e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"X257","priority":8},{"id":"X354","priority":4},{"id":"R638","priority":5},{"id":"V760","priority":5},{"id":"U725","priority":1}]
Checked answer
{"ids": ["X257", "R638"]}
Seed

invalid_json_object

[{"id":"X257"},{"id":"X354"},{"id":"R638"},{"id":"V760"},{"id":"U725"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["X257","R638"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X257","X354"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X257","R638"]}
16 tokens
0113 · lookup · wording 0 · same526df7165e4481b0fa925b3b5268c5daa003153c2e1e9122fcc16e0271590369
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record D689. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"D689","city":"Suva","units":15},{"id":"X624","city":"Riga","units":17},{"id":"I226","city":"Suva","units":14},{"id":"S954","city":"Perth","units":13},{"id":"R460","city":"Riga","units":6}]
Checked answer
{"city": "Suva", "units": 15}
Seed

Correct

{
  "city": "Suva",
  "units": 15
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":15}
12 tokens
0114 · filter · wording 0 · gain08e6b349f43e0a5f6906b8f65e4e8d13c994a64115e3bf3631b57127c800b3f4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"H538","status":"ready"},{"id":"J246","status":"ready"},{"id":"U294","status":"pending"},{"id":"P278","status":"pending"},{"id":"W747","status":"ready"}]
Checked answer
{"ids": ["H538", "J246", "W747"]}
Seed

wrong_schema_or_value

{"ids": ["H538", "J246", "U294", "P278", "W747"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H538","W747","J246"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H538","W747","J246"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["H538","J246","W747"]}
21 tokens
0115 · total · wording 0 · samecc7482fbaef70bf2bbf086ebc7f4dc818df5b00d4b348e7f83d3681aa5219643
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"D711","units":9,"price":13},{"id":"O985","units":16,"price":4},{"id":"Z339","units":13,"price":8},{"id":"J983","units":16,"price":11},{"id":"Q336","units":14,"price":9}]
Checked answer
{"total": 587}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0116 · sort · wording 0 · gainfacf235047877f495dd194a8277ddd0a348bb23368d2e5e2ac116b3289079fcb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"R561","priority":5},{"id":"J426","priority":7},{"id":"Z414","priority":9},{"id":"L551","priority":9},{"id":"V328","priority":6}]
Checked answer
{"ids": ["L551", "Z414"]}
Seed

wrong_schema_or_value

{"ids": ["L551", "V328", "Z414", "J426", "R561"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L551","Z414"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L551","Z414"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L551","Z414"]}
16 tokens
0117 · lookup · wording 1 · gain41aef5f209bf9514354158066fb64b03e58b978006905fa9c6aececbbc53879b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id T887 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"U585","city":"Perth","units":4},{"id":"T887","city":"Perth","units":17},{"id":"V950","city":"Suva","units":13},{"id":"I993","city":"Kyoto","units":7},{"id":"V843","city":"Oslo","units":2}]
Checked answer
{"city": "Perth", "units": 17}
Seed

wrong_schema_or_value

{"city":"Perth","units":4}
11 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":17}
12 tokens
0118 · filter · wording 1 · gaina511d3b752ea461765cd6a7f7c687728ca870921c6de7739a6fe37d9642fa98f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"N860","status":"closed"},{"id":"Z407","status":"closed"},{"id":"A135","status":"pending"},{"id":"T907","status":"pending"},{"id":"L468","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["A135", "L468", "N860", "T907", "Z407"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0119 · total · wording 1 · same0f707b784e82afcec473c6dce1e9e5786d7e2c8f3dfbdc5f0fa67148b15ba21d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"X312","units":0,"price":13},{"id":"O185","units":10,"price":1},{"id":"A365","units":10,"price":7},{"id":"A359","units":2,"price":13},{"id":"S290","units":19,"price":1}]
Checked answer
{"total": 125}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0120 · sort · wording 1 · gain4468055348561dc6ff19312807354bc1127d397a4e929388b6ac5e4d6340ce8a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"K475","priority":3},{"id":"S105","priority":1},{"id":"G265","priority":2},{"id":"B553","priority":4},{"id":"K921","priority":6}]
Checked answer
{"ids": ["K921", "B553"]}
Seed

invalid_json_object

[{"id":"K475","priority":3},{"id":"S105","priority":1},{"id":"G265","priority":2},{"id":"B553","priority":4},{"id":"K921","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["K921","B553"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K921","B553"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K921","B553"]}
16 tokens
0121 · lookup · wording 2 · same24eb7d51c1b548a5aa9b705e0d6d96701f3f92c1fa2d20560cd45e6f298658f0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"A308","city":"Riga","units":7},{"id":"J120","city":"Bern","units":18},{"id":"X612","city":"Bern","units":2},{"id":"U739","city":"Bern","units":8},{"id":"Q494","city":"Suva","units":10}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0122 · filter · wording 2 · same8f4b51ac891c29d852fffe31aed1e076aa75bcab2499f8a9d1fe04903fa7701b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"M738","status":"pending"},{"id":"R662","status":"closed"},{"id":"I556","status":"pending"},{"id":"Q960","status":"closed"},{"id":"V743","status":"ready"}]
Checked answer
{"ids": ["V743"]}
Seed

invalid_json_object

[{"id":"V743","status":"ready"}]
15 tokens
1 GPU · AdamW

Correct

{"ids":["V743"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["V743"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["V743"]]
11 tokens
0123 · total · wording 2 · same99c048af1527e957e8e6086a214d0cb136d3b35edda700a7316edce35ff23e34
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R874","units":19,"price":18},{"id":"K325","units":2,"price":12},{"id":"A738","units":19,"price":7},{"id":"L228","units":8,"price":18},{"id":"K272","units":9,"price":2}]
Checked answer
{"total": 661}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0124 · sort · wording 2 · gain708cdc5e9ec1d1bee4d1c393539f4a17b5c3244b63dd1db267919382200543a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K344","priority":8},{"id":"M378","priority":6},{"id":"V338","priority":2},{"id":"V105","priority":3},{"id":"J856","priority":1}]
Checked answer
{"ids": ["K344", "M378"]}
Seed

wrong_schema_or_value

{"ids": ["V105", "V338", "K344", "M378", "J856"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["K344","M378"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K344","M378"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K344","M378"]}
16 tokens
0125 · lookup · wording 3 · sameed29fd432f1dde63d9f9c586a6501f658f739d5c3b22b09719a4e25391fdec2c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=E672. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"G855","city":"Riga","units":9},{"id":"J150","city":"Bern","units":8},{"id":"E672","city":"Oslo","units":10},{"id":"T617","city":"Accra","units":12},{"id":"R701","city":"Bern","units":0}]
Checked answer
{"city": "Oslo", "units": 10}
Seed

Correct

{"city":"Oslo","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":10}
12 tokens
0126 · filter · wording 3 · samea8e2935f9d8b34f157556289c72a480fba2fa96d690e3ffa4045d3e2e0193af2
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"O804","status":"ready"},{"id":"M638","status":"closed"},{"id":"F424","status":"ready"},{"id":"A448","status":"pending"},{"id":"D336","status":"ready"}]
Checked answer
{"ids": ["D336", "F424", "O804"]}
Seed

wrong_schema_or_value

{"ids": ["A448", "D336", "F424", "O804", "M638"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D336","F424","O804"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["D336","F424","O804"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F424","D336","O804"]}
21 tokens
0127 · total · wording 3 · same946a99db7f106e8abbadc0d36e9213dbdafb22d80265f8ed6824b089c8e85af3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"X933","units":16,"price":16},{"id":"V222","units":14,"price":10},{"id":"B838","units":12,"price":6},{"id":"Z162","units":14,"price":19},{"id":"Z644","units":15,"price":13}]
Checked answer
{"total": 929}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0128 · sort · wording 3 · same43082e09847d99ff1d43bcb9ef0a03ba34e00d6b9ce9e542d768479106de98f1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"R723","priority":4},{"id":"F289","priority":8},{"id":"Y855","priority":7},{"id":"R559","priority":9},{"id":"R993","priority":2}]
Checked answer
{"ids": ["R559", "F289"]}
Seed

invalid_json_object

[{"id":"R723","priority":4},{"id":"R559","priority":9},{"id":"R993","priority":2}]
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["F289","R559"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["F289","R559"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F289","R559"]}
16 tokens
0129 · lookup · wording 0 · same75fba1047186da169da29372240d7f28d16f695c7dad93d535f98c9bb5ecf5a4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Q492. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"O313","city":"Oslo","units":2},{"id":"P767","city":"Lima","units":14},{"id":"X400","city":"Kyoto","units":6},{"id":"Q492","city":"Bern","units":13},{"id":"D710","city":"Bern","units":0}]
Checked answer
{"city": "Bern", "units": 13}
Seed

Correct

{
  "city": "Bern",
  "units": 13
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":13}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":13}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":13}
11 tokens
0130 · filter · wording 0 · gaina7b82b7f55d79beaf49b62af6d4ef86b76c0f1e5015de47dc87b759564372c7e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"C245","status":"pending"},{"id":"C773","status":"ready"},{"id":"M701","status":"ready"},{"id":"E451","status":"closed"},{"id":"G492","status":"pending"}]
Checked answer
{"ids": ["C773", "M701"]}
Seed

wrong_schema_or_value

{"ids": ["C245", "C773", "M701", "E451", "G492"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C773","M701"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C773","M701"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C773","M701"]}
16 tokens
0131 · total · wording 0 · samec3b337742c5f267007e2f8c93ef32ea0058ae7f565a4bbcb816c021be8abf41b
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"I264","units":10,"price":15},{"id":"A512","units":19,"price":4},{"id":"E748","units":4,"price":11},{"id":"M684","units":14,"price":18},{"id":"V559","units":19,"price":5}]
Checked answer
{"total": 617}
Seed

wrong_schema_or_value

{"total": 150}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0132 · sort · wording 0 · same402c12143cf41da88b4eef9bbfc570f8b2264e3ef2da058082b20774195f50c3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"I833","priority":2},{"id":"J695","priority":7},{"id":"U662","priority":3},{"id":"K981","priority":7},{"id":"X708","priority":3}]
Checked answer
{"ids": ["J695", "K981"]}
Seed

wrong_schema_or_value

{"ids": ["U662", "J695", "I833", "K981", "X708"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["K981","J695"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["K981","J695"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K981","J695"]}
16 tokens
0133 · lookup · wording 1 · same1a2d1c069b036ca9b6f1c30c7547b00c8bd483f1417234b1f517953b54e1a4b7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id B802 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"B802","city":"Perth","units":11},{"id":"F425","city":"Lima","units":15},{"id":"G278","city":"Accra","units":12},{"id":"N884","city":"Kyoto","units":6},{"id":"Q806","city":"Riga","units":5}]
Checked answer
{"city": "Perth", "units": 11}
Seed

Correct

{"city":"Perth","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":11}
12 tokens
0134 · filter · wording 1 · gain3ae0064c3f5a98fa1589b9cef929e14ccaa2a153f3e4ed3fc1d1e5734e0ccabe
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"I240","status":"closed"},{"id":"E705","status":"closed"},{"id":"C478","status":"pending"},{"id":"B687","status":"ready"},{"id":"W166","status":"ready"}]
Checked answer
{"ids": ["B687", "W166"]}
Seed

wrong_schema_or_value

{
  "ids": ["B687", "C478", "E705", "I240", "W166"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["B687","W166"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B687","W166"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B687","W166"]}
16 tokens
0135 · total · wording 1 · same9fbb6f31f0bb6f5b9eb748a815615c69f6b6361d000ee78411c9c184da9c5fc5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"J722","units":6,"price":15},{"id":"W864","units":19,"price":19},{"id":"P912","units":7,"price":13},{"id":"D450","units":9,"price":17},{"id":"G957","units":4,"price":11}]
Checked answer
{"total": 739}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0136 · sort · wording 1 · gain20cf9ae10e2875f08dc9368bf1e48c3e262a9d6c5cebd1331f580f63f7cbe1b5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V725","priority":9},{"id":"S559","priority":5},{"id":"W920","priority":6},{"id":"B109","priority":2},{"id":"W469","priority":9}]
Checked answer
{"ids": ["V725", "W469"]}
Seed

invalid_json_object

[{"id":"V725","priority":9},{"id":"W469","priority":9}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["V725","W469"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W469","V725"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["V725","W469"]}
16 tokens
0137 · lookup · wording 2 · gain63e64ffec57c1caf3865c9515526b7d91f25526cbe62f96eb3a97436e498ab57
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to U863? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"F924","city":"Accra","units":8},{"id":"F534","city":"Kyoto","units":2},{"id":"U863","city":"Kyoto","units":14},{"id":"K831","city":"Bern","units":16},{"id":"F479","city":"Perth","units":10}]
Checked answer
{"city": "Kyoto", "units": 14}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":14}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":14}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":14}
13 tokens
0138 · filter · wording 2 · same648ec22cfb5003e9b34ff6aa23253ff4ccb9a88d1483fde8c5812add4f8d7da7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"P167","status":"ready"},{"id":"F520","status":"closed"},{"id":"J600","status":"closed"},{"id":"P345","status":"pending"},{"id":"U928","status":"pending"}]
Checked answer
{"ids": ["P167"]}
Seed

invalid_json_object

[{"id":"P167","status":"ready"},{"id":"F520","status":"closed"},{"id":"J600","status":"closed"},{"id":"P345","status":"pending"},{"id":"U928","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["P167"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["P167"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["P167"]]
11 tokens
0139 · total · wording 2 · samee69d90a44ffee3fccdfd17c3e3e870a8bd329e1cd5e68bd474691006cd20efd4
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R293","units":18,"price":18},{"id":"M242","units":9,"price":18},{"id":"A662","units":9,"price":15},{"id":"X519","units":6,"price":14},{"id":"J895","units":7,"price":18}]
Checked answer
{"total": 831}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0140 · sort · wording 2 · same7267e93c8333dac121750fb586f5f366fb7c74b42e71b6cc13e4a3541bb3183c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"A362","priority":6},{"id":"U992","priority":1},{"id":"K126","priority":9},{"id":"K507","priority":3},{"id":"Z431","priority":7}]
Checked answer
{"ids": ["K126", "Z431"]}
Seed

wrong_schema_or_value

{"ids": ["U992", "K126"]}
17 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["K126","A362"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["K126","A362"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K126","A362"]}
16 tokens
0141 · lookup · wording 3 · gaina47af6f8c502c72a45a47c276c959b01511010b6b0750397f5b088ff73a03d8b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E334","city":"Accra","units":19},{"id":"N510","city":"Bern","units":2},{"id":"C771","city":"Riga","units":17},{"id":"P658","city":"Perth","units":16},{"id":"C827","city":"Suva","units":0}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Accra","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0142 · filter · wording 3 · gain8b4f32502e99eeceda91be1e71f4c711ceadbebd9ed742243ad0493a127ed173
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D831","status":"closed"},{"id":"W418","status":"closed"},{"id":"Z293","status":"closed"},{"id":"E221","status":"ready"},{"id":"P524","status":"pending"}]
Checked answer
{"ids": ["E221"]}
Seed

wrong_schema_or_value

{"ids": ["E221", "P524", "D831", "W418", "Z293"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E221"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["E221"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["E221"]}
11 tokens
0143 · total · wording 3 · same7bff77fdf2b40da5dd278b3bcf8a0ac6c90f8ce02e68117e1af1d481f92712dc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"J943","units":8,"price":13},{"id":"E489","units":6,"price":9},{"id":"F389","units":12,"price":3},{"id":"I154","units":4,"price":1},{"id":"P544","units":3,"price":18}]
Checked answer
{"total": 252}
Seed

wrong_schema_or_value

{
  "total": 104
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0144 · sort · wording 3 · gain375eb986194bb45b0b176140fd7689606e26c907a5dab629d2f9ae54057efcf6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"Y208","priority":4},{"id":"W807","priority":4},{"id":"Z591","priority":6},{"id":"D326","priority":1},{"id":"X912","priority":8}]
Checked answer
{"ids": ["X912", "Z591"]}
Seed

invalid_json_object

[{"id":"Y208","priority":4},{"id":"Z591","priority":6},{"id":"D326","priority":1},{"id":"X912","priority":8}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["X912","Z591"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["X912","Z591"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X912","Z591"]}
16 tokens
0145 · lookup · wording 0 · samea10ecee289cc0ff81347963f30489148aeb3db1d2f1b7b15e4a17c59e6248f47
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record U440. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"U742","city":"Lima","units":9},{"id":"T738","city":"Suva","units":11},{"id":"B350","city":"Riga","units":1},{"id":"J308","city":"Perth","units":10},{"id":"U440","city":"Suva","units":11}]
Checked answer
{"city": "Suva", "units": 11}
Seed

Correct

{
  "city": "Suva",
  "units": 11
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":11}
12 tokens
0146 · filter · wording 0 · gaindb9cd93510cc870fb276136c7f7fb627d331954b9c25a1fe83b3c17df22abf14
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"I607","status":"closed"},{"id":"X911","status":"pending"},{"id":"U203","status":"closed"},{"id":"N335","status":"closed"},{"id":"X790","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["U203", "N335", "X911", "X790", "I607"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0147 · total · wording 0 · same2794a45205566b816c6084fc02ffb98ea3f9091864da6fd497c727b02772b16a
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P154","units":19,"price":7},{"id":"V361","units":19,"price":15},{"id":"G223","units":8,"price":13},{"id":"M712","units":15,"price":9},{"id":"L276","units":6,"price":6}]
Checked answer
{"total": 693}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0148 · sort · wording 0 · gainffeb077bce51e3c16a40bd9912741fc762cf70a1a0974de0f59fc15c5c3b41ed
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E677","priority":7},{"id":"J496","priority":2},{"id":"Y825","priority":1},{"id":"Z651","priority":5},{"id":"D700","priority":5}]
Checked answer
{"ids": ["E677", "D700"]}
Seed

wrong_schema_or_value

{"ids": ["Y825", "J496", "E677", "Z651", "D700"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E677","D700"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E677","D700"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E677","D700"]}
16 tokens
0149 · lookup · wording 1 · gaind1916806a439543fd5e6fef9b6b6e8e8f236ee2c50ec52166a5aa0d29a2ef206
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id B731 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"O806","city":"Lima","units":6},{"id":"R119","city":"Accra","units":14},{"id":"G675","city":"Oslo","units":2},{"id":"Y811","city":"Bern","units":15},{"id":"B731","city":"Riga","units":7}]
Checked answer
{"city": "Riga", "units": 7}
Seed

wrong_schema_or_value

{"city":"Lima","units":6}
11 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":7}
11 tokens
0150 · filter · wording 1 · gain632b4ab0d3d6eced6b515f24270f2d51caee2b3bfab9aa6d620823aeae2c94af
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M334","status":"pending"},{"id":"I579","status":"closed"},{"id":"B504","status":"pending"},{"id":"S746","status":"pending"},{"id":"Q462","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["M334", "I579", "B504", "S746", "Q462"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0151 · total · wording 1 · same6f75c8c70220212767ed6ba5deddd2b14cad05643859bafcc3f97353d1bb063c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"I778","units":4,"price":17},{"id":"D646","units":8,"price":18},{"id":"Z666","units":4,"price":11},{"id":"R168","units":14,"price":11},{"id":"P472","units":9,"price":4}]
Checked answer
{"total": 446}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0152 · sort · wording 1 · gainaa60f86288b48dc3b0474075404c7f1cdce69ca26b9f837ac2cf4f8ab8b7f7fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V772","priority":3},{"id":"C372","priority":1},{"id":"Y274","priority":1},{"id":"K107","priority":5},{"id":"E916","priority":5}]
Checked answer
{"ids": ["E916", "K107"]}
Seed

invalid_json_object

[{"id":"V772","priority":3},{"id":"C372","priority":1},{"id":"Y274","priority":1},{"id":"K107","priority":5},{"id":"E916","priority":5}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["K107","E916"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E916","K107"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E916","K107"]}
16 tokens
0153 · lookup · wording 2 · gain401ac3c40bc044470d56f2c4e81777e02b977467b8f20fa20f6784fff32a694a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to B817? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"V186","city":"Bern","units":0},{"id":"B817","city":"Kyoto","units":18},{"id":"G407","city":"Lima","units":11},{"id":"O934","city":"Perth","units":10},{"id":"X104","city":"Bern","units":18}]
Checked answer
{"city": "Kyoto", "units": 18}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":18}
13 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":18}
13 tokens
0154 · filter · wording 2 · samec5c1e936a25b2d979e62ae3bfc6000eb61b2f10468a94d1e579187177760c662
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"V336","status":"closed"},{"id":"R797","status":"ready"},{"id":"U701","status":"ready"},{"id":"W194","status":"closed"},{"id":"H451","status":"ready"}]
Checked answer
{"ids": ["H451", "R797", "U701"]}
Seed

invalid_json_object

[{"id":"R797","status":"ready"},{"id":"U701","status":"ready"},{"id":"W194","status":"closed"},{"id":"H451","status":"ready"}]
51 tokens
1 GPU · AdamW

invalid_json_object

["H451","R797","U701"]
17 tokens
4 GPUs · DDP

Correct

{"ids":["H451","R797","U701"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["H451","R797","V336"]]
21 tokens
0155 · total · wording 2 · samea20b1903080c4e45e3c525b75497399e1764964bc40c86fea04b7a05e5d745ef
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"U280","units":10,"price":5},{"id":"C998","units":12,"price":8},{"id":"Q287","units":19,"price":7},{"id":"S419","units":1,"price":7},{"id":"K466","units":10,"price":13}]
Checked answer
{"total": 416}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":548}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0156 · sort · wording 2 · same090d4afae6d7fdf334d9283a740204f0e415ab7dc418ecbd8ebe360b6507ae8d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"M122","priority":1},{"id":"L776","priority":3},{"id":"L212","priority":5},{"id":"C237","priority":4},{"id":"R159","priority":9}]
Checked answer
{"ids": ["R159", "L212"]}
Seed

invalid_json_object

{"ids": ["M122", "L212", "C237", "R159", "L776", "L212", "L776", "L212", "L776", "L212", "L776", "L212", "L776", "L212", "L776", "L2
96 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["R159","C237"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["R159","C237"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["R159","C237"]}
16 tokens
0157 · lookup · wording 3 · gain4a8723f53c82f2f4f645eb9a89f9bca574825b2241a6facba2df6d8e0bb2dc31
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=E851. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"C314","city":"Riga","units":3},{"id":"E851","city":"Perth","units":17},{"id":"C799","city":"Perth","units":13},{"id":"F516","city":"Riga","units":13},{"id":"M257","city":"Bern","units":4}]
Checked answer
{"city": "Perth", "units": 17}
Seed

wrong_schema_or_value

{"city": "Riga", "units": 3}
14 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":17}
12 tokens
0158 · filter · wording 3 · gain70eb332c7f1b6b27ce66a48c3860fa5cf8c9c455884b516721ef7c768a5c16e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"U218","status":"pending"},{"id":"S752","status":"pending"},{"id":"E755","status":"closed"},{"id":"P742","status":"ready"},{"id":"Y907","status":"pending"}]
Checked answer
{"ids": ["P742"]}
Seed

wrong_schema_or_value

{"ids": ["U218", "S752", "E755", "P742", "Y907"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P742"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["P742"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["P742"]}
11 tokens
0159 · total · wording 3 · same3efa6b797b0c76a8530520d7a8f1cb20f9c8bfee976aa18a6086260c90ca7c9d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"E705","units":14,"price":15},{"id":"Q829","units":4,"price":19},{"id":"H123","units":7,"price":19},{"id":"N675","units":18,"price":7},{"id":"E113","units":19,"price":12}]
Checked answer
{"total": 773}
Seed

invalid_json_object

{
  "total": 15 + 72 + 133 + 114 + 135 = 427
}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0160 · sort · wording 3 · same933246a1b86dccf1ce6e0b780d13494547c95e2de624d0d8e5e7a715c124e03d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"N243","priority":7},{"id":"T512","priority":1},{"id":"I187","priority":3},{"id":"Z312","priority":6},{"id":"X332","priority":6}]
Checked answer
{"ids": ["N243", "X332"]}
Seed

invalid_json_object

[{"id":"N243","priority":7},{"id":"T512","priority":1},{"id":"I187","priority":3},{"id":"Z312","priority":6},{"id":"X332","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["N243","X332"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N243","X332"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N243","Z312"]}
16 tokens
0161 · lookup · wording 0 · gainb54bb6f9b69ee3ba3eb932b52c289bc7f3e338fa3953125af4325d6a4b691e6b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"X871","city":"Bern","units":15},{"id":"K341","city":"Suva","units":13},{"id":"B842","city":"Kyoto","units":16},{"id":"Z269","city":"Perth","units":9},{"id":"F471","city":"Bern","units":1}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Bern","units":15}]
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0162 · filter · wording 0 · gaindf36ed0ad563005f3b1730030176a427480d2cf7b4d2fcc4472197e3544f7841
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"G411","status":"closed"},{"id":"K648","status":"closed"},{"id":"O293","status":"pending"},{"id":"Q777","status":"ready"},{"id":"T179","status":"ready"}]
Checked answer
{"ids": ["Q777", "T179"]}
Seed

wrong_schema_or_value

{"ids": ["O293", "Q777", "T179", "G411", "K648"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Q777","T179"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Q777","T179"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q777","T179"]}
16 tokens
0163 · total · wording 0 · same5f3cbf06c3225d1db6997387166cd9e92d32fc84cda9a74edbd075b7b743e489
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"W347","units":2,"price":5},{"id":"Y441","units":9,"price":16},{"id":"Q195","units":19,"price":5},{"id":"B867","units":8,"price":13},{"id":"W615","units":8,"price":7}]
Checked answer
{"total": 409}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0164 · sort · wording 0 · gain191ad5393a8adb661d568755d1a1d61d3f8f2978b18051bff5341cbe3bedc9a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"C829","priority":4},{"id":"I723","priority":8},{"id":"F298","priority":9},{"id":"L400","priority":5},{"id":"X282","priority":2}]
Checked answer
{"ids": ["F298", "I723"]}
Seed

wrong_schema_or_value

{"ids": ["I723", "F298", "L400", "C829"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I723","F298"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F298","I723"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F298","I723"]}
16 tokens
0165 · lookup · wording 1 · gain702c49cdd8cd5bb0c8a936e99e303aac7247dff71eb8e6ee0d2757c573c5cb53
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z418 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M127","city":"Riga","units":16},{"id":"Z418","city":"Lima","units":14},{"id":"Z986","city":"Accra","units":9},{"id":"U983","city":"Bern","units":16},{"id":"W495","city":"Riga","units":10}]
Checked answer
{"city": "Lima", "units": 14}
Seed

wrong_schema_or_value

{"city":"Riga","units":16}
12 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":14}
12 tokens
0166 · filter · wording 1 · gain5ea505c7dbf3377cb52e47cca8e5d99b618a16e8d3698f9b3e1b38b5410ca4f5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"H710","status":"pending"},{"id":"Q416","status":"ready"},{"id":"P202","status":"pending"},{"id":"A315","status":"closed"},{"id":"B688","status":"pending"}]
Checked answer
{"ids": ["Q416"]}
Seed

wrong_schema_or_value

{
  "ids": ["A315", "B688", "H710", "Q416"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["Q416"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Q416"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Q416"]}
11 tokens
0167 · total · wording 1 · samea9ce6206f9a09aa451f4b7d409037cd146375c7b89e0d9f7fbdb936a3ad6a093
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"T854","units":17,"price":19},{"id":"S958","units":2,"price":8},{"id":"N243","units":8,"price":18},{"id":"Y942","units":12,"price":2},{"id":"Y468","units":0,"price":1}]
Checked answer
{"total": 507}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0168 · sort · wording 1 · gain81500d5e26bbbd640981cf59ede8171af75d023a353288673565cf7cd936b4eb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"C669","priority":5},{"id":"I236","priority":2},{"id":"W966","priority":9},{"id":"B275","priority":7},{"id":"A414","priority":4}]
Checked answer
{"ids": ["W966", "B275"]}
Seed

invalid_json_object

[{"id":"C669","priority":5},{"id":"I236","priority":2},{"id":"W966","priority":9},{"id":"B275","priority":7},{"id":"A414","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["W966","B275"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W966","C669"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["W966","B275"]}
16 tokens
0169 · lookup · wording 2 · gain059f1ccfd93ed217592cb13e3c0af5bb8c3e64e825739b8b30e16fbebaace252
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to T721? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"M348","city":"Lima","units":16},{"id":"T721","city":"Accra","units":13},{"id":"W388","city":"Riga","units":6},{"id":"Z819","city":"Lima","units":18},{"id":"G934","city":"Perth","units":17}]
Checked answer
{"city": "Accra", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":13}
12 tokens
0170 · filter · wording 2 · same1ef6a60f6e046fdfc1f26933770cb5d770a512d5abd3d7af564bed27a74ec029
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"T312","status":"closed"},{"id":"U500","status":"closed"},{"id":"G282","status":"ready"},{"id":"O348","status":"closed"},{"id":"K385","status":"ready"}]
Checked answer
{"ids": ["G282", "K385"]}
Seed

invalid_json_object

[{"id":"G282"},{"id":"K385"},{"id":"O348"},{"id":"T312"},{"id":"U500"}]
43 tokens
1 GPU · AdamW

invalid_json_object

["K385","G282"]
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["K385","G282"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G282","K385"]]
16 tokens
0171 · total · wording 2 · sameb67869c96f253104c8f97c384d0aa758067798a6e322e386f2eaf4cc9ea1f8b1
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"P974","units":19,"price":1},{"id":"Q253","units":12,"price":17},{"id":"H887","units":3,"price":15},{"id":"W935","units":19,"price":16},{"id":"J627","units":4,"price":17}]
Checked answer
{"total": 640}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0172 · sort · wording 2 · same887d53b503c295bb1a74c1e98d1f93726260556a4b113fc31c2c978f96b7d8be
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"T783","priority":6},{"id":"V728","priority":7},{"id":"P420","priority":2},{"id":"V918","priority":8},{"id":"S501","priority":5}]
Checked answer
{"ids": ["V918", "V728"]}
Seed

wrong_schema_or_value

{"ids": ["V728", "V918", "P420", "T783", "S501"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["V918","T783"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V728","T783"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["V918","T783"]}
16 tokens
0173 · lookup · wording 3 · gain8d2b8758e532fe68585446ec17b986b70146290f304986deed34def81acaee61
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=C185. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"R549","city":"Riga","units":2},{"id":"L675","city":"Oslo","units":17},{"id":"V632","city":"Kyoto","units":3},{"id":"D559","city":"Oslo","units":7},{"id":"C185","city":"Oslo","units":0}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

wrong_schema_or_value

{"city":"Riga","units":2}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":0}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":0}
10 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":0}
11 tokens
0174 · filter · wording 3 · gaind6899f60286d84d400446f140c355a4e218648a2b1031a86f0ab91ec21df2c4f
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"W732","status":"ready"},{"id":"F215","status":"closed"},{"id":"A130","status":"pending"},{"id":"W282","status":"pending"},{"id":"J985","status":"pending"}]
Checked answer
{"ids": ["W732"]}
Seed

wrong_schema_or_value

{"ids": ["A130", "F215", "J985", "W282", "W732"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["W732"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["W732"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["W732"]}
11 tokens
0175 · total · wording 3 · same96b22aef6a04630f86da899954109ea5822be40523db7b067d6177b534e6e112
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"D408","units":8,"price":5},{"id":"Z220","units":5,"price":15},{"id":"D617","units":8,"price":9},{"id":"Y257","units":14,"price":2},{"id":"G975","units":11,"price":10}]
Checked answer
{"total": 325}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0176 · sort · wording 3 · same804f0f4549963d4a66198303c1d256c94598a5bbc7cdb2922aee23e4a9cad10d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J821","priority":8},{"id":"J165","priority":7},{"id":"Q357","priority":5},{"id":"V944","priority":9},{"id":"D364","priority":4}]
Checked answer
{"ids": ["V944", "J821"]}
Seed

invalid_json_object

[{"id":"J821"},{"id":"J165"},{"id":"Q357"},{"id":"V944"},{"id":"D364"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["V944","J821"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["J821","V944"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J821","V944"]}
16 tokens
0177 · lookup · wording 0 · same64dc5ebeaa3441363a41c5235ed1f7bf761a7b83a20aededa384b0ed113e5d9d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J411. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"J411","city":"Riga","units":16},{"id":"M699","city":"Accra","units":13},{"id":"I998","city":"Kyoto","units":16},{"id":"G698","city":"Bern","units":3},{"id":"Y700","city":"Suva","units":3}]
Checked answer
{"city": "Riga", "units": 16}
Seed

Correct

{
  "city": "Riga",
  "units": 16
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":16}
12 tokens
0178 · filter · wording 0 · gainb29e1526504e1f6187fcbd4f6255dca1568a0db35c714397e30ebde2bb5cfaf6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"G547","status":"pending"},{"id":"M409","status":"ready"},{"id":"B935","status":"pending"},{"id":"H405","status":"closed"},{"id":"H304","status":"pending"}]
Checked answer
{"ids": ["M409"]}
Seed

wrong_schema_or_value

{"ids": ["B935", "G547", "M409", "H304", "H405"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M409"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M409"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M409"]}
11 tokens
0179 · total · wording 0 · sameb9df39b7db2b75f44464162988548ff2b49797a439c68ed2599b4ffbd3ab7569
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"F553","units":15,"price":11},{"id":"R844","units":8,"price":10},{"id":"O199","units":9,"price":15},{"id":"P188","units":17,"price":11},{"id":"V592","units":0,"price":16}]
Checked answer
{"total": 567}
Seed

invalid_json_object

{"total": 11*15 + 10*8 + 15*9 + 11*17 + 16*0 = 165}
41 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0180 · sort · wording 0 · gain81c7ffb681966ee7fdc0b04385c5f661c1f0ff10ae5863d0f5a61d5bcbaa9ef7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"W314","priority":7},{"id":"P612","priority":3},{"id":"M834","priority":2},{"id":"Q280","priority":3},{"id":"D889","priority":8}]
Checked answer
{"ids": ["D889", "W314"]}
Seed

wrong_schema_or_value

{"ids": ["P612", "W314", "D889", "M834", "Q280"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D889","W314"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D889","W314"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D889","W314"]}
16 tokens
0181 · lookup · wording 1 · gainb30bd295bec1291dc5ea82c39668c96b265f35c7b05a768c90d8c748c0f30ba7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"U255","city":"Kyoto","units":19},{"id":"P832","city":"Oslo","units":7},{"id":"R298","city":"Suva","units":1},{"id":"W902","city":"Suva","units":6},{"id":"Q640","city":"Accra","units":0}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":19}
13 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0182 · filter · wording 1 · gaind4ced6eeec367ea232cfe94a072b806864ade9e64db780cb8b4f8b5126cc51f4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"A364","status":"ready"},{"id":"D388","status":"ready"},{"id":"N316","status":"closed"},{"id":"E874","status":"pending"},{"id":"D422","status":"pending"}]
Checked answer
{"ids": ["A364", "D388"]}
Seed

wrong_schema_or_value

{
  "ids": ["A364", "D388", "N316", "E874", "D422"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["A364","D388"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A364","D388"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A364","D388"]}
16 tokens
0183 · total · wording 1 · same926293e3f103b640a09c6e4bf9795753724ffe95079a06b43fc5680e09d00b1c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"R897","units":1,"price":9},{"id":"V567","units":15,"price":10},{"id":"B117","units":8,"price":9},{"id":"W196","units":3,"price":2},{"id":"R980","units":11,"price":1}]
Checked answer
{"total": 248}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0184 · sort · wording 1 · samecd2ca2e709124c58462518699790cef1c1830ad07cbb95e817924c677db689e2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Y350","priority":3},{"id":"T798","priority":9},{"id":"K874","priority":5},{"id":"L856","priority":6},{"id":"V730","priority":7}]
Checked answer
{"ids": ["T798", "V730"]}
Seed

invalid_json_object

[{"id":"Y350","priority":3},{"id":"V730","priority":7}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["T798","V730"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["T798","V730"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["T798","K874"]}
16 tokens
0185 · lookup · wording 2 · gain34621f64454334498ea64a8eea22ed290a300397a596560a0c1e280c46de821a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to K850? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"K850","city":"Kyoto","units":2},{"id":"J191","city":"Oslo","units":3},{"id":"S905","city":"Kyoto","units":7},{"id":"G431","city":"Bern","units":16},{"id":"W275","city":"Oslo","units":14}]
Checked answer
{"city": "Kyoto", "units": 2}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":2}
12 tokens
0186 · filter · wording 2 · samef4fb68f53928683fc2598b7dab25f74ce394c30b337315fc2823de1eb0371e9a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"M330","status":"pending"},{"id":"T388","status":"closed"},{"id":"F764","status":"pending"},{"id":"J614","status":"ready"},{"id":"G113","status":"closed"}]
Checked answer
{"ids": ["J614"]}
Seed

invalid_json_object

[{"id":"J614","status":"ready"},{"id":"G113","status":"closed"},{"id":"M330","status":"pending"},{"id":"T388","status":"closed"},{"id":"F764","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["J614"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["J614"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["J614"]]
11 tokens
0187 · total · wording 2 · same9c7da9fc516dab5664db7ca5b38dd821823c97c29a96def184cd79172af59f0e
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"U948","units":4,"price":7},{"id":"P799","units":4,"price":7},{"id":"V292","units":2,"price":10},{"id":"T401","units":14,"price":13},{"id":"E840","units":4,"price":16}]
Checked answer
{"total": 322}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0188 · sort · wording 2 · gain97518838e88515064bd48fac6e82ff7f6ff07cce68ca2cfe412dc280a0afdeb9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"R801","priority":4},{"id":"K978","priority":6},{"id":"G245","priority":7},{"id":"J766","priority":9},{"id":"T811","priority":6}]
Checked answer
{"ids": ["J766", "G245"]}
Seed

wrong_schema_or_value

{"ids": ["R801", "K978", "G245", "J766", "T811"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J766","G245"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J766","G245"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J766","G245"]}
16 tokens
0189 · lookup · wording 3 · samefcf8dab25d4ae0a332ac5b486fba51c4123b9566461804c4364c3d81b43198ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=R859. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"V186","city":"Kyoto","units":8},{"id":"X837","city":"Oslo","units":10},{"id":"Z922","city":"Accra","units":1},{"id":"R859","city":"Bern","units":5},{"id":"R423","city":"Suva","units":16}]
Checked answer
{"city": "Bern", "units": 5}
Seed

Correct

{"city":"Bern","units":5}
10 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":5}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":5}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":5}
10 tokens
0190 · filter · wording 3 · gain2bdffbaad33b1095d66f8d44729c39d77f4bc1f453592e96fbc3054d1a980def
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Q542","status":"closed"},{"id":"P244","status":"pending"},{"id":"C582","status":"pending"},{"id":"O283","status":"closed"},{"id":"J765","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["C582", "J765", "P244", "O283", "Q542"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0191 · total · wording 3 · same7b6abe15db6d581f1364d4de8e721a37d6af69b70aab33f081d245c5044ec01c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"O517","units":10,"price":9},{"id":"D896","units":2,"price":3},{"id":"V778","units":0,"price":14},{"id":"J149","units":4,"price":16},{"id":"Q856","units":16,"price":12}]
Checked answer
{"total": 352}
Seed

invalid_json_object

{
  "total": 10 * 9 + 2 * 3 + 0 * 14 + 4 * 16 + 16 * 12
}
42 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0192 · sort · wording 3 · same501e3740d5a302b331fa0e4fbf36a8603a358362e6b771fc9b1dec31638e573e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"K820","priority":9},{"id":"N659","priority":4},{"id":"L981","priority":9},{"id":"E808","priority":3},{"id":"A179","priority":4}]
Checked answer
{"ids": ["K820", "L981"]}
Seed

invalid_json_object

[{"id":"K820","priority":9},{"id":"N659","priority":4},{"id":"L981","priority":9},{"id":"E808","priority":3},{"id":"A179","priority":4}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["L981","K820"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L981","K820"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["L981","K820"]}
16 tokens
0193 · lookup · wording 0 · gain11135f56a153564b33715ed60d4305dd80e6049ac8a67bc4af87cbcc7d7d4b3f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Y129. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y836","city":"Accra","units":12},{"id":"X765","city":"Accra","units":19},{"id":"B191","city":"Perth","units":14},{"id":"S184","city":"Accra","units":13},{"id":"Y129","city":"Accra","units":7}]
Checked answer
{"city": "Accra", "units": 7}
Seed

invalid_json_object

[{"city":"Accra","units":12}]
13 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":7}
11 tokens
0194 · filter · wording 0 · same79d27a083da29d196929d2b25543d84d07abf94c74d9329a1b2aab5d3b0c01ea
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"X108","status":"ready"},{"id":"V419","status":"closed"},{"id":"V112","status":"ready"},{"id":"U180","status":"pending"},{"id":"D990","status":"pending"}]
Checked answer
{"ids": ["V112", "X108"]}
Seed

wrong_schema_or_value

{"ids": ["X108", "V112", "U180", "D990"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X108","V112"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X108","V112"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["X108","V112"]}
16 tokens
0195 · total · wording 0 · samecc57254637b308be3d8e6c247270404dfb85e167c51084bc4523dd0ee05f58af
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"G170","units":17,"price":13},{"id":"W206","units":19,"price":13},{"id":"Y795","units":11,"price":15},{"id":"U350","units":10,"price":16},{"id":"U792","units":13,"price":2}]
Checked answer
{"total": 819}
Seed

wrong_schema_or_value

{"total": 130}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0196 · sort · wording 0 · same6811e3098deee1cda17aba54b5910cd72b33ed502cc03ad1bd4da1f68563b820
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"R222","priority":1},{"id":"Y946","priority":3},{"id":"H716","priority":5},{"id":"P546","priority":6},{"id":"B745","priority":9}]
Checked answer
{"ids": ["B745", "P546"]}
Seed

wrong_schema_or_value

{"ids": ["R222", "Y946", "H716", "P546", "B745"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B745","P546"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["B745","H716"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["B745","H716"]}
16 tokens
0197 · lookup · wording 1 · sameddffb654704c29bf4b9af328cb3bd3beaed9a4d6c91e754dab4cb15fe40711a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id A660 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"F697","city":"Perth","units":4},{"id":"E917","city":"Suva","units":19},{"id":"I252","city":"Kyoto","units":11},{"id":"D948","city":"Bern","units":15},{"id":"A660","city":"Riga","units":18}]
Checked answer
{"city": "Riga", "units": 18}
Seed

Correct

{"city":"Riga","units":18}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":18}
12 tokens
0198 · filter · wording 1 · gain3c7a99bcfbad526785d1a8751a1b2c10fab10c1a4f4b2f352de85551cf15f812
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"B458","status":"closed"},{"id":"U662","status":"pending"},{"id":"M311","status":"ready"},{"id":"E912","status":"ready"},{"id":"T753","status":"ready"}]
Checked answer
{"ids": ["E912", "M311", "T753"]}
Seed

wrong_schema_or_value

{
  "ids": ["B458", "M311", "E912", "T753"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["E912","M311","T753"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["E912","M311","T753"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["E912","M311","T753"]}
21 tokens
0199 · total · wording 1 · same259259a1b29e55d418d72ddb70cd18b0355b83d675cc8d1f4804ddc376c986a0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"M754","units":6,"price":17},{"id":"I597","units":6,"price":15},{"id":"A403","units":7,"price":5},{"id":"U910","units":12,"price":5},{"id":"O840","units":3,"price":16}]
Checked answer
{"total": 335}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":278}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0200 · sort · wording 1 · gain8579ddf307032a5b1e4711ccbaacc0497d7c1ea5b64ab875e84e435bfbafa7a5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"G740","priority":3},{"id":"P538","priority":9},{"id":"O237","priority":9},{"id":"F189","priority":4},{"id":"G539","priority":3}]
Checked answer
{"ids": ["O237", "P538"]}
Seed

invalid_json_object

[{"id":"G740","priority":3},{"id":"P538","priority":9},{"id":"O237","priority":9},{"id":"F189","priority":4},{"id":"G539","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["O237","P538"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O237","P538"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["O237","P538"]}
16 tokens
0201 · lookup · wording 2 · same73c4845f4bdf29561bfef97c67602f41a6a0912720aa2d03222452583223e2a6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E774","city":"Suva","units":14},{"id":"E986","city":"Bern","units":4},{"id":"P886","city":"Kyoto","units":16},{"id":"K208","city":"Accra","units":19},{"id":"P762","city":"Perth","units":17}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0202 · filter · wording 2 · same0ff414a17d3f5ad73a6248560f51d8e1eef8c9aef5c44b5f198b2b77a7c495a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"E674","status":"closed"},{"id":"F246","status":"pending"},{"id":"T956","status":"ready"},{"id":"L641","status":"pending"},{"id":"T476","status":"closed"}]
Checked answer
{"ids": ["T956"]}
Seed

invalid_json_object

[{"id":"T476","status":"closed"},{"id":"T956","status":"ready"},{"id":"F246","status":"pending"},{"id":"E674","status":"closed"}]
51 tokens
1 GPU · AdamW

Correct

{"ids":["T956"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T956"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["T956"]]
11 tokens
0203 · total · wording 2 · samec31975cfe20310d77075f166b8ef0099917ab2fa2bdb1d5ceaebbefde5d636e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y997","units":11,"price":7},{"id":"E274","units":18,"price":9},{"id":"N337","units":8,"price":10},{"id":"K891","units":11,"price":1},{"id":"N305","units":0,"price":13}]
Checked answer
{"total": 330}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0204 · sort · wording 2 · gain9d7f4fa5a70a24b295cd595501787e80bba2c403ce23f28d47436d83bfa744b1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J661","priority":8},{"id":"Q599","priority":5},{"id":"T927","priority":7},{"id":"V245","priority":2},{"id":"H524","priority":9}]
Checked answer
{"ids": ["H524", "J661"]}
Seed

invalid_json_object

{"ids": ["J661", "V245", "H524", "T927", "Q599", "V245", "H524", "T927", "Q599", "V245", "H524", "T927", "Q599", "V245", "H524", "T9
96 tokens
1 GPU · AdamW

Correct

{"ids":["H524","J661"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H524","J661"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H524","J661"]}
16 tokens
0205 · lookup · wording 3 · gainba82ba6ce4244cb8ea3b43b56e9d4dbfc86ac395f55ed3c01870e98446c6628f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=B434. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Y961","city":"Riga","units":0},{"id":"K255","city":"Riga","units":14},{"id":"L868","city":"Riga","units":3},{"id":"E747","city":"Accra","units":11},{"id":"B434","city":"Kyoto","units":4}]
Checked answer
{"city": "Kyoto", "units": 4}
Seed

wrong_schema_or_value

{"city": "Riga", "units": 14}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":4}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":4}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":4}
12 tokens
0206 · filter · wording 3 · same9a82afaafe2329394e03e690b7d477169b92df662bca2fdf2edde93ed10a1737
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"A902","status":"closed"},{"id":"M630","status":"pending"},{"id":"W637","status":"closed"},{"id":"T972","status":"ready"},{"id":"Z326","status":"ready"}]
Checked answer
{"ids": ["T972", "Z326"]}
Seed

wrong_schema_or_value

{"ids": ["A902", "M630", "W637", "T972", "Z326"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["T972","Z326"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["T972","Z326"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z326","T972"]}
16 tokens
0207 · total · wording 3 · same7ff4614020cc12af3d2220dce95f033af650be6fe1423fce0730d298c3cf1e5a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"H516","units":10,"price":6},{"id":"D762","units":1,"price":4},{"id":"U351","units":12,"price":1},{"id":"X917","units":12,"price":14},{"id":"V868","units":8,"price":15}]
Checked answer
{"total": 364}
Seed

invalid_json_object

{
  "total": 10 * 6 + 1 * 4 + 12 * 1 + 12 * 14 + 8 * 15
}
42 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":388}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0208 · sort · wording 3 · gain961bd2dafd5f60a418d87d387d46fef48c562fa74c32ed8a0b357d90dcc25d00
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"G746","priority":9},{"id":"P121","priority":9},{"id":"B978","priority":6},{"id":"Q508","priority":8},{"id":"P706","priority":5}]
Checked answer
{"ids": ["G746", "P121"]}
Seed

invalid_json_object

[{"id":"P706","priority":5},{"id":"P121","priority":9},{"id":"B978","priority":6},{"id":"G746","priority":9},{"id":"Q508","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["G746","P121"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G746","P121"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G746","P121"]}
16 tokens
0209 · lookup · wording 0 · same6bd84b5d89e35dbad57382afe3fa1ac51ed93d20f2d3fe7635d8eec88d60fd31
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record O675. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"O675","city":"Oslo","units":9},{"id":"P458","city":"Oslo","units":5},{"id":"X647","city":"Oslo","units":9},{"id":"K952","city":"Kyoto","units":16},{"id":"N481","city":"Lima","units":12}]
Checked answer
{"city": "Oslo", "units": 9}
Seed

Correct

{
  "city": "Oslo",
  "units": 9
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":9}
11 tokens
0210 · filter · wording 0 · gain0c7cd4bc6a24d592835dec88b71cdaa6bc3fd6f9a3e57c55f83337cf2985380c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P606","status":"closed"},{"id":"Q486","status":"ready"},{"id":"J629","status":"pending"},{"id":"O349","status":"pending"},{"id":"I215","status":"ready"}]
Checked answer
{"ids": ["I215", "Q486"]}
Seed

wrong_schema_or_value

{"ids": ["J629", "O349", "P606", "Q486", "I215"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["I215","Q486"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I215","Q486"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I215","Q486"]}
16 tokens
0211 · total · wording 0 · same30e38d9088d87364b65695c97058f17862e57300f4d78b9ab83d89f113178c34
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"X585","units":4,"price":15},{"id":"U277","units":19,"price":18},{"id":"K451","units":7,"price":19},{"id":"O361","units":11,"price":9},{"id":"K726","units":7,"price":17}]
Checked answer
{"total": 753}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0212 · sort · wording 0 · gaindd126d55e58d79c1a06fce959302c6a2858417275d045cf80386e9218c01abd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"X598","priority":2},{"id":"Z990","priority":4},{"id":"P918","priority":3},{"id":"S125","priority":2},{"id":"Y123","priority":2}]
Checked answer
{"ids": ["Z990", "P918"]}
Seed

wrong_schema_or_value

{"ids": ["X598", "Y123", "S125", "Z990"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P918","Z990"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Z990","P918"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z990","P918"]}
16 tokens
0213 · lookup · wording 1 · same4e454198fdfd30f892937b9367cf49b9f7ea825dc0549b6496e626e8da33e9a3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id O723 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"K312","city":"Bern","units":14},{"id":"Y150","city":"Oslo","units":1},{"id":"M373","city":"Lima","units":12},{"id":"O723","city":"Bern","units":2},{"id":"O832","city":"Lima","units":4}]
Checked answer
{"city": "Bern", "units": 2}
Seed

Correct

{"city":"Bern","units":2}
10 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":2}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":2}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":2}
10 tokens
0214 · filter · wording 1 · gain4352a9e8867520fd5b7c36ad655ad9b6304a860cb63d2e7d36f0c674d56f8073
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"G472","status":"closed"},{"id":"E603","status":"ready"},{"id":"A439","status":"closed"},{"id":"Y251","status":"pending"},{"id":"Z144","status":"ready"}]
Checked answer
{"ids": ["E603", "Z144"]}
Seed

wrong_schema_or_value

{
  "ids": ["A439", "E603", "G472", "Y251"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["E603","Z144"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E603","Z144"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E603","Z144"]}
16 tokens
0215 · total · wording 1 · same158d172b9931a055380acfe4b1730fb8331f7989147cafd7b29a343648aaf4ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D938","units":6,"price":13},{"id":"Q422","units":19,"price":19},{"id":"M196","units":18,"price":1},{"id":"V342","units":15,"price":6},{"id":"N312","units":14,"price":17}]
Checked answer
{"total": 785}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0216 · sort · wording 1 · sameaf8867c0e8849034a83ef9e9e8db020b769f4d258b9b222b0ed01d32139a2482
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"G297","priority":8},{"id":"V199","priority":3},{"id":"E560","priority":2},{"id":"R783","priority":2},{"id":"O329","priority":2}]
Checked answer
{"ids": ["G297", "V199"]}
Seed

invalid_json_object

[{"id":"G297","priority":8},{"id":"V199","priority":3},{"id":"E560","priority":2},{"id":"R783","priority":2},{"id":"O329","priority":2}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["G297","E560"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["G297","E560"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G297","E560"]}
16 tokens
0217 · lookup · wording 2 · gain4931196d23aac54f4dc123abb24effdeb4f51ba134c74d1fb785629ba52a0987
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to F977? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T485","city":"Perth","units":18},{"id":"J646","city":"Perth","units":2},{"id":"L357","city":"Lima","units":7},{"id":"A363","city":"Riga","units":7},{"id":"F977","city":"Kyoto","units":17}]
Checked answer
{"city": "Kyoto", "units": 17}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":17}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":17}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":17}
13 tokens
0218 · filter · wording 2 · samed1d656c9aed8b89ea126efc3505f3e2c100d7d547f978b5c0db44f324b2d85e4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Q269","status":"pending"},{"id":"S805","status":"pending"},{"id":"A911","status":"pending"},{"id":"T568","status":"ready"},{"id":"I395","status":"closed"}]
Checked answer
{"ids": ["T568"]}
Seed

invalid_json_object

[{"id":"T568","status":"ready"}]
15 tokens
1 GPU · AdamW

Correct

{"ids":["T568"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T568"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["T568"]]
11 tokens
0219 · total · wording 2 · same321ed83597f5eaee3911af19dbf813a06a2c8d82edbe4f5ff4eb983e516073bc
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"M505","units":15,"price":6},{"id":"J623","units":12,"price":12},{"id":"K831","units":16,"price":6},{"id":"E456","units":6,"price":9},{"id":"U294","units":14,"price":8}]
Checked answer
{"total": 496}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0220 · sort · wording 2 · gain23142c011a1953589c0e3285815ca1ec93c2f5371a85343fafdb554403ce6629
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"L280","priority":1},{"id":"P533","priority":7},{"id":"B284","priority":8},{"id":"F368","priority":6},{"id":"N502","priority":5}]
Checked answer
{"ids": ["B284", "P533"]}
Seed

wrong_schema_or_value

{"ids": ["L280", "P533", "B284", "F368", "N502"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B284","P533"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B284","P533"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B284","P533"]}
16 tokens
0221 · lookup · wording 3 · gainf0dc4ceb2e8da2e40299fff3583da28203133b512f40c66c3a64ee17a34b8743
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Z703","city":"Accra","units":15},{"id":"B387","city":"Riga","units":8},{"id":"M704","city":"Bern","units":1},{"id":"D492","city":"Accra","units":9},{"id":"H133","city":"Oslo","units":11}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Accra","units":15}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0222 · filter · wording 3 · gain764b526e5ca3cece144eb93d9e5a8a441ad24e3e7938ec98328478fd245d74e6
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Y794","status":"ready"},{"id":"R346","status":"closed"},{"id":"M694","status":"closed"},{"id":"E287","status":"pending"},{"id":"O444","status":"ready"}]
Checked answer
{"ids": ["O444", "Y794"]}
Seed

wrong_schema_or_value

{"ids": ["O444", "Y794", "R346", "M694", "E287"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["E287","Y794","O444"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E287","O444","Y794"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["O444","Y794"]}
16 tokens
0223 · total · wording 3 · same0f976bf1425648033ce6b392dd80da7f7ce862dd9d0cbeff7c052ccd171ca8a9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Y807","units":5,"price":17},{"id":"M994","units":2,"price":17},{"id":"B224","units":17,"price":18},{"id":"C629","units":18,"price":5},{"id":"L202","units":6,"price":1}]
Checked answer
{"total": 521}
Seed

invalid_json_object

{
  "total": 17 + 34 + 30 + 95 + 108
}
28 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0224 · sort · wording 3 · gainb705a8ec72c768a8635590b7ef662815e269c08a7586e9c19769c2d88a4ddab4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"B486","priority":4},{"id":"Y489","priority":5},{"id":"S834","priority":1},{"id":"E362","priority":6},{"id":"S397","priority":3}]
Checked answer
{"ids": ["E362", "Y489"]}
Seed

invalid_json_object

[{"id":"S397","priority":3},{"id":"S834","priority":1},{"id":"Y489","priority":5},{"id":"B486","priority":4},{"id":"E362","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["E362","Y489"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E362","Y489"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E362","Y489"]}
16 tokens
0225 · lookup · wording 0 · same3e83ede9a87a46ce84467330e77c2fe8ef34a84eb257f04ff6254b0fa4738a90
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Y963. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W495","city":"Suva","units":8},{"id":"Y963","city":"Riga","units":13},{"id":"S208","city":"Suva","units":7},{"id":"A698","city":"Lima","units":2},{"id":"R398","city":"Kyoto","units":18}]
Checked answer
{"city": "Riga", "units": 13}
Seed

Correct

{
  "city": "Riga",
  "units": 13
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":13}
12 tokens
0226 · filter · wording 0 · gainff746a73ab7550fb03a31f996cb8bd0cdb6ef023c20a98a85efa626b9d9780bc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"A433","status":"closed"},{"id":"S345","status":"ready"},{"id":"Q532","status":"pending"},{"id":"M350","status":"pending"},{"id":"E536","status":"pending"}]
Checked answer
{"ids": ["S345"]}
Seed

wrong_schema_or_value

{"ids": ["S345", "M350", "E536", "A433", "Q532"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["S345"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["S345"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["S345"]}
11 tokens
0227 · total · wording 0 · samecbe40129ffe4bf8c9de6fbbe36cd08b68d2e56a5819ea7f38a6ca3a74b3f0b46
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"S721","units":10,"price":1},{"id":"R676","units":11,"price":1},{"id":"I601","units":14,"price":1},{"id":"Y566","units":12,"price":19},{"id":"Y105","units":4,"price":1}]
Checked answer
{"total": 267}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0228 · sort · wording 0 · same5c6fbfc358ef1932cbc7db933ecfb9484c3171dda36b29313b614aab3a3a65bf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J421","priority":6},{"id":"R989","priority":5},{"id":"O640","priority":8},{"id":"F552","priority":9},{"id":"K905","priority":2}]
Checked answer
{"ids": ["F552", "O640"]}
Seed

wrong_schema_or_value

{"ids": ["R989", "J421", "O640", "F552", "K905"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F552","O640"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F552","O640"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F552","J421"]}
16 tokens
0229 · lookup · wording 1 · same9229bba90f1b1d8f239ee052cdf375d8ceb133f57799417112bff4b1740a3221
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Y665 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"O304","city":"Accra","units":3},{"id":"D164","city":"Bern","units":3},{"id":"I811","city":"Lima","units":4},{"id":"Y665","city":"Oslo","units":0},{"id":"L402","city":"Perth","units":3}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

Correct

{"city":"Oslo","units":0}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":0}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":0}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":0}
11 tokens
0230 · filter · wording 1 · same6deaa15edb319a0186468cfea1f3dbcec7f5a50dd2ac460daf4234b40b9a94ef
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"P280","status":"ready"},{"id":"J438","status":"ready"},{"id":"K781","status":"ready"},{"id":"I804","status":"closed"},{"id":"L317","status":"pending"}]
Checked answer
{"ids": ["J438", "K781", "P280"]}
Seed

wrong_schema_or_value

{
  "ids": ["P280", "J438", "K781"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":["J438","K781","P280"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["J438","K781","P280"]}
21 tokens
4 GPUs · local windows

invalid_json_object

{"ids":["J438","K781","P280"]]}
22 tokens
0231 · total · wording 1 · same3eb6018306b63790641fa91ca49de74aca2782a2b8ea95f27ef00d0482d0988f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"H940","units":5,"price":18},{"id":"P420","units":3,"price":15},{"id":"E759","units":11,"price":12},{"id":"V715","units":16,"price":15},{"id":"U710","units":19,"price":12}]
Checked answer
{"total": 735}
Seed

wrong_schema_or_value

{"total": 108}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0232 · sort · wording 1 · same8b01eaca2668dd1b995bb7d7fe46c40fbe504933295e42c0eb744dcfef5d27ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"T961","priority":9},{"id":"I566","priority":4},{"id":"N476","priority":6},{"id":"N584","priority":9},{"id":"T694","priority":2}]
Checked answer
{"ids": ["N584", "T961"]}
Seed

invalid_json_object

[{"id":"T961","priority":9},{"id":"N584","priority":9},{"id":"N476","priority":6},{"id":"I566","priority":4},{"id":"T694","priority":2}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["N584","T961"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N584","T961"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N476","T961"]}
16 tokens
0233 · lookup · wording 2 · gain28534ff2307292cf590b3edf10230cf401eaa06b9e519d7cab59eec5f70397e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M255? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"R102","city":"Accra","units":18},{"id":"H709","city":"Accra","units":14},{"id":"I483","city":"Lima","units":19},{"id":"A392","city":"Riga","units":18},{"id":"M255","city":"Perth","units":0}]
Checked answer
{"city": "Perth", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":0}
11 tokens
0234 · filter · wording 2 · same167d1969a35594a0f857e2ae813d018d5436e89e35680ae6c434d73cbd7f2d37
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"D721","status":"pending"},{"id":"M690","status":"pending"},{"id":"Q960","status":"closed"},{"id":"G575","status":"ready"},{"id":"F989","status":"closed"}]
Checked answer
{"ids": ["G575"]}
Seed

invalid_json_object

[{"id":"G575"},{"id":"F989"},{"id":"M690"},{"id":"Q960"},{"id":"D721"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["G575"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["G575"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G575"]]
11 tokens
0235 · total · wording 2 · same5fb88d0068da4fdf9896200ed77008ed5a11a0edbd8b2a830d2f14e1e1bd2a79
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"H523","units":4,"price":17},{"id":"N492","units":7,"price":19},{"id":"G584","units":3,"price":19},{"id":"C398","units":4,"price":8},{"id":"Q632","units":4,"price":14}]
Checked answer
{"total": 346}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":278}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0236 · sort · wording 2 · same3ef94137ebb0942cc943b0f922b5891780b37ad014ad744f4c5a81bec2c9f77b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K875","priority":3},{"id":"E783","priority":6},{"id":"Z206","priority":5},{"id":"H162","priority":3},{"id":"S969","priority":1}]
Checked answer
{"ids": ["E783", "Z206"]}
Seed

wrong_schema_or_value

{"ids": ["K875", "E783", "Z206", "H162", "S969"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["E783","K875"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E783","K875"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E783","K875"]}
16 tokens
0237 · lookup · wording 3 · same7e150ff7995a0a16e2b85c6f8e039869ccf6e24ff43af9c27069c908154f3c8e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T348. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"C506","city":"Accra","units":10},{"id":"U750","city":"Lima","units":5},{"id":"T348","city":"Lima","units":18},{"id":"Y591","city":"Suva","units":17},{"id":"B547","city":"Accra","units":8}]
Checked answer
{"city": "Lima", "units": 18}
Seed

Correct

{"city": "Lima", "units": 18}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":18}
12 tokens
0238 · filter · wording 3 · gain05a8a3fa9a84f5aef56ecadf937c78074c4798e7ad21ab78f092b091276bfbe0
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"C156","status":"closed"},{"id":"H363","status":"closed"},{"id":"I375","status":"closed"},{"id":"U987","status":"pending"},{"id":"N487","status":"ready"}]
Checked answer
{"ids": ["N487"]}
Seed

wrong_schema_or_value

{"ids": ["I375", "N487", "U987", "C156", "H363"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["N487"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["N487"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["N487"]}
11 tokens
0239 · total · wording 3 · sameb5dd181eb7ff909fc94b5103e9a7323dfde1978d618ce88714bd9654b01df0f8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"K721","units":1,"price":3},{"id":"B477","units":5,"price":14},{"id":"O833","units":5,"price":12},{"id":"W317","units":7,"price":16},{"id":"Y787","units":19,"price":13}]
Checked answer
{"total": 492}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0240 · sort · wording 3 · gainb3d5e019f59bff228dd5606329c5fbf509f322528a718427a487aaa922440bc3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"W535","priority":3},{"id":"E898","priority":9},{"id":"Z680","priority":6},{"id":"Y596","priority":9},{"id":"M938","priority":2}]
Checked answer
{"ids": ["E898", "Y596"]}
Seed

invalid_json_object

[{"id":"W535","priority":3},{"id":"Y596","priority":9},{"id":"M938","priority":2}]
38 tokens
1 GPU · AdamW

Correct

{"ids":["E898","Y596"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E898","Y596"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E898","Y596"]}
16 tokens
0241 · lookup · wording 0 · gain687348b5aeaa914d8ab0671126cbb2a82a3d40f0902f7e671fea8c9f74bebe8b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W340","city":"Suva","units":7},{"id":"M361","city":"Riga","units":6},{"id":"I752","city":"Accra","units":16},{"id":"M331","city":"Kyoto","units":5},{"id":"H364","city":"Kyoto","units":15}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Suva","units":7},{"city":"Riga","units":6},{"city":"Accra","units":16},{"city":"Kyoto","units":5},{"city":"Kyoto","units":15}]
56 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0242 · filter · wording 0 · gaincdb7c8f02080be0b12a86b20d965f96213957be61640813cc30cd15d490f92a1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Q700","status":"pending"},{"id":"I771","status":"closed"},{"id":"V949","status":"pending"},{"id":"U862","status":"closed"},{"id":"S575","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["U862", "S575", "V949", "I771", "Q700"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0243 · total · wording 0 · samea1ae86644b1d63e970559bd5b822960170a3f6da3364757fc75b28d544406187
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Q169","units":13,"price":15},{"id":"W455","units":17,"price":9},{"id":"X370","units":10,"price":12},{"id":"G597","units":18,"price":14},{"id":"J111","units":15,"price":1}]
Checked answer
{"total": 735}
Seed

invalid_json_object

{"total": 13*15 + 17*9 + 10*12 + 18*14 + 15*1}
37 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0244 · sort · wording 0 · gain6f3d3a3dd16daa8bcc62755eaebf9b4aa9230a9b5c45594ffb49d3687c52e097
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z843","priority":6},{"id":"B686","priority":6},{"id":"X715","priority":9},{"id":"P657","priority":1},{"id":"P567","priority":3}]
Checked answer
{"ids": ["X715", "B686"]}
Seed

wrong_schema_or_value

{"ids": ["X715", "P657", "P567", "B686", "Z843"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X715","Z843"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X715","Z843"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X715","B686"]}
16 tokens
0245 · lookup · wording 1 · gaind02a08c8c634bedc05ecdd51309ed3200b06aaf53d62cabdd32eb3c782cf7acf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id O189 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"Z162","city":"Perth","units":1},{"id":"M397","city":"Lima","units":15},{"id":"J273","city":"Oslo","units":15},{"id":"S575","city":"Lima","units":3},{"id":"O189","city":"Kyoto","units":0}]
Checked answer
{"city": "Kyoto", "units": 0}
Seed

wrong_schema_or_value

{"city":"Perth","units":1}
11 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":0}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":0}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":0}
12 tokens
0246 · filter · wording 1 · samed6e8025c14011442f820ba67fe24b070dec5607874b3a84cddc1b934c0d72d12
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"B125","status":"pending"},{"id":"V162","status":"ready"},{"id":"D704","status":"ready"},{"id":"R422","status":"ready"},{"id":"M173","status":"ready"}]
Checked answer
{"ids": ["D704", "M173", "R422", "V162"]}
Seed

wrong_schema_or_value

{
  "ids": ["B125", "D704", "M173", "R422"]
}
32 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D704","M173","R422"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["D704","M173","R422","V162"]}
26 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D704","M173","V162"]}
21 tokens
0247 · total · wording 1 · sameef971225cbf69d4db3ef91ada9e651ac8ae0bc6a90fb3ad9edd03aa781ee9a81
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"K850","units":7,"price":6},{"id":"Z482","units":10,"price":16},{"id":"N867","units":2,"price":9},{"id":"P327","units":12,"price":14},{"id":"B974","units":6,"price":2}]
Checked answer
{"total": 400}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0248 · sort · wording 1 · gain0a2934936afc799965616c9ad4a5d9f54f40929e17330b869a1ceb39d21cce17
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"H115","priority":7},{"id":"D458","priority":8},{"id":"S349","priority":3},{"id":"I532","priority":6},{"id":"F475","priority":6}]
Checked answer
{"ids": ["D458", "H115"]}
Seed

invalid_json_object

[{"id":"H115","priority":7},{"id":"D458","priority":8},{"id":"S349","priority":3},{"id":"I532","priority":6},{"id":"F475","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["D458","H115"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D458","H115"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D458","H115"]}
16 tokens
0249 · lookup · wording 2 · gaina68c1e73a952520b897a3b97ce946fc8ccbe6d6d771442e0ee0b1fdfed1f4d12
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Y839? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"W719","city":"Bern","units":6},{"id":"X629","city":"Lima","units":16},{"id":"Y839","city":"Suva","units":10},{"id":"N495","city":"Kyoto","units":0},{"id":"U511","city":"Suva","units":0}]
Checked answer
{"city": "Suva", "units": 10}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":10}
12 tokens
0250 · filter · wording 2 · samedcb2594da65252ce7d6b3d90123e9d8bca8730bcb28fb2518bf4b49bfedbd52b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"R588","status":"closed"},{"id":"W378","status":"ready"},{"id":"X176","status":"ready"},{"id":"M154","status":"pending"},{"id":"M190","status":"pending"}]
Checked answer
{"ids": ["W378", "X176"]}
Seed

invalid_json_object

[{"id":"W378","status":"ready"},{"id":"X176","status":"ready"},{"id":"M154","status":"pending"},{"id":"M190","status":"pending"}]
51 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X176","W378"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["W378","X176"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["W378","X176"]]
16 tokens
0251 · total · wording 2 · samebb006136f42350500681eb7f915ce101f24492c38596b75647fccec92f7d23ba
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R283","units":16,"price":9},{"id":"Q870","units":13,"price":11},{"id":"U210","units":10,"price":19},{"id":"A640","units":19,"price":16},{"id":"N399","units":14,"price":13}]
Checked answer
{"total": 963}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0252 · sort · wording 2 · gain03b30e847d4f360c4c0cec41caf6ba2188d26847765caf02489eebd3b4a3d66a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"X339","priority":7},{"id":"E781","priority":5},{"id":"V146","priority":3},{"id":"W826","priority":1},{"id":"X481","priority":3}]
Checked answer
{"ids": ["X339", "E781"]}
Seed

wrong_schema_or_value

{"ids": ["X339", "V146", "W826", "X481", "E781"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["X339","E781"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["X339","E781"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X339","E781"]}
16 tokens
0253 · lookup · wording 3 · gaine1e6f4830cfa93b5ad5591eda10e572217951fa41ec13c384ee0d8a0f46306b7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=O242. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"D738","city":"Suva","units":3},{"id":"M362","city":"Kyoto","units":11},{"id":"O813","city":"Lima","units":8},{"id":"L165","city":"Riga","units":3},{"id":"O242","city":"Perth","units":13}]
Checked answer
{"city": "Perth", "units": 13}
Seed

wrong_schema_or_value

{"city":"Suva","units":3}
11 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":13}
12 tokens
0254 · filter · wording 3 · gain5c47384f822b7fd86ee447081375714293ab787b041d1b09d465e5c6d73358bf
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"T874","status":"closed"},{"id":"S878","status":"closed"},{"id":"A302","status":"pending"},{"id":"S367","status":"closed"},{"id":"I760","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A302", "I760", "S367", "S878", "T874"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0255 · total · wording 3 · sameedd9f9c68d94953dcc289cffb79f675bd90c56bddd6779ca1b59be3102ed4dde
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"S617","units":7,"price":4},{"id":"P972","units":14,"price":11},{"id":"W575","units":14,"price":16},{"id":"P708","units":5,"price":17},{"id":"V598","units":18,"price":19}]
Checked answer
{"total": 833}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0256 · sort · wording 3 · same3b3e779664cfd76867edbb9dc557ed2105b10c07b2a09c9909f7de63350afacc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"P802","priority":6},{"id":"E835","priority":3},{"id":"K659","priority":1},{"id":"O437","priority":5},{"id":"D137","priority":7}]
Checked answer
{"ids": ["D137", "P802"]}
Seed

invalid_json_object

[{"id":"P802","priority":6},{"id":"O437","priority":5},{"id":"D137","priority":7},{"id":"K659","priority":1},{"id":"E835","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["D137","P802"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D137","P802"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D137","E835"]}
16 tokens
0257 · lookup · wording 0 · gaincc5151ddaac8da4addc4009ce1318058a8ab26920d1ed8542149405d56f653d5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record M268. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"T747","city":"Accra","units":9},{"id":"E161","city":"Accra","units":7},{"id":"M268","city":"Riga","units":14},{"id":"W840","city":"Bern","units":6},{"id":"S872","city":"Lima","units":5}]
Checked answer
{"city": "Riga", "units": 14}
Seed

invalid_json_object

[{"city":"Riga","units":14}]
13 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":14}
12 tokens
0258 · filter · wording 0 · gain87f890851f1921a81641f7b17f4e7bed2a9f91c45832ae5ce824498ced7f9506
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"A637","status":"ready"},{"id":"J191","status":"ready"},{"id":"E700","status":"closed"},{"id":"H824","status":"pending"},{"id":"C922","status":"closed"}]
Checked answer
{"ids": ["A637", "J191"]}
Seed

wrong_schema_or_value

{"ids": ["A637", "J191", "C922"]}
23 tokens
1 GPU · AdamW

Correct

{"ids":["A637","J191"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A637","J191"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A637","J191"]}
16 tokens
0259 · total · wording 0 · same3d54c228305a6b1fbaf242355d36e950c007f4ec9a371795af340b62add5c407
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"H507","units":3,"price":14},{"id":"E516","units":1,"price":11},{"id":"P814","units":7,"price":9},{"id":"J429","units":1,"price":17},{"id":"B661","units":9,"price":16}]
Checked answer
{"total": 277}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":278}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0260 · sort · wording 0 · same4a095a8466aa450e502434c4510895286c4dc592ee31abb4f3d2236ed01f27ec
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"B734","priority":5},{"id":"B797","priority":8},{"id":"K396","priority":8},{"id":"H796","priority":5},{"id":"P817","priority":6}]
Checked answer
{"ids": ["B797", "K396"]}
Seed

wrong_schema_or_value

{"ids": ["B797", "B734", "H796", "K396", "P817"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["K396","B797"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B797","K396"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K396","B797"]}
16 tokens
0261 · lookup · wording 1 · gaine15991d48b8e37d89a16f35528919e587f0741c3dd6d15a189e2a5ddb24d18e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"C624","city":"Oslo","units":19},{"id":"S405","city":"Bern","units":17},{"id":"S157","city":"Oslo","units":14},{"id":"U336","city":"Bern","units":8},{"id":"S179","city":"Riga","units":7}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0262 · filter · wording 1 · gain5fe98c889077146d055093285e0dec99a3d8170b86567d203d559d0be8d4d176
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"O904","status":"pending"},{"id":"W446","status":"closed"},{"id":"G460","status":"pending"},{"id":"T195","status":"pending"},{"id":"P653","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["O904", "W446", "G460", "T195", "P653"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0263 · total · wording 1 · same867565f089152a5c93e4a81247177b8871fe159dcb9c0bfce71c9a4155aed85d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"C925","units":9,"price":4},{"id":"V247","units":15,"price":7},{"id":"H653","units":16,"price":8},{"id":"T966","units":1,"price":14},{"id":"B699","units":14,"price":10}]
Checked answer
{"total": 423}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0264 · sort · wording 1 · gain3f2e0ab57580940f421b94b17510ba8965f9923ac14d98b058f4edeaf800ffd0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"W538","priority":8},{"id":"A515","priority":8},{"id":"U713","priority":5},{"id":"E817","priority":3},{"id":"Y714","priority":7}]
Checked answer
{"ids": ["A515", "W538"]}
Seed

invalid_json_object

[{"id":"W538","priority":8},{"id":"A515","priority":8},{"id":"U713","priority":5},{"id":"E817","priority":3},{"id":"Y714","priority":7}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A515","W538"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W538","A515"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A515","W538"]}
16 tokens
0265 · lookup · wording 2 · gain5583e2c3e82d3871c0d46aa98a7df896d23c75571cc2c95a8b479e3ff07a0ffa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to L724? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E457","city":"Kyoto","units":0},{"id":"J733","city":"Perth","units":14},{"id":"L724","city":"Perth","units":3},{"id":"F467","city":"Riga","units":0},{"id":"I851","city":"Perth","units":19}]
Checked answer
{"city": "Perth", "units": 3}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":3}
11 tokens
0266 · filter · wording 2 · samec53d28ea9139b285b12bbbd317abd2dd2143850ab6100a7f9a84981cc0744b6a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"J745","status":"closed"},{"id":"H159","status":"pending"},{"id":"D235","status":"closed"},{"id":"K784","status":"closed"},{"id":"Q164","status":"ready"}]
Checked answer
{"ids": ["Q164"]}
Seed

invalid_json_object

[{"id":"Q164"}]
11 tokens
1 GPU · AdamW

invalid_json_object

["Q164"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["Q164"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["Q164"]]
11 tokens
0267 · total · wording 2 · samea9fdb220cd9d189dcc3b369faa48b5549894408f853e9f4e66ebcac3df9f1d82
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"T683","units":19,"price":19},{"id":"L971","units":16,"price":13},{"id":"Q853","units":13,"price":7},{"id":"A482","units":4,"price":18},{"id":"K987","units":19,"price":13}]
Checked answer
{"total": 979}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0268 · sort · wording 2 · gaina85c3db099b2d05fd641655a9c9a7790c7f7f85fe71746568cb55506547c5425
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K458","priority":8},{"id":"O751","priority":1},{"id":"G580","priority":8},{"id":"C200","priority":1},{"id":"O183","priority":4}]
Checked answer
{"ids": ["G580", "K458"]}
Seed

wrong_schema_or_value

{"ids": ["O183", "O751", "C200", "G580", "K458"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G580","K458"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G580","K458"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G580","K458"]}
16 tokens
0269 · lookup · wording 3 · samecd0024a8ebffd33c4a16d59280dc5d83afe0296f403a1c5fc4c69272eb960c1c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=F324. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E127","city":"Oslo","units":5},{"id":"F324","city":"Oslo","units":14},{"id":"X705","city":"Bern","units":16},{"id":"U596","city":"Perth","units":15},{"id":"O937","city":"Oslo","units":16}]
Checked answer
{"city": "Oslo", "units": 14}
Seed

Correct

{"city":"Oslo","units":14}
12 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":14}
12 tokens
0270 · filter · wording 3 · same355d2b12e2dd662f3b203d2869041d11d624e8f6ecf58fde45debe01f0b0e766
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"N665","status":"ready"},{"id":"U774","status":"closed"},{"id":"X961","status":"ready"},{"id":"X845","status":"ready"},{"id":"S234","status":"pending"}]
Checked answer
{"ids": ["N665", "X845", "X961"]}
Seed

wrong_schema_or_value

{"ids": ["N665", "X845", "X961", "U774", "S234"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["N665","X961","X845"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N665","X961","X845"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N665","X961","X845"]}
21 tokens
0271 · total · wording 3 · same54c024a3d30c9952b6a8f77073b9be6049e289f2fa8d8c405857bca8719b447b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"Y695","units":3,"price":19},{"id":"Q231","units":11,"price":17},{"id":"T870","units":2,"price":13},{"id":"C997","units":12,"price":8},{"id":"A184","units":8,"price":5}]
Checked answer
{"total": 406}
Seed

invalid_json_object

{
  "total": 19 + 17 + 13 + 8 + 5 = 69
}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":378}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0272 · sort · wording 3 · gaind1ec21ff207b7f31fe339321e9c3483d3de16358a0014f372eb285061a9479ee
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"U578","priority":8},{"id":"Y533","priority":5},{"id":"N287","priority":7},{"id":"O735","priority":8},{"id":"J243","priority":7}]
Checked answer
{"ids": ["O735", "U578"]}
Seed

invalid_json_object

[{"id":"U578"},{"id":"Y533"},{"id":"N287"},{"id":"O735"},{"id":"J243"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["O735","U578"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O735","U578"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["O735","U578"]}
16 tokens
0273 · lookup · wording 0 · samebcb0f1978f1669624dbfff17bb6b262ba01ef1030028c1ddc2d318bef0e1705e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record R434. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"T680","city":"Lima","units":1},{"id":"H445","city":"Bern","units":1},{"id":"T756","city":"Bern","units":4},{"id":"R434","city":"Perth","units":6},{"id":"X415","city":"Lima","units":16}]
Checked answer
{"city": "Perth", "units": 6}
Seed

Correct

{
  "city": "Perth",
  "units": 6
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":6}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":6}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":6}
11 tokens
0274 · filter · wording 0 · gaind9a5ed99d20be2a7e692eb0e58ec93a48cb891792a08377b8e426cb31c273d7f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"D578","status":"closed"},{"id":"R601","status":"closed"},{"id":"A723","status":"ready"},{"id":"D729","status":"pending"},{"id":"B339","status":"pending"}]
Checked answer
{"ids": ["A723"]}
Seed

wrong_schema_or_value

{"ids": ["A723", "D729", "D578", "R601", "B339"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A723"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["A723"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["A723"]}
11 tokens
0275 · total · wording 0 · same45de579a23305966ef3203fb707dbe3838e174b0d5ce4f9e69c1bc10f72a04f0
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"B197","units":4,"price":9},{"id":"J829","units":13,"price":4},{"id":"W783","units":5,"price":3},{"id":"Q228","units":7,"price":11},{"id":"O743","units":3,"price":3}]
Checked answer
{"total": 189}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0276 · sort · wording 0 · gain2f19c5873746cbb604e1f64e04267f0a73ca90ace50999695f8545c9f0b0dcc6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"A313","priority":7},{"id":"N397","priority":3},{"id":"Z328","priority":1},{"id":"X478","priority":1},{"id":"F785","priority":2}]
Checked answer
{"ids": ["A313", "N397"]}
Seed

wrong_schema_or_value

{"ids": ["N397", "X478", "F785", "Z328", "A313"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["A313","F785"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A313","N397"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A313","N397"]}
16 tokens
0277 · lookup · wording 1 · same6a3317548453dfd2bf282400ee5629427053949d70a19f6df002e7ebfa6c7c0f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id G230 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"G230","city":"Suva","units":14},{"id":"L492","city":"Perth","units":7},{"id":"R349","city":"Riga","units":12},{"id":"V821","city":"Bern","units":1},{"id":"H364","city":"Suva","units":1}]
Checked answer
{"city": "Suva", "units": 14}
Seed

Correct

{"city":"Suva","units":14}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":14}
12 tokens
0278 · filter · wording 1 · gain756c2db27f7ed7809ecb8b0ce36e87871268977110df686e8c7335b28a7dcd0a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"E253","status":"pending"},{"id":"X581","status":"closed"},{"id":"D269","status":"ready"},{"id":"M513","status":"pending"},{"id":"K177","status":"pending"}]
Checked answer
{"ids": ["D269"]}
Seed

wrong_schema_or_value

{
  "ids": ["D269", "E253", "K177", "M513", "X581"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["D269"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["D269"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["D269"]}
11 tokens
0279 · total · wording 1 · samee7d7af86c45643ff718da28761e982205ed6848ad28731b26cf2f1edb5c24800
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"Q200","units":15,"price":16},{"id":"O912","units":19,"price":12},{"id":"E317","units":10,"price":10},{"id":"W584","units":2,"price":8},{"id":"D342","units":13,"price":14}]
Checked answer
{"total": 766}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0280 · sort · wording 1 · gain074f27761f4d46802d9b0c2497a2bbb63ff643c3e934e7dc19827cd3c9179557
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Z885","priority":3},{"id":"L920","priority":2},{"id":"N730","priority":8},{"id":"H504","priority":2},{"id":"S685","priority":1}]
Checked answer
{"ids": ["N730", "Z885"]}
Seed

invalid_json_object

[{"id":"Z885","priority":3},{"id":"L920","priority":2},{"id":"N730","priority":8},{"id":"H504","priority":2},{"id":"S685","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["N730","Z885"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N730","Z885"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["N730","Z885"]}
16 tokens
0281 · lookup · wording 2 · same41816838253bd5812efcbb2274502de528e2e04d76031651a02241db78ee1115
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Z968","city":"Oslo","units":17},{"id":"C443","city":"Oslo","units":19},{"id":"J260","city":"Perth","units":15},{"id":"P990","city":"Kyoto","units":1},{"id":"U593","city":"Lima","units":2}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0282 · filter · wording 2 · samea4e5bfa032b6a1ab1dbad44b2b1f0be0d7d418b8f6098ef2d1661dd54d2ab676
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"I831","status":"ready"},{"id":"R867","status":"pending"},{"id":"Q143","status":"closed"},{"id":"I536","status":"pending"},{"id":"X687","status":"closed"}]
Checked answer
{"ids": ["I831"]}
Seed

invalid_json_object

[{"id":"I831","status":"ready"},{"id":"R867","status":"pending"},{"id":"Q143","status":"closed"},{"id":"I536","status":"pending"},{"id":"X687","status":"closed"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["I831"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["I831"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["I831"]]
11 tokens
0283 · total · wording 2 · sameda15fe50010d676117d75c78aaac68b599e5c92318b8f9de1fb63e0ffda774dd
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"N488","units":1,"price":13},{"id":"Q723","units":6,"price":13},{"id":"M534","units":17,"price":2},{"id":"X126","units":9,"price":11},{"id":"H250","units":13,"price":11}]
Checked answer
{"total": 367}
Seed

wrong_schema_or_value

{"total": 13}
8 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0284 · sort · wording 2 · samec78305b19611d0b8ab8e6207f5650739fbbf34d31dc9ddd30fb0e46dcba216e8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Y962","priority":8},{"id":"F682","priority":6},{"id":"Y605","priority":9},{"id":"K731","priority":5},{"id":"N595","priority":1}]
Checked answer
{"ids": ["Y605", "Y962"]}
Seed

wrong_schema_or_value

{"ids": ["Y962", "K731"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["Y605","Y962"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Y605","Y962"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F682","Y605"]}
16 tokens
0285 · lookup · wording 3 · gain4da9f4ca8d1c6a292b52c0b2221838f34e470fd83d2acc69d817d00a979f7d22
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=W554. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"G618","city":"Kyoto","units":17},{"id":"Y933","city":"Riga","units":12},{"id":"I989","city":"Bern","units":1},{"id":"S143","city":"Suva","units":15},{"id":"W554","city":"Lima","units":18}]
Checked answer
{"city": "Lima", "units": 18}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":17}
13 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":18}
12 tokens
0286 · filter · wording 3 · gain4d8a757585363c57df259720245dad9ebe74dc10846a6fda59546960a4054501
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"T569","status":"pending"},{"id":"Z731","status":"pending"},{"id":"H377","status":"pending"},{"id":"T902","status":"pending"},{"id":"V319","status":"ready"}]
Checked answer
{"ids": ["V319"]}
Seed

wrong_schema_or_value

{"ids": ["H377", "T902", "V319", "T569", "Z731"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["V319"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["V319"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["V319"]}
11 tokens
0287 · total · wording 3 · samee251fb92ac38ec5a551b54307ac99cc273f0bce398bb93bc316e7e7335bef7ef
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"S125","units":11,"price":8},{"id":"Q759","units":6,"price":10},{"id":"Y886","units":9,"price":12},{"id":"I135","units":15,"price":8},{"id":"M580","units":3,"price":16}]
Checked answer
{"total": 424}
Seed

wrong_schema_or_value

{
  "total": 30
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0288 · sort · wording 3 · gaindfd037167b1852411c3de55985d390131811b28291d8a24c792209235243fb98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"T473","priority":7},{"id":"G191","priority":9},{"id":"I575","priority":4},{"id":"Z421","priority":4},{"id":"T746","priority":9}]
Checked answer
{"ids": ["G191", "T746"]}
Seed

invalid_json_object

[{"id":"T473","priority":7},{"id":"G191","priority":9},{"id":"I575","priority":4},{"id":"Z421","priority":4},{"id":"T746","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["G191","T746"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G191","T746"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G191","T746"]}
16 tokens
0289 · lookup · wording 0 · gaineedc2536fab1fe994e22aec66dd03d182d0ccabcf814706875457dc0dfba9861
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record L149. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"S929","city":"Lima","units":1},{"id":"L149","city":"Kyoto","units":19},{"id":"W538","city":"Riga","units":2},{"id":"U279","city":"Kyoto","units":18},{"id":"V155","city":"Riga","units":2}]
Checked answer
{"city": "Kyoto", "units": 19}
Seed

wrong_schema_or_value

{
  "city": "Lima",
  "units": 1
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":19}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":19}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":19}
13 tokens
0290 · filter · wording 0 · gained56e308fd5f19db4e6b50b8fc6dd4c04d88fe5f28ebe87a2959c4584bd1fb26
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"W366","status":"pending"},{"id":"C510","status":"ready"},{"id":"R158","status":"ready"},{"id":"O501","status":"closed"},{"id":"N177","status":"ready"}]
Checked answer
{"ids": ["C510", "N177", "R158"]}
Seed

wrong_schema_or_value

{"ids": ["C510", "R158", "N177"]}
23 tokens
1 GPU · AdamW

Correct

{"ids":["C510","N177","R158"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["C510","N177","R158"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["C510","N177","R158"]}
21 tokens
0291 · total · wording 0 · same64872e3db648240ce11f47f67a52c98ac9da61a53a43eae37b5ea9c023364f66
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"W810","units":8,"price":5},{"id":"Y586","units":10,"price":14},{"id":"V932","units":11,"price":3},{"id":"V815","units":4,"price":17},{"id":"B821","units":2,"price":18}]
Checked answer
{"total": 317}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0292 · sort · wording 0 · gainc3c33f1b318ccb2b88e6e763038d8a444701d1fc406e1d0d2d5d75db7b5661dc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U104","priority":9},{"id":"T432","priority":2},{"id":"D702","priority":9},{"id":"S475","priority":3},{"id":"F544","priority":8}]
Checked answer
{"ids": ["D702", "U104"]}
Seed

wrong_schema_or_value

{"ids": ["U104", "T432", "D702", "S475", "F544"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D702","U104"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D702","U104"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D702","U104"]}
16 tokens
0293 · lookup · wording 1 · same40d8863de9e6e0636621173345dd207917db86db83ba75eec582d925f6fe3eb1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id B553 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"B553","city":"Bern","units":18},{"id":"P477","city":"Lima","units":4},{"id":"S992","city":"Suva","units":11},{"id":"K994","city":"Lima","units":19},{"id":"S753","city":"Oslo","units":16}]
Checked answer
{"city": "Bern", "units": 18}
Seed

Correct

{"city":"Bern","units":18}
11 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":18}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":18}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":18}
11 tokens
0294 · filter · wording 1 · gain0a24c8e30df81c0fe5b9cd4582bef65a796b5760710babedac97f4728fd1fb36
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"A862","status":"closed"},{"id":"T516","status":"closed"},{"id":"O223","status":"closed"},{"id":"X867","status":"pending"},{"id":"B449","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["A862", "T516", "O223", "X867"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0295 · total · wording 1 · same7c9b9e05f606638d180c79abf2bea452ecc90d668e2006e09fece5c9f5c0d59f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"P698","units":10,"price":3},{"id":"E925","units":15,"price":5},{"id":"Y774","units":18,"price":17},{"id":"K543","units":12,"price":18},{"id":"I629","units":11,"price":12}]
Checked answer
{"total": 759}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0296 · sort · wording 1 · gain7b29f612e000996e1a0a5aa5a144fd5223382a0f1435bfd0999c505aea788dc3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"G440","priority":3},{"id":"O240","priority":8},{"id":"R256","priority":2},{"id":"W481","priority":4},{"id":"U149","priority":3}]
Checked answer
{"ids": ["O240", "W481"]}
Seed

invalid_json_object

[{"id":"G440","priority":3},{"id":"O240","priority":8},{"id":"R256","priority":2},{"id":"W481","priority":4},{"id":"U149","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["O240","W481"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O240","W481"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["O240","W481"]}
16 tokens
0297 · lookup · wording 2 · gain042eb481da4deb9e2bf1abf5be4d9215d01eee964462aaabfab6d6240c1b2f0a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to J343? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"J343","city":"Riga","units":13},{"id":"O892","city":"Suva","units":9},{"id":"O365","city":"Kyoto","units":6},{"id":"G728","city":"Lima","units":4},{"id":"V759","city":"Accra","units":12}]
Checked answer
{"city": "Riga", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":13}
12 tokens
0298 · filter · wording 2 · same24025ec49ef18864a3339d14c0921af6cd9e624bc59a668d65ca9305825fa4ac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"F990","status":"ready"},{"id":"J219","status":"closed"},{"id":"J142","status":"pending"},{"id":"O518","status":"ready"},{"id":"V621","status":"closed"}]
Checked answer
{"ids": ["F990", "O518"]}
Seed

invalid_json_object

[{"id":"F990"},{"id":"O518"},{"id":"V621"},{"id":"J142"},{"id":"J219"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["F990","O518"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F990","O518"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["F990","O518"]]
16 tokens
0299 · total · wording 2 · samebbd7a277ced028661d3522f7582e47b45c1f4c0a2c0f434f866cec21a9543260
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I807","units":3,"price":13},{"id":"G594","units":7,"price":16},{"id":"F147","units":5,"price":2},{"id":"O621","units":4,"price":2},{"id":"C485","units":14,"price":5}]
Checked answer
{"total": 239}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0300 · sort · wording 2 · gainbf0c7f16cbbba964d3cde0b8770925085957e05584d1e3e5918c19a669ddd2bd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"D255","priority":9},{"id":"R659","priority":8},{"id":"A714","priority":5},{"id":"L888","priority":9},{"id":"Z339","priority":5}]
Checked answer
{"ids": ["D255", "L888"]}
Seed

wrong_schema_or_value

{"ids": ["D255", "A714", "L888", "Z339", "R659"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D255","L888"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L888","D255"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D255","L888"]}
16 tokens
0301 · lookup · wording 3 · gain36b229feb6ff150ece21c7023a6814cdfa257eb4e2f62ef171ccc8b665a27bcc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"J115","city":"Oslo","units":7},{"id":"J733","city":"Perth","units":11},{"id":"J290","city":"Perth","units":4},{"id":"N690","city":"Lima","units":19},{"id":"S834","city":"Accra","units":19}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0302 · filter · wording 3 · samed71a7cde8c3127daae8ccdba51359733e2580e5238d2ff08c5479c5ebb826e01
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"K258","status":"ready"},{"id":"G343","status":"ready"},{"id":"T491","status":"pending"},{"id":"Z721","status":"closed"},{"id":"I754","status":"ready"}]
Checked answer
{"ids": ["G343", "I754", "K258"]}
Seed

wrong_schema_or_value

{"ids": ["K258", "G343", "T491", "Z721", "I754"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G343","I754","K258"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["I754","K258","G343"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I754","K258","G343"]}
21 tokens
0303 · total · wording 3 · samef27ffbada50b51cd3eb4d57e65e57a440d6d9818735e15a2d4bb9245d87aa3b1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"C788","units":0,"price":12},{"id":"X579","units":8,"price":16},{"id":"S921","units":8,"price":4},{"id":"N523","units":17,"price":13},{"id":"G707","units":11,"price":16}]
Checked answer
{"total": 557}
Seed

invalid_json_object

{
  "total": 12 + 128 + 12 + 132 + 176
}
30 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0304 · sort · wording 3 · gain0b2dad29e7ec5cbaea8eb2469969650563b82e423940e6603df184aedae85974
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"P626","priority":7},{"id":"F579","priority":5},{"id":"L141","priority":2},{"id":"G843","priority":3},{"id":"N210","priority":9}]
Checked answer
{"ids": ["N210", "P626"]}
Seed

invalid_json_object

[{"id":"P626","priority":7},{"id":"F579","priority":5},{"id":"L141","priority":2},{"id":"G843","priority":3},{"id":"N210","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["N210","P626"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N210","P626"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["N210","P626"]}
16 tokens
0305 · lookup · wording 0 · samecae68f823b316b2fb1e55c51eac21f4bcffa09fe78b37f9ca1fdce3aaae50037
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record M795. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Q112","city":"Accra","units":18},{"id":"N156","city":"Oslo","units":3},{"id":"X914","city":"Riga","units":18},{"id":"P661","city":"Kyoto","units":13},{"id":"M795","city":"Bern","units":9}]
Checked answer
{"city": "Bern", "units": 9}
Seed

Correct

{
  "city": "Bern",
  "units": 9
}
17 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":9}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":9}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":9}
10 tokens
0306 · filter · wording 0 · gain842402fabe1e87d33d6b5a12841a9a3f3d71180ad16a7c91e04dcd9262faa376
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"O681","status":"pending"},{"id":"Y922","status":"ready"},{"id":"Y316","status":"closed"},{"id":"Y114","status":"pending"},{"id":"T319","status":"pending"}]
Checked answer
{"ids": ["Y922"]}
Seed

wrong_schema_or_value

{"ids": ["Y114", "Y316", "Y922", "O681"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["Y922"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Y922"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Y922"]}
11 tokens
0307 · total · wording 0 · samee217977ab4ce1ef38ae9fb604f4c44de63c0d9e845e779e9ecc04c842341b07d
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P313","units":0,"price":10},{"id":"E113","units":16,"price":13},{"id":"C713","units":5,"price":10},{"id":"J784","units":11,"price":3},{"id":"K999","units":2,"price":2}]
Checked answer
{"total": 295}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":418}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0308 · sort · wording 0 · gaina8995a8354a582a425d8c03e759056249f7ab95c357895bf6d902c1abdfc9354
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"O163","priority":1},{"id":"T427","priority":3},{"id":"C741","priority":4},{"id":"D338","priority":4},{"id":"Z195","priority":3}]
Checked answer
{"ids": ["C741", "D338"]}
Seed

wrong_schema_or_value

{"ids": ["O163", "T427", "C741", "D338", "Z195"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C741","D338"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C741","D338"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C741","D338"]}
16 tokens
0309 · lookup · wording 1 · gain4210e8a05039b42d751c533673309554d39bf30d95749f57570b9915b8d37c6c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id N100 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D169","city":"Suva","units":14},{"id":"Y749","city":"Lima","units":13},{"id":"N100","city":"Perth","units":14},{"id":"J422","city":"Perth","units":1},{"id":"E858","city":"Lima","units":7}]
Checked answer
{"city": "Perth", "units": 14}
Seed

wrong_schema_or_value

{"city":"Suva","units":14}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":14}
12 tokens
0310 · filter · wording 1 · same2599d85884bc77ce48ae88d6595e7bc4332437a829c947991bac3a310cf94e49
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"J980","status":"ready"},{"id":"B733","status":"closed"},{"id":"A148","status":"pending"},{"id":"D184","status":"closed"},{"id":"D961","status":"ready"}]
Checked answer
{"ids": ["D961", "J980"]}
Seed

wrong_schema_or_value

{
  "ids": ["A148", "B733", "D184", "D961", "J980"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["D961","J980"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D961","J980"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D184","J980"]}
16 tokens
0311 · total · wording 1 · samee6fc9791d48b925b5633918ebdb44571911d4c04641a0337f41edbcf079e610a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"J339","units":12,"price":18},{"id":"U329","units":11,"price":6},{"id":"L543","units":11,"price":14},{"id":"E426","units":17,"price":18},{"id":"J852","units":10,"price":7}]
Checked answer
{"total": 812}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0312 · sort · wording 1 · same3c52326069d6be584095581c47abaaacc015084539f82e3690e14fc1171dfcd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J928","priority":2},{"id":"F971","priority":8},{"id":"D515","priority":6},{"id":"R144","priority":8},{"id":"H528","priority":4}]
Checked answer
{"ids": ["F971", "R144"]}
Seed

invalid_json_object

[{"id":"J928","priority":2},{"id":"F971","priority":8},{"id":"D515","priority":6},{"id":"R144","priority":8},{"id":"H528","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["F971","R144"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F971","R144"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D515","R144"]}
16 tokens
0313 · lookup · wording 2 · gain9f0cce31d3f9da45930d42e861604f36ba49f7f9875185b659b73672dc3795c1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to F136? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"F136","city":"Perth","units":0},{"id":"G600","city":"Oslo","units":14},{"id":"N434","city":"Accra","units":2},{"id":"U869","city":"Bern","units":16},{"id":"C908","city":"Bern","units":14}]
Checked answer
{"city": "Perth", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":0}
11 tokens
0314 · filter · wording 2 · samea420a0a21db8aa08eebf1d22856bde5cfb2894ace342545212cbbb48c87c35fb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"X401","status":"pending"},{"id":"W825","status":"pending"},{"id":"G923","status":"ready"},{"id":"I141","status":"pending"},{"id":"K611","status":"closed"}]
Checked answer
{"ids": ["G923"]}
Seed

invalid_json_object

[{"id":"G923","status":"ready"},{"id":"I141","status":"pending"},{"id":"K611","status":"closed"}]
39 tokens
1 GPU · AdamW

Correct

{"ids":["G923"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["G923"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G923"]]
11 tokens
0315 · total · wording 2 · same520ba826f209f1209bff6b143f9d456e03b9ecd4af806f00a5aa468fc7448bd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Q371","units":4,"price":9},{"id":"D295","units":11,"price":19},{"id":"T452","units":0,"price":17},{"id":"A189","units":14,"price":2},{"id":"J418","units":4,"price":3}]
Checked answer
{"total": 285}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":408}
8 tokens
0316 · sort · wording 2 · same62a279972de1ba1a89de5a21e4636d3f69cf3da3deec649e01db29c265e6c337
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K820","priority":3},{"id":"T710","priority":4},{"id":"R208","priority":4},{"id":"D374","priority":9},{"id":"D836","priority":3}]
Checked answer
{"ids": ["D374", "R208"]}
Seed

wrong_schema_or_value

{"ids": ["K820", "T710", "R208", "D374", "D836"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D374","R208"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D374","T710"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D374","K820"]}
16 tokens
0317 · lookup · wording 3 · same1da805acc724a2035d494362b5238fb2825de60af28eb14092ebb0e3338cd755
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T964. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"C248","city":"Riga","units":13},{"id":"X356","city":"Oslo","units":14},{"id":"T964","city":"Riga","units":15},{"id":"Q284","city":"Accra","units":16},{"id":"B478","city":"Bern","units":15}]
Checked answer
{"city": "Riga", "units": 15}
Seed

Correct

{"city":"Riga","units":15}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":15}
12 tokens
0318 · filter · wording 3 · gain625c0c94d9fbe42acce09bf1a2f03a31eb6b9924bced0000f0b7f35ca0be57d0
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"X835","status":"closed"},{"id":"T498","status":"pending"},{"id":"S963","status":"closed"},{"id":"Z402","status":"pending"},{"id":"W270","status":"ready"}]
Checked answer
{"ids": ["W270"]}
Seed

wrong_schema_or_value

{"ids": ["W270", "X835", "S963", "T498", "Z402"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["W270"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["W270"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["W270"]}
11 tokens
0319 · total · wording 3 · same89916163460cda23f163962f6b32f76ee366bd8fcfc2eba31ca99f15656a2ffd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"V731","units":2,"price":7},{"id":"L800","units":7,"price":11},{"id":"K145","units":4,"price":7},{"id":"H773","units":12,"price":4},{"id":"A545","units":3,"price":3}]
Checked answer
{"total": 176}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":288}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":278}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0320 · sort · wording 3 · same9030b8d75ce07833797b9417308d83c40d386f59af8afc4a98765ddc59eecc06
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J688","priority":4},{"id":"R821","priority":9},{"id":"X517","priority":2},{"id":"R224","priority":8},{"id":"H886","priority":3}]
Checked answer
{"ids": ["R821", "R224"]}
Seed

invalid_json_object

[{"id":"J688"},{"id":"R821"},{"id":"X517"},{"id":"R224"},{"id":"H886"}]
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["R224","R821"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["R821","J688"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["R821","J688"]}
16 tokens
0321 · lookup · wording 0 · gainbc86aeb3b3f236c5efd144765bcae088b14cc954224574dfc04ef0c82c2f5919
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"P392","city":"Kyoto","units":5},{"id":"H624","city":"Lima","units":19},{"id":"J943","city":"Perth","units":19},{"id":"M414","city":"Oslo","units":8},{"id":"B558","city":"Bern","units":12}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Kyoto",
  "units": 5
}
19 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0322 · filter · wording 0 · gain0e3b83ada2f5e5a73ab8ba1d5170ed7af56bec0594e47dd969732146bfcc9c7c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"F738","status":"ready"},{"id":"X369","status":"ready"},{"id":"R406","status":"pending"},{"id":"H487","status":"pending"},{"id":"E663","status":"pending"}]
Checked answer
{"ids": ["F738", "X369"]}
Seed

wrong_schema_or_value

{"ids": ["F738", "X369", "H487", "E663"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["F738","X369"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F738","X369"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F738","X369"]}
16 tokens
0323 · total · wording 0 · samec460d9839cc160f137dd63d6bbc4b89fa7799141a7093525deb1b1cf30469413
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"D537","units":14,"price":9},{"id":"C664","units":12,"price":15},{"id":"U624","units":12,"price":13},{"id":"Z359","units":11,"price":13},{"id":"P707","units":19,"price":5}]
Checked answer
{"total": 700}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0324 · sort · wording 0 · gain63b6352ab892db8e3b95a3331fe831c599a87c4dd5274d12df1b5293d0cf8284
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U655","priority":8},{"id":"S594","priority":9},{"id":"Z866","priority":3},{"id":"U871","priority":7},{"id":"N809","priority":7}]
Checked answer
{"ids": ["S594", "U655"]}
Seed

wrong_schema_or_value

{"ids": ["U655", "S594", "Z866", "U871", "N809"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["S594","U655"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S594","U655"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S594","U655"]}
16 tokens
0325 · lookup · wording 1 · samefcc1778d71aebf9f322e98eaf43d8ba30ce5abfd9c394bfe835443639d937468
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z822 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"Z822","city":"Perth","units":10},{"id":"A193","city":"Bern","units":17},{"id":"Q656","city":"Lima","units":16},{"id":"Z815","city":"Suva","units":10},{"id":"Y988","city":"Suva","units":3}]
Checked answer
{"city": "Perth", "units": 10}
Seed

Correct

{"city":"Perth","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":10}
12 tokens
0326 · filter · wording 1 · same5dacd98bd79d602df39417c115e61ae187f1ea38b55ae0f8410fede2943e2340
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"U355","status":"pending"},{"id":"K466","status":"ready"},{"id":"I984","status":"ready"},{"id":"H714","status":"ready"},{"id":"G370","status":"ready"}]
Checked answer
{"ids": ["G370", "H714", "I984", "K466"]}
Seed

wrong_schema_or_value

{
  "ids": ["U355", "K466", "I984", "H714", "G370"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H714","I984","K466"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["G370","H714","I984","K466"]}
26 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I984","H714","G370"]}
21 tokens
0327 · total · wording 1 · samea2e605fb8ed3dac5c608e1575bb6926b703dd2b0fdf5a86c89ee82cc1632445c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"N835","units":14,"price":12},{"id":"P348","units":3,"price":6},{"id":"D397","units":0,"price":18},{"id":"X778","units":4,"price":15},{"id":"F431","units":17,"price":2}]
Checked answer
{"total": 280}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":408}
8 tokens
0328 · sort · wording 1 · gain4bcd4e9f7f83d690769ae70cbdcc07daa89d493406f20139e9fd4b1e51114bce
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Z497","priority":5},{"id":"C946","priority":5},{"id":"Z179","priority":9},{"id":"T713","priority":8},{"id":"V102","priority":7}]
Checked answer
{"ids": ["Z179", "T713"]}
Seed

invalid_json_object

[{"id":"Z497","priority":5},{"id":"C946","priority":5},{"id":"Z179","priority":9},{"id":"T713","priority":8},{"id":"V102","priority":7}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["Z179","T713"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Z179","T713"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z179","T713"]}
16 tokens
0329 · lookup · wording 2 · gainbc18a1d5f2e260fd5c34c6c927c69c1f0a3ed60aa9a612121a8ee3d396273b71
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to D745? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"P384","city":"Kyoto","units":10},{"id":"W628","city":"Lima","units":0},{"id":"D745","city":"Kyoto","units":13},{"id":"L610","city":"Perth","units":12},{"id":"N526","city":"Accra","units":10}]
Checked answer
{"city": "Kyoto", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": 13
}
17 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":13}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":13}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":13}
13 tokens
0330 · filter · wording 2 · gainfa349c44566bc07036a15705ea0b97bccefc2288ef97f6e35b7fdd47a32cd675
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S447","status":"closed"},{"id":"U416","status":"pending"},{"id":"A618","status":"pending"},{"id":"V434","status":"pending"},{"id":"V157","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"A618","status":"pending"},{"id":"V157","status":"closed"},{"id":"V434","status":"pending"},{"id":"S447","status":"closed"}]
51 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0331 · total · wording 2 · same816e81d8cf72370c4fe34c31ab87172a2740d3011cf0c7f01e9c286bcf7ed0e6
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"T591","units":14,"price":7},{"id":"S192","units":16,"price":9},{"id":"X175","units":14,"price":6},{"id":"H683","units":4,"price":10},{"id":"F629","units":7,"price":8}]
Checked answer
{"total": 422}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0332 · sort · wording 2 · gain786ce94556b8b45e287d6324de4df51cdee1b7d802f0b44455801f26954d4c05
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"F864","priority":6},{"id":"P527","priority":1},{"id":"N714","priority":3},{"id":"Y726","priority":3},{"id":"D885","priority":8}]
Checked answer
{"ids": ["D885", "F864"]}
Seed

wrong_schema_or_value

{"ids": ["F864", "P527", "N714", "Y726", "D885"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D885","F864"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D885","F864"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D885","F864"]}
16 tokens
0333 · lookup · wording 3 · gaind8bade26a8d9470b722f77aa081a924c127d989414793aa44eb9dc9ddeccdbc1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=R704. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T373","city":"Oslo","units":10},{"id":"R704","city":"Suva","units":10},{"id":"F957","city":"Oslo","units":3},{"id":"L535","city":"Oslo","units":14},{"id":"P165","city":"Riga","units":0}]
Checked answer
{"city": "Suva", "units": 10}
Seed

wrong_schema_or_value

{"city":"Oslo","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":10}
12 tokens
0334 · filter · wording 3 · gainebec30d52781749cda6d132b64f762fe8bf2be24fa6a53365d65849e262110cc
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"W704","status":"closed"},{"id":"X692","status":"ready"},{"id":"H480","status":"ready"},{"id":"U389","status":"closed"},{"id":"M818","status":"pending"}]
Checked answer
{"ids": ["H480", "X692"]}
Seed

wrong_schema_or_value

{"ids": ["X692", "H480", "U389", "M818", "W704"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["H480","X692"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H480","X692"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H480","X692"]}
16 tokens
0335 · total · wording 3 · samef52a7939143bcc7dbb4d538d673d3c5ded1b99726fb70d19b1a9f28619ef91ce
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"J835","units":15,"price":19},{"id":"S573","units":8,"price":8},{"id":"W496","units":12,"price":2},{"id":"M548","units":4,"price":17},{"id":"X347","units":5,"price":1}]
Checked answer
{"total": 446}
Seed

invalid_json_object

{
  "total": 19 + 8 + 2 + 17 + 1 = 47
}
28 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0336 · sort · wording 3 · gain3c2767798e6c6007e036dd28f7cefcc1d3c07f68828c23a9d1a5d0fa4985ffcb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J135","priority":8},{"id":"K105","priority":4},{"id":"X828","priority":1},{"id":"Z995","priority":1},{"id":"L284","priority":1}]
Checked answer
{"ids": ["J135", "K105"]}
Seed

invalid_json_object

[{"id":"Z995","priority":1},{"id":"L284","priority":1},{"id":"J135","priority":8},{"id":"K105","priority":4},{"id":"X828","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J135","K105"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J135","K105"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J135","K105"]}
16 tokens
0337 · lookup · wording 0 · gain3c96f37dc5e7e03c4574f22a0bba3f5a3ed96a3ba66c7672bd8f84f6544f98bd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record G956. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"A537","city":"Kyoto","units":17},{"id":"G956","city":"Accra","units":12},{"id":"Z461","city":"Riga","units":2},{"id":"T373","city":"Suva","units":14},{"id":"E837","city":"Bern","units":11}]
Checked answer
{"city": "Accra", "units": 12}
Seed

wrong_schema_or_value

{
  "city": "Kyoto",
  "units": 17
}
20 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":12}
12 tokens
0338 · filter · wording 0 · gaina8ed46250f4e94ae32cc7a617cf67486cc0e9a5d860abcc0bfcd33ddf6eb328c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"C626","status":"closed"},{"id":"U343","status":"ready"},{"id":"C569","status":"ready"},{"id":"J676","status":"closed"},{"id":"G990","status":"pending"}]
Checked answer
{"ids": ["C569", "U343"]}
Seed

wrong_schema_or_value

{"ids": ["C569", "C626", "U343", "J676", "G990"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C569","U343"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["C569","J676"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C569","U343"]}
16 tokens
0339 · total · wording 0 · same36b1ce06c80b620ad37cb51f3235113bde421adbb7288645c586def42dac45c7
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"T883","units":7,"price":16},{"id":"W237","units":0,"price":5},{"id":"J467","units":7,"price":3},{"id":"W465","units":17,"price":2},{"id":"M158","units":15,"price":7}]
Checked answer
{"total": 272}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0340 · sort · wording 0 · same1f379492419e5b146fc5dcf059eec47754390f99b0c4d2e5874d82af01a3fa69
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"K483","priority":8},{"id":"R263","priority":2},{"id":"Q974","priority":8},{"id":"Q663","priority":4},{"id":"T453","priority":3}]
Checked answer
{"ids": ["K483", "Q974"]}
Seed

wrong_schema_or_value

{"ids": ["R263", "K483", "T453", "Q663", "Q974"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["K483","Q974"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K483","Q974"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K483","Q663"]}
16 tokens
0341 · lookup · wording 1 · gaincb4cfed012e8b9abbca14cd98168616f7f9d5382d491bfd912ba59681a47d2bf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"K968","city":"Bern","units":12},{"id":"B259","city":"Accra","units":15},{"id":"C247","city":"Suva","units":1},{"id":"R857","city":"Kyoto","units":3},{"id":"Z457","city":"Oslo","units":11}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Bern","units":12}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0342 · filter · wording 1 · gain506907e6def8443614540a3dcb59cfc620d2f73262c91cf7260b71103d0a81d4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"F743","status":"closed"},{"id":"T808","status":"closed"},{"id":"Z145","status":"pending"},{"id":"J672","status":"ready"},{"id":"X110","status":"pending"}]
Checked answer
{"ids": ["J672"]}
Seed

wrong_schema_or_value

{
  "ids": ["F743", "T808", "Z145", "J672", "X110"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["J672"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["J672"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["J672"]}
11 tokens
0343 · total · wording 1 · same6a29c82c0ea4919efa18889a39c6ac43373b4ea2a7851051ceb1b65125a804cc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"E729","units":9,"price":5},{"id":"Z755","units":2,"price":9},{"id":"S440","units":13,"price":1},{"id":"K264","units":2,"price":9},{"id":"R737","units":6,"price":12}]
Checked answer
{"total": 166}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0344 · sort · wording 1 · gain25f7f79533fc497e65ce5c36185de8e3404452baf4253da7381f0067dc1c70b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"P961","priority":6},{"id":"Y328","priority":3},{"id":"M750","priority":6},{"id":"J376","priority":3},{"id":"O985","priority":4}]
Checked answer
{"ids": ["M750", "P961"]}
Seed

invalid_json_object

[{"id":"P961","priority":6},{"id":"M750","priority":6},{"id":"J376","priority":3},{"id":"O985","priority":4}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["M750","P961"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M750","P961"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M750","P961"]}
16 tokens
0345 · lookup · wording 2 · gaina07dc6f50f574666692122a61f283c6edb769baa38d66e0c6b26ea932ae3b886
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to X610? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"S137","city":"Riga","units":1},{"id":"U521","city":"Lima","units":16},{"id":"K403","city":"Lima","units":6},{"id":"X610","city":"Bern","units":0},{"id":"Y286","city":"Kyoto","units":8}]
Checked answer
{"city": "Bern", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":0}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":0}
10 tokens
0346 · filter · wording 2 · same0f79ae6d4908ae6b2a38b8326a816a47aa3bec97d430493216e4b433a80e1f84
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"I110","status":"closed"},{"id":"L598","status":"pending"},{"id":"W240","status":"pending"},{"id":"N829","status":"ready"},{"id":"V746","status":"closed"}]
Checked answer
{"ids": ["N829"]}
Seed

invalid_json_object

[{"id":"N829"}]
11 tokens
1 GPU · AdamW

Correct

{"ids":["N829"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["N829"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["N829"]]
11 tokens
0347 · total · wording 2 · samec81ed8b2274771a1c2a6c8313bc0609717bbc686fce76db2725f29a49a5998c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"M898","units":9,"price":19},{"id":"K942","units":2,"price":7},{"id":"E350","units":17,"price":9},{"id":"V944","units":8,"price":5},{"id":"E309","units":4,"price":6}]
Checked answer
{"total": 402}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0348 · sort · wording 2 · same744813978c5fb86c95689e998ac13f449e16e2e4f6e494c1bc60df48ab03129c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S439","priority":8},{"id":"X422","priority":3},{"id":"U768","priority":2},{"id":"Y833","priority":2},{"id":"B287","priority":4}]
Checked answer
{"ids": ["S439", "B287"]}
Seed

wrong_schema_or_value

{"ids": ["U768", "Y833", "S439", "X422", "B287"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["S439","B287"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S439","B287"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S439","X422"]}
16 tokens
0349 · lookup · wording 3 · sameaf685cfb7234121e8566d5e458dfc4b7d4a684d67e72caab22c0cf108b7e8a6d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=D551. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"J908","city":"Bern","units":10},{"id":"M109","city":"Accra","units":16},{"id":"K294","city":"Riga","units":3},{"id":"F975","city":"Perth","units":17},{"id":"D551","city":"Suva","units":16}]
Checked answer
{"city": "Suva", "units": 16}
Seed

Correct

{"city": "Suva", "units": 16}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":16}
12 tokens
0350 · filter · wording 3 · gain2b222f99ea844395d9b0a166bb2854cfedb0d63b26fff87411880158d947c3ec
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"N493","status":"pending"},{"id":"S258","status":"closed"},{"id":"L382","status":"closed"},{"id":"B311","status":"ready"},{"id":"H516","status":"closed"}]
Checked answer
{"ids": ["B311"]}
Seed

wrong_schema_or_value

{"ids": ["B311", "H516", "N493", "S258", "L382"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B311"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["B311"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["B311"]}
11 tokens
0351 · total · wording 3 · sameebc9cebf6cedd8d090a114aac7bc39f5aec5de6bbf0114299b0fb37762895699
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"L288","units":9,"price":3},{"id":"J835","units":13,"price":12},{"id":"N401","units":10,"price":13},{"id":"J462","units":6,"price":6},{"id":"T367","units":17,"price":7}]
Checked answer
{"total": 468}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0352 · sort · wording 3 · samed5dfe6388f983ebb6c431320e3b8bf6515edcadeece65b626582dc9127c999c6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"X364","priority":1},{"id":"D255","priority":1},{"id":"T219","priority":3},{"id":"C671","priority":8},{"id":"G860","priority":7}]
Checked answer
{"ids": ["C671", "G860"]}
Seed

invalid_json_object

[{"id":"X364","priority":1},{"id":"D255","priority":1},{"id":"T219","priority":3},{"id":"C671","priority":8},{"id":"G860","priority":7}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["C671","G860"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C671","G860"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C671","D255"]}
16 tokens
0353 · lookup · wording 0 · same5473ba6fcc3287c392c80750aba8ab77184e517d0714e542b40556ee8024791f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record S608. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"F852","city":"Lima","units":18},{"id":"K758","city":"Lima","units":17},{"id":"S608","city":"Accra","units":12},{"id":"W545","city":"Riga","units":4},{"id":"E334","city":"Kyoto","units":18}]
Checked answer
{"city": "Accra", "units": 12}
Seed

Correct

{
  "city": "Accra",
  "units": 12
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":12}
12 tokens
0354 · filter · wording 0 · same2252237c7a769a1424cce7cf894ecd970b10971cfea67e8371576f1cc6804969
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"N223","status":"ready"},{"id":"P278","status":"closed"},{"id":"L710","status":"pending"},{"id":"E461","status":"ready"},{"id":"U399","status":"ready"}]
Checked answer
{"ids": ["E461", "N223", "U399"]}
Seed

wrong_schema_or_value

{"ids": ["N223", "P278", "L710", "E461", "U399"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E461","N223","U399"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["E461","N223","U399"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E461","N223","N223"]}
21 tokens
0355 · total · wording 0 · samefc33f59d061ed7d30d287d746574bf60141fb258acd41e229fc4b10308cfdde8
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Z521","units":14,"price":8},{"id":"B792","units":15,"price":15},{"id":"P277","units":13,"price":14},{"id":"A232","units":14,"price":1},{"id":"B172","units":19,"price":1}]
Checked answer
{"total": 552}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0356 · sort · wording 0 · same40683d18e49fe25a0fe4ba8e98f3177daaa8cefe978032cae9bb241163cc392d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"T225","priority":9},{"id":"Q576","priority":2},{"id":"L291","priority":6},{"id":"D231","priority":3},{"id":"C782","priority":1}]
Checked answer
{"ids": ["T225", "L291"]}
Seed

wrong_schema_or_value

{"ids": ["T225", "Q576", "L291", "D231", "C782"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["T225","L291"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["T225","L291"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["T225","D231"]}
16 tokens
0357 · lookup · wording 1 · samefd869bea8633063fecde4a5c62c69297001ac70b15dd7bec35a4b12728a4e9b9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id D923 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D923","city":"Accra","units":5},{"id":"P442","city":"Bern","units":14},{"id":"A204","city":"Suva","units":8},{"id":"F678","city":"Riga","units":2},{"id":"R120","city":"Suva","units":4}]
Checked answer
{"city": "Accra", "units": 5}
Seed

Correct

{"city":"Accra","units":5}
11 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":5}
11 tokens
0358 · filter · wording 1 · gain76ebf7b16dc67d74a5c9b0df2961f8cc8417d4f536805b1383c1628bfdfb3a66
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T905","status":"closed"},{"id":"J414","status":"closed"},{"id":"C656","status":"pending"},{"id":"P962","status":"ready"},{"id":"O126","status":"pending"}]
Checked answer
{"ids": ["P962"]}
Seed

wrong_schema_or_value

{
  "ids": ["C656", "J414", "P962", "O126"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["P962"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["P962"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["P962"]}
11 tokens
0359 · total · wording 1 · same55bba104bb082a13c3ce1c72011456b50b9fcc554def99faf8f83c00ea8a6a27
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"Y185","units":6,"price":18},{"id":"V549","units":19,"price":7},{"id":"H691","units":18,"price":11},{"id":"M576","units":6,"price":8},{"id":"T829","units":9,"price":7}]
Checked answer
{"total": 550}
Seed

wrong_schema_or_value

{"total": 102}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":548}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0360 · sort · wording 1 · same3bcb8eb1dd79b1306fa16b5140bb42275f4b601a7e52f1b5819af264409f1b58
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"B133","priority":9},{"id":"G781","priority":9},{"id":"R647","priority":6},{"id":"M360","priority":7},{"id":"X775","priority":3}]
Checked answer
{"ids": ["B133", "G781"]}
Seed

invalid_json_object

[{"id":"B133","priority":9},{"id":"G781","priority":9},{"id":"R647","priority":6},{"id":"M360","priority":7},{"id":"X775","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["B133","G781"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B133","G781"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G781","B133"]}
16 tokens
0361 · lookup · wording 2 · sameb5e48e87a66a0e6925edfa17eb49c4e5e56362752890471735a68c62f1a8b18b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"V249","city":"Kyoto","units":5},{"id":"U242","city":"Oslo","units":13},{"id":"P414","city":"Suva","units":19},{"id":"X863","city":"Kyoto","units":14},{"id":"O274","city":"Suva","units":13}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0362 · filter · wording 2 · same72329ecf5e17415d5c00298fb1270e3b421322e8a91814fc0a1c0bae4e88981b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Q497","status":"ready"},{"id":"I192","status":"closed"},{"id":"Z964","status":"closed"},{"id":"T197","status":"ready"},{"id":"N115","status":"closed"}]
Checked answer
{"ids": ["Q497", "T197"]}
Seed

invalid_json_object

[{"id":"T197","status":"ready"},{"id":"Z964","status":"closed"},{"id":"Q497","status":"ready"},{"id":"N115","status":"closed"}]
51 tokens
1 GPU · AdamW

Correct

{"ids":["Q497","T197"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Q497","T197"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["Q497","T197"]]
16 tokens
0363 · total · wording 2 · samec14c51a41f86928fe1b0ee67710654342d01da1d89efdafbd5cb2890008feedc
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R879","units":11,"price":6},{"id":"S228","units":12,"price":15},{"id":"T707","units":6,"price":1},{"id":"Z890","units":17,"price":19},{"id":"X247","units":9,"price":5}]
Checked answer
{"total": 620}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0364 · sort · wording 2 · gain79d65fe6377703f1f8c588080e0abdc1692907802e38588b4456ecd0d1e875f7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S691","priority":9},{"id":"G144","priority":3},{"id":"N990","priority":1},{"id":"R922","priority":4},{"id":"V119","priority":8}]
Checked answer
{"ids": ["S691", "V119"]}
Seed

wrong_schema_or_value

{"ids": ["S691", "G144", "N990", "R922", "V119"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["S691","V119"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S691","V119"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S691","V119"]}
16 tokens
0365 · lookup · wording 3 · gain0f0865bf5df4ed79564606ae91838d49a500e0f9c2dd43c7b1e51b8ceae93d47
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T672. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E446","city":"Riga","units":18},{"id":"K627","city":"Bern","units":16},{"id":"T672","city":"Accra","units":9},{"id":"S936","city":"Riga","units":9},{"id":"H127","city":"Lima","units":2}]
Checked answer
{"city": "Accra", "units": 9}
Seed

wrong_schema_or_value

{"city":"Riga","units":18}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":9}
11 tokens
0366 · filter · wording 3 · gaindb4459be4f74a88f919aaf291338b7733f6eb62e6b1818dc6728814773dda1cb
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"B842","status":"closed"},{"id":"F338","status":"ready"},{"id":"H588","status":"ready"},{"id":"R235","status":"ready"},{"id":"M142","status":"pending"}]
Checked answer
{"ids": ["F338", "H588", "R235"]}
Seed

wrong_schema_or_value

{"ids": ["B842", "F338", "H588", "R235", "M142"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F338","H588","R235"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["F338","H588","R235"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["F338","H588","R235"]}
21 tokens
0367 · total · wording 3 · same66e18e36762ca2ad1512e363a8b29da613b1033c37da32048eaa221b211046c0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"M961","units":0,"price":17},{"id":"T511","units":5,"price":11},{"id":"K542","units":9,"price":2},{"id":"O598","units":14,"price":19},{"id":"O828","units":7,"price":8}]
Checked answer
{"total": 395}
Seed

invalid_json_object

{
  "total": 17 + 11 + 2 + 19 + 8 = 59
}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0368 · sort · wording 3 · same350107808cbcb2e6f9af8b111623dfc956df050d315aa0a6c4e517bd2ce60f50
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"O284","priority":5},{"id":"P448","priority":6},{"id":"P614","priority":1},{"id":"P515","priority":9},{"id":"F277","priority":7}]
Checked answer
{"ids": ["P515", "F277"]}
Seed

invalid_json_object

[{"id":"O284","priority":5},{"id":"P614","priority":1},{"id":"P448","priority":6},{"id":"P515","priority":9},{"id":"F277","priority":7}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["P515","F277"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P515","O284"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O284","F277"]}
16 tokens
0369 · lookup · wording 0 · same66734863742606e17fc119550bf7f0ce582ea85194c52c93380994a6e00b09ac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record H756. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H756","city":"Lima","units":15},{"id":"P394","city":"Lima","units":5},{"id":"N824","city":"Lima","units":3},{"id":"I847","city":"Riga","units":15},{"id":"M833","city":"Bern","units":14}]
Checked answer
{"city": "Lima", "units": 15}
Seed

Correct

{
  "city": "Lima",
  "units": 15
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":15}
12 tokens
0370 · filter · wording 0 · gain96c9754c8ac7dc2abfd1fa766d79e6246d3fabe73583333724a3c3c0008c485a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"R542","status":"pending"},{"id":"B690","status":"pending"},{"id":"T228","status":"pending"},{"id":"P954","status":"closed"},{"id":"C461","status":"ready"}]
Checked answer
{"ids": ["C461"]}
Seed

wrong_schema_or_value

{"ids": ["C461", "P954", "T228", "R542", "B690"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C461"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["C461"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["C461"]}
11 tokens
0371 · total · wording 0 · same66080884a7b04c6a810d6942b682aec3f03e549908b6c477cea5d048d81cf093
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"H201","units":17,"price":2},{"id":"C697","units":10,"price":5},{"id":"Q167","units":7,"price":19},{"id":"N941","units":14,"price":5},{"id":"Y908","units":15,"price":1}]
Checked answer
{"total": 302}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0372 · sort · wording 0 · same79943deac77073e559d209e37107679cfe115390bea928d349ca8b7d04b7b9b2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J323","priority":7},{"id":"R579","priority":2},{"id":"H330","priority":6},{"id":"V271","priority":1},{"id":"G877","priority":7}]
Checked answer
{"ids": ["G877", "J323"]}
Seed

wrong_schema_or_value

{"ids": ["R579", "J323", "H330", "V271", "G877"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G877","J323"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G877","J323"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J323","G877"]}
16 tokens
0373 · lookup · wording 1 · gainf10479e869b444acbb7db8a241763d7286992d4fce22139b9b2dec86ea12d2a0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id R444 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M553","city":"Kyoto","units":19},{"id":"Z431","city":"Perth","units":11},{"id":"W633","city":"Bern","units":17},{"id":"R444","city":"Oslo","units":19},{"id":"J154","city":"Oslo","units":16}]
Checked answer
{"city": "Oslo", "units": 19}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":19}
13 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":19}
12 tokens
0374 · filter · wording 1 · gaind6d84bafb3299ea32783719f857b6a36d025c848b39391be0313938d4b916e68
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"I980","status":"ready"},{"id":"B201","status":"pending"},{"id":"V111","status":"closed"},{"id":"S151","status":"closed"},{"id":"B621","status":"pending"}]
Checked answer
{"ids": ["I980"]}
Seed

wrong_schema_or_value

{
  "ids": ["I980", "B201", "V111", "S151", "B621"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["I980"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["I980"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["I980"]}
11 tokens
0375 · total · wording 1 · samef452544d51a77fffc1327b1029aa483941f1fd35a317f146906eed82a9d1e8d7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"F504","units":16,"price":12},{"id":"E891","units":3,"price":9},{"id":"C490","units":2,"price":8},{"id":"P849","units":7,"price":12},{"id":"K366","units":10,"price":10}]
Checked answer
{"total": 419}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0376 · sort · wording 1 · gainb88b9fe6b41841324c862a821a6ed95ce928cce4c01bc8a916e84a930166f8d4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"X557","priority":4},{"id":"G482","priority":6},{"id":"W131","priority":3},{"id":"J151","priority":8},{"id":"F922","priority":3}]
Checked answer
{"ids": ["J151", "G482"]}
Seed

invalid_json_object

[{"id":"X557","priority":4},{"id":"G482","priority":6},{"id":"W131","priority":3},{"id":"J151","priority":8},{"id":"F922","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J151","G482"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J151","G482"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J151","G482"]}
16 tokens
0377 · lookup · wording 2 · gainffd5c915dafbfc02226801dffbeb7f3d211cbc953d701639c7178c1181d75acd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to A664? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"O779","city":"Lima","units":3},{"id":"Y810","city":"Perth","units":5},{"id":"U544","city":"Perth","units":11},{"id":"A664","city":"Riga","units":7},{"id":"U169","city":"Oslo","units":1}]
Checked answer
{"city": "Riga", "units": 7}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":7}
11 tokens
0378 · filter · wording 2 · samee1160770fba3815cb1e0a73b67687324b82d680ef56a821e9ce6e7a66a2a2449
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"B717","status":"closed"},{"id":"D570","status":"closed"},{"id":"T194","status":"pending"},{"id":"G119","status":"ready"},{"id":"Y573","status":"closed"}]
Checked answer
{"ids": ["G119"]}
Seed

invalid_json_object

[{"id":"G119"}]
11 tokens
1 GPU · AdamW

invalid_json_object

["G119"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["G119"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G119"]]
11 tokens
0379 · total · wording 2 · same0d92152fb18c2874051a1a7179c505ba236d85dee6272314a1d130a6f80bd134
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"F696","units":15,"price":15},{"id":"V590","units":7,"price":18},{"id":"J455","units":12,"price":15},{"id":"H281","units":1,"price":3},{"id":"Q790","units":10,"price":14}]
Checked answer
{"total": 674}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0380 · sort · wording 2 · gain2202e4f27157c53af7015e392ab52aa5da3e60c5d5f4d3895ab00f2c4bc68f11
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"C674","priority":7},{"id":"B836","priority":4},{"id":"H888","priority":6},{"id":"O982","priority":4},{"id":"G194","priority":9}]
Checked answer
{"ids": ["G194", "C674"]}
Seed

wrong_schema_or_value

{"ids": ["C674", "H888"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["G194","C674"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G194","C674"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G194","C674"]}
16 tokens
0381 · lookup · wording 3 · gaind5f07e007ee998875312f22799d03bebce91aa7dc9727b269ddc765b8acd00e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"X151","city":"Oslo","units":19},{"id":"C181","city":"Kyoto","units":5},{"id":"E815","city":"Riga","units":1},{"id":"L698","city":"Oslo","units":10},{"id":"A326","city":"Accra","units":18}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0382 · filter · wording 3 · gain1352e4ee540ed773f6db44c029bf9d8185c6d4f3fca9348769bd9b0b3eeb1aa1
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Z647","status":"pending"},{"id":"I128","status":"pending"},{"id":"L485","status":"closed"},{"id":"D374","status":"pending"},{"id":"A681","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A681", "D374", "I128", "L485", "Z647"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0383 · total · wording 3 · sameba321429085fa2cde1123329bd96fff0542aba4de1fdb28b524f1eaa12adf937
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"S576","units":13,"price":2},{"id":"S942","units":0,"price":14},{"id":"W105","units":12,"price":2},{"id":"W714","units":0,"price":16},{"id":"Q606","units":6,"price":2}]
Checked answer
{"total": 62}
Seed

wrong_schema_or_value

{
  "total": 40
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":388}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0384 · sort · wording 3 · same0dcdce0f4c0c36762e9bffbdb806e7bb634999285bb14417e53422d9077a4e26
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"I402","priority":8},{"id":"J912","priority":8},{"id":"U730","priority":5},{"id":"K527","priority":3},{"id":"D232","priority":9}]
Checked answer
{"ids": ["D232", "I402"]}
Seed

invalid_json_object

[{"id":"I402","priority":8},{"id":"J912","priority":8},{"id":"U730","priority":5},{"id":"K527","priority":3},{"id":"D232","priority":9}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D232","J912"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D232","I402"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D232","J912"]}
16 tokens
0385 · lookup · wording 0 · samed1119dd4e5dded0a7cbbdf5feb845508cfab4202f674eb4c2195c7403939841a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record R685. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"R685","city":"Riga","units":3},{"id":"V157","city":"Suva","units":8},{"id":"W906","city":"Kyoto","units":16},{"id":"M135","city":"Riga","units":4},{"id":"N455","city":"Perth","units":11}]
Checked answer
{"city": "Riga", "units": 3}
Seed

Correct

{
  "city": "Riga",
  "units": 3
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":3}
11 tokens
0386 · filter · wording 0 · same6608e23053739a1d6a46a714ba533e4f781a675fab1e851d6bab4d3aa2f1c56e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P683","status":"ready"},{"id":"N364","status":"pending"},{"id":"N561","status":"ready"},{"id":"P575","status":"closed"},{"id":"V783","status":"closed"}]
Checked answer
{"ids": ["N561", "P683"]}
Seed

wrong_schema_or_value

{"ids": ["P683", "N364", "N561", "P575", "V783"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P683","N561"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P683","N561"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N364","P683"]}
16 tokens
0387 · total · wording 0 · samefcd6be527e75edf521dfee74adbd5de55991fd2e0001a9b122a904b744169a7f
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y346","units":10,"price":17},{"id":"O585","units":15,"price":1},{"id":"T305","units":10,"price":2},{"id":"N549","units":3,"price":7},{"id":"X837","units":1,"price":6}]
Checked answer
{"total": 232}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0388 · sort · wording 0 · same9847bb2f10a147f15ddfc11dfb41d6f79c225bbaa78ef4c483adea8b35219092
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J410","priority":4},{"id":"M497","priority":8},{"id":"V721","priority":6},{"id":"K881","priority":6},{"id":"R588","priority":2}]
Checked answer
{"ids": ["M497", "K881"]}
Seed

wrong_schema_or_value

{"ids": ["M497", "V721", "K881", "J410"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["M497","K881"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["M497","V721"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["M497","J410"]}
16 tokens
0389 · lookup · wording 1 · samec6eb115e556db7cc2c96b3e123b9a18694948bdca38446149dc7b4df1896d113
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id T395 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"T395","city":"Accra","units":11},{"id":"D961","city":"Lima","units":17},{"id":"J909","city":"Suva","units":11},{"id":"D255","city":"Accra","units":2},{"id":"T872","city":"Lima","units":4}]
Checked answer
{"city": "Accra", "units": 11}
Seed

Correct

{"city":"Accra","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":11}
12 tokens
0390 · filter · wording 1 · gaine6411d512484cebd040ffa5e169132e51ca615db56a3aa77d64f35b40881812d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"Z587","status":"pending"},{"id":"D374","status":"pending"},{"id":"E220","status":"pending"},{"id":"F320","status":"closed"},{"id":"I694","status":"ready"}]
Checked answer
{"ids": ["I694"]}
Seed

wrong_schema_or_value

{
  "ids": ["I694", "F320", "D374", "E220", "Z587"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["I694"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["I694"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["I694"]}
11 tokens
0391 · total · wording 1 · samec537eb3e16fcd4cbf25bef4af65546cce0b8acce705212567358c7dc61c9d618
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"R472","units":5,"price":1},{"id":"K241","units":11,"price":19},{"id":"K808","units":17,"price":4},{"id":"V248","units":11,"price":15},{"id":"L605","units":18,"price":8}]
Checked answer
{"total": 591}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0392 · sort · wording 1 · gainf1423da48cdbacc1c6a33e4f0dc296439265a28450996531405a1fbf56f23592
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"K558","priority":7},{"id":"N431","priority":3},{"id":"Z469","priority":1},{"id":"I793","priority":6},{"id":"B280","priority":4}]
Checked answer
{"ids": ["K558", "I793"]}
Seed

invalid_json_object

[{"id":"Z469","priority":1},{"id":"I793","priority":6},{"id":"K558","priority":7},{"id":"N431","priority":3},{"id":"B280","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["K558","I793"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K558","I793"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K558","I793"]}
16 tokens
0393 · lookup · wording 2 · gain31ce103421d80711bad5ab19455100e36db4682253443229711de2ddbb3b05e5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to A440? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T784","city":"Oslo","units":14},{"id":"A440","city":"Riga","units":14},{"id":"T777","city":"Riga","units":17},{"id":"L315","city":"Oslo","units":7},{"id":"F366","city":"Bern","units":0}]
Checked answer
{"city": "Riga", "units": 14}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":14}
12 tokens
0394 · filter · wording 2 · same576a8cc259da80c9f2aa7142f820d8a98ce5d2f7329f487ce36a2c7fd86b2cd0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"U312","status":"ready"},{"id":"U588","status":"pending"},{"id":"Z499","status":"ready"},{"id":"G802","status":"ready"},{"id":"D355","status":"ready"}]
Checked answer
{"ids": ["D355", "G802", "U312", "Z499"]}
Seed

invalid_json_object

[{"id":"U312","status":"ready"},{"id":"U588","status":"pending"},{"id":"Z499","status":"ready"},{"id":"G802","status":"ready"},{"id":"D355","status":"ready"}]
63 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D355","G802","Z499"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D355","G802","Z499","U312"]}
26 tokens
4 GPUs · local windows

invalid_json_object

["ids":["D355","G802","U312","Z499"]]
26 tokens
0395 · total · wording 2 · same6ecd8853ee68ef0102c888e5539defaba087bf870b11385e1d1e2654d15f71c5
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"U432","units":1,"price":18},{"id":"Y249","units":11,"price":6},{"id":"W647","units":3,"price":7},{"id":"T915","units":19,"price":4},{"id":"L192","units":10,"price":19}]
Checked answer
{"total": 371}
Seed

wrong_schema_or_value

{"total": 108}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0396 · sort · wording 2 · same21a1b73224edcfe4405f5cb217c72ac40a8796855b25b442956950e4c0ad73a1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Z960","priority":6},{"id":"D369","priority":3},{"id":"K653","priority":4},{"id":"D196","priority":2},{"id":"G938","priority":1}]
Checked answer
{"ids": ["Z960", "K653"]}
Seed

wrong_schema_or_value

{"ids": ["D369", "D196", "G938", "K653", "Z960"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Z960","D369"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Z960","D369"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z960","D369"]}
16 tokens
0397 · lookup · wording 3 · samebe46cdd8f1b5868e966c45003dc19e6379f5703f340554f345bd40f431304890
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=P935. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P935","city":"Oslo","units":6},{"id":"E763","city":"Suva","units":12},{"id":"B865","city":"Oslo","units":4},{"id":"N695","city":"Accra","units":11},{"id":"Y775","city":"Perth","units":4}]
Checked answer
{"city": "Oslo", "units": 6}
Seed

Correct

{"city":"Oslo","units":6}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":6}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":6}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":6}
11 tokens
0398 · filter · wording 3 · gain74118666b55dcd015e9f00afe1ff2a96653f4683f6b423be43413376ef7dfcf9
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"R244","status":"ready"},{"id":"Z873","status":"closed"},{"id":"D106","status":"closed"},{"id":"F488","status":"pending"},{"id":"I716","status":"pending"}]
Checked answer
{"ids": ["R244"]}
Seed

wrong_schema_or_value

{"ids": ["R244", "Z873", "D106", "F488", "I716"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["R244"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["R244"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["R244"]}
11 tokens
0399 · total · wording 3 · sameb4263898ede62ee2cf68f5c71e41b782aa92d0867ff5db999a57401f6fd6bcc7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"H945","units":10,"price":6},{"id":"A393","units":5,"price":9},{"id":"B636","units":6,"price":15},{"id":"X841","units":1,"price":10},{"id":"P130","units":13,"price":5}]
Checked answer
{"total": 270}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":378}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0400 · sort · wording 3 · same45546f54f934e92a6fcae3cc82c433a1cc84ce610f0e6d2483290e9fe628928d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"P148","priority":6},{"id":"N382","priority":9},{"id":"R876","priority":7},{"id":"V576","priority":6},{"id":"X810","priority":4}]
Checked answer
{"ids": ["N382", "R876"]}
Seed

invalid_json_object

[{"id":"P148","priority":6},{"id":"R876","priority":7},{"id":"V576","priority":6},{"id":"N382","priority":9},{"id":"X810","priority":4}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["N382","P148"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N382","P148"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N382","P148"]}
16 tokens
0401 · lookup · wording 0 · gain8ce38e46c5b9b9409c9f3e1476af218664c1d03fe93cf2fbc495949d43cbb26d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"C742","city":"Oslo","units":16},{"id":"N446","city":"Kyoto","units":7},{"id":"W293","city":"Perth","units":5},{"id":"V441","city":"Bern","units":16},{"id":"D421","city":"Perth","units":2}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Oslo",
  "units": 16
}
19 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0402 · filter · wording 0 · gain1c547ae57d0e1a1ca4d11880c1ea124a6ef51b039497552be70a52fd554dc6e9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Q643","status":"ready"},{"id":"Q999","status":"closed"},{"id":"T787","status":"closed"},{"id":"E210","status":"ready"},{"id":"Q613","status":"closed"}]
Checked answer
{"ids": ["E210", "Q643"]}
Seed

wrong_schema_or_value

{"ids": ["E210", "Q613", "Q643", "Q999", "T787"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E210","Q643"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E210","Q643"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E210","Q643"]}
16 tokens
0403 · total · wording 0 · same5b653d3a46b965bf29ec55f0063a7d4e3a4b9e9c6fed71e01eaf313575b51446
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P191","units":6,"price":4},{"id":"K469","units":3,"price":12},{"id":"J450","units":8,"price":7},{"id":"U563","units":16,"price":1},{"id":"H854","units":8,"price":4}]
Checked answer
{"total": 164}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0404 · sort · wording 0 · gain1a47f3db4db767f22d6eec87d5ca49cd2bbf14bcf828bf6dfdc5bf15c8922790
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J466","priority":9},{"id":"M925","priority":1},{"id":"V691","priority":1},{"id":"P937","priority":2},{"id":"U984","priority":9}]
Checked answer
{"ids": ["J466", "U984"]}
Seed

wrong_schema_or_value

{"ids": ["M925", "V691", "P937", "J466"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["J466","U984"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J466","U984"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J466","U984"]}
16 tokens
0405 · lookup · wording 1 · same9d4f8c2975de6d2d77ad6a620fb43e3a6e97697479c64f4c77466620e3f80c27
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id E354 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"E354","city":"Perth","units":16},{"id":"G537","city":"Perth","units":18},{"id":"P482","city":"Perth","units":2},{"id":"L504","city":"Oslo","units":4},{"id":"C505","city":"Oslo","units":8}]
Checked answer
{"city": "Perth", "units": 16}
Seed

Correct

{"city":"Perth","units":16}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":16}
12 tokens
0406 · filter · wording 1 · gainc95db303556de29ade9094c52b320e2f927903f078af1dfde150cfcf19c481b6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"N531","status":"ready"},{"id":"K916","status":"ready"},{"id":"M185","status":"ready"},{"id":"E973","status":"closed"},{"id":"S307","status":"pending"}]
Checked answer
{"ids": ["K916", "M185", "N531"]}
Seed

wrong_schema_or_value

{
  "ids": ["M185", "N531", "K916", "E973"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["K916","M185","N531"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["K916","M185","N531"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["K916","M185","N531"]}
21 tokens
0407 · total · wording 1 · same3a3e6f49d35a6f673aeaed6c398e5ca8f93ffab4be2acad86fd087d3b724ead7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D106","units":3,"price":17},{"id":"J980","units":5,"price":12},{"id":"X994","units":19,"price":11},{"id":"R553","units":3,"price":18},{"id":"D350","units":0,"price":3}]
Checked answer
{"total": 374}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0408 · sort · wording 1 · gain0b5c3febe3c101a4bc41591bb6607580309a24cfa5720f27d0be85ff81942982
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"H592","priority":4},{"id":"A621","priority":8},{"id":"H500","priority":4},{"id":"X433","priority":7},{"id":"Z157","priority":2}]
Checked answer
{"ids": ["A621", "X433"]}
Seed

invalid_json_object

[{"id":"H592","priority":4},{"id":"A621","priority":8},{"id":"H500","priority":4},{"id":"X433","priority":7},{"id":"Z157","priority":2}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A621","X433"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A621","H500"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A621","X433"]}
16 tokens
0409 · lookup · wording 2 · gain13b07dcfd6500bda3237f6309106a27868a560b155fb21c6395f10fcdc92b6fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M657? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"M657","city":"Oslo","units":0},{"id":"U581","city":"Oslo","units":19},{"id":"C132","city":"Perth","units":9},{"id":"T227","city":"Accra","units":15},{"id":"D668","city":"Accra","units":9}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":0}
11 tokens
0410 · filter · wording 2 · same9e91e6cf801fe052d7de552079cac5697c16f00a53618ffff4bf0b0d29b48575
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"G534","status":"closed"},{"id":"L840","status":"ready"},{"id":"U678","status":"closed"},{"id":"U340","status":"pending"},{"id":"U796","status":"pending"}]
Checked answer
{"ids": ["L840"]}
Seed

invalid_json_object

[{"id":"U340","status":"pending"},{"id":"U796","status":"pending"},{"id":"U678","status":"closed"},{"id":"G534","status":"closed"},{"id":"L840","status":"ready"}]
63 tokens
1 GPU · AdamW

invalid_json_object

["L840"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["L840"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["L840"]]
11 tokens
0411 · total · wording 2 · same54b89f01ba9dc201188164279ea09556c6272639d47b7979b97f72f99fa811f8
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y499","units":12,"price":14},{"id":"Y999","units":2,"price":15},{"id":"V178","units":4,"price":11},{"id":"X247","units":10,"price":11},{"id":"W657","units":10,"price":2}]
Checked answer
{"total": 372}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0412 · sort · wording 2 · gaind582088249d7740b482ea23ab16ff4141a3128d0776146bec843421673a5a924
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"T891","priority":7},{"id":"X196","priority":5},{"id":"B412","priority":4},{"id":"P711","priority":7},{"id":"R570","priority":3}]
Checked answer
{"ids": ["P711", "T891"]}
Seed

wrong_schema_or_value

{"ids": ["B412", "P711"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["P711","T891"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P711","T891"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P711","T891"]}
16 tokens
0413 · lookup · wording 3 · gain79eda8ffebe760aa9283fd469ff3846ce989f2aefdd50adbce431ef4fa49b682
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=S148. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"S333","city":"Perth","units":3},{"id":"S148","city":"Accra","units":4},{"id":"A270","city":"Suva","units":16},{"id":"F786","city":"Bern","units":14},{"id":"H695","city":"Kyoto","units":17}]
Checked answer
{"city": "Accra", "units": 4}
Seed

wrong_schema_or_value

{"city":"Perth","units":3}
11 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":4}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":4}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":4}
11 tokens
0414 · filter · wording 3 · gain3b0b26ba531f195962ae4d395ac6a71ad95bd40c756b89a1d0673f9c15b9dda1
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"K950","status":"ready"},{"id":"J742","status":"ready"},{"id":"F727","status":"pending"},{"id":"X206","status":"closed"},{"id":"Y995","status":"pending"}]
Checked answer
{"ids": ["J742", "K950"]}
Seed

wrong_schema_or_value

{"ids": ["J742", "K950", "F727", "X206", "Y995"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J742","K950"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J742","K950"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J742","K950"]}
16 tokens
0415 · total · wording 3 · same2686486dbcbf221e7433aaf3b3960ee0d5d2ae52a65d8f691f24518a2411ebdf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"U107","units":12,"price":15},{"id":"E579","units":13,"price":7},{"id":"T222","units":3,"price":5},{"id":"T754","units":13,"price":2},{"id":"M178","units":19,"price":11}]
Checked answer
{"total": 521}
Seed

wrong_schema_or_value

{
  "total": 34
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0416 · sort · wording 3 · sameab8b8eba65e77928740566b9a073846e075771e9a3333806f398e9a4a1ec1b31
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"W920","priority":7},{"id":"E454","priority":3},{"id":"A497","priority":2},{"id":"D822","priority":4},{"id":"L853","priority":5}]
Checked answer
{"ids": ["W920", "L853"]}
Seed

invalid_json_object

[{"id":"W920","priority":7},{"id":"D822","priority":4}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["W920","L853"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["W920","L853"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["W920","E454"]}
16 tokens
0417 · lookup · wording 0 · gain6d3db8a39acd4d3f38275536ddce72be4aecef9333287e74cbe99e81a8069fe3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record D407. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"U812","city":"Riga","units":2},{"id":"E342","city":"Oslo","units":7},{"id":"P585","city":"Perth","units":11},{"id":"D407","city":"Perth","units":13},{"id":"E254","city":"Lima","units":6}]
Checked answer
{"city": "Perth", "units": 13}
Seed

invalid_json_object

[{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga","units":2},{"city":"Riga
96 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":13}
12 tokens
0418 · filter · wording 0 · gaine7138a29743ef136832721ce9d3466c5ef6794849626d315607f3a5382baf1c3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"W420","status":"ready"},{"id":"E646","status":"pending"},{"id":"J606","status":"closed"},{"id":"I223","status":"closed"},{"id":"X344","status":"pending"}]
Checked answer
{"ids": ["W420"]}
Seed

wrong_schema_or_value

{"ids": ["E646", "J606", "I223", "W420", "X344"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["W420"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["W420"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["W420"]}
11 tokens
0419 · total · wording 0 · same90c64680be9984bd75fd422df3cdbbb15f5310038b0d3f5e296fcc5d07754a06
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P620","units":2,"price":7},{"id":"W684","units":0,"price":11},{"id":"N321","units":10,"price":9},{"id":"J365","units":10,"price":10},{"id":"F634","units":5,"price":14}]
Checked answer
{"total": 274}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0420 · sort · wording 0 · same4050473183de365934372fe96c3c188134c2992cb0c27f826a72641464c5de52
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"O555","priority":1},{"id":"Q200","priority":5},{"id":"L246","priority":5},{"id":"I498","priority":9},{"id":"J728","priority":5}]
Checked answer
{"ids": ["I498", "J728"]}
Seed

wrong_schema_or_value

{"ids": ["O555", "J728", "I498", "L246", "Q200"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["I498","J728"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I498","J728"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I498","O555"]}
16 tokens
0421 · lookup · wording 1 · gain28a6097c40d8f15640bc8338cf20758d4a3a2639330d36be04327f64a29c2a73
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"X633","city":"Riga","units":8},{"id":"A839","city":"Kyoto","units":18},{"id":"U666","city":"Oslo","units":2},{"id":"F414","city":"Kyoto","units":13},{"id":"H403","city":"Kyoto","units":1}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":8}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0422 · filter · wording 1 · samebe680a1ad5fd578bed196e1461b710506d00cc010d48fde57bbe09a5f1990b46
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"V706","status":"closed"},{"id":"O206","status":"pending"},{"id":"P586","status":"ready"},{"id":"U244","status":"ready"},{"id":"O870","status":"pending"}]
Checked answer
{"ids": ["P586", "U244"]}
Seed

wrong_schema_or_value

{
  "ids": ["O206", "O870", "P586", "U244"]
}
32 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["U244","P586"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P586","U244"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O206","U244"]}
16 tokens
0423 · total · wording 1 · same49891277734e0f499321a24b116804ee0924e397604101e9387e3d836f3f9eec
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"G308","units":0,"price":1},{"id":"D489","units":0,"price":1},{"id":"F751","units":4,"price":11},{"id":"E915","units":8,"price":6},{"id":"G508","units":15,"price":18}]
Checked answer
{"total": 362}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":288}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":288}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0424 · sort · wording 1 · same6c2600cbfa358b51e83af88007f993c62d9051103a9f5100f0b563510a156fd0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"O743","priority":2},{"id":"C949","priority":8},{"id":"Z794","priority":4},{"id":"R935","priority":3},{"id":"W113","priority":5}]
Checked answer
{"ids": ["C949", "W113"]}
Seed

invalid_json_object

[{"id":"O743","priority":2},{"id":"Z794","priority":4},{"id":"R935","priority":3},{"id":"W113","priority":5}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["C949","W113"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["C949","O743"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C949","O743"]}
16 tokens
0425 · lookup · wording 2 · gain8ac98dd309dae9b01ad2e36c83a4de229f7e6eb55f38cec2520858525cb62657
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to A920? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"W629","city":"Kyoto","units":8},{"id":"Z843","city":"Lima","units":6},{"id":"K438","city":"Kyoto","units":6},{"id":"A920","city":"Riga","units":12},{"id":"B105","city":"Riga","units":3}]
Checked answer
{"city": "Riga", "units": 12}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":12}
12 tokens
0426 · filter · wording 2 · same57305e16b986a3d55053113feb4c1d2766de90930d39585c9a7366a4d378bad4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Y237","status":"pending"},{"id":"M122","status":"ready"},{"id":"B202","status":"closed"},{"id":"Z859","status":"ready"},{"id":"S971","status":"ready"}]
Checked answer
{"ids": ["M122", "S971", "Z859"]}
Seed

invalid_json_object

[{"id":"M122","status":"ready"},{"id":"S971","status":"ready"},{"id":"Y237","status":"pending"},{"id":"B202","status":"closed"},{"id":"Z859","status":"ready"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["M122","S971","Z859"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["M122","S971","Z859"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["S971","M122","Z859"]]
21 tokens
0427 · total · wording 2 · samea6b6e4a826deb901f83510ac31519a4528d2abf31aa56a50d25be92eaa9405d9
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R164","units":15,"price":1},{"id":"L714","units":12,"price":17},{"id":"D820","units":4,"price":12},{"id":"N587","units":13,"price":13},{"id":"C991","units":10,"price":3}]
Checked answer
{"total": 466}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0428 · sort · wording 2 · gain993fba6b6c832ca68a701e00599b918bc6d77df7712fdfe9fec903fdcb7568e8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"X893","priority":7},{"id":"K348","priority":1},{"id":"B288","priority":5},{"id":"L747","priority":3},{"id":"P478","priority":6}]
Checked answer
{"ids": ["X893", "P478"]}
Seed

wrong_schema_or_value

{"ids": ["X893", "K348", "B288", "L747", "P478"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["X893","P478"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["X893","P478"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X893","P478"]}
16 tokens
0429 · lookup · wording 3 · same5486519f73b76cc4fab8902e7905b5c112c23b0799112d986c2a18b7e775b3d3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=R481. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"R481","city":"Lima","units":0},{"id":"I244","city":"Lima","units":6},{"id":"P392","city":"Accra","units":2},{"id":"I950","city":"Riga","units":19},{"id":"D684","city":"Kyoto","units":17}]
Checked answer
{"city": "Lima", "units": 0}
Seed

Correct

{"city": "Lima", "units": 0}
14 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":0}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":0}
11 tokens
0430 · filter · wording 3 · gainc6c0502ca54ed3fc50afc07b7415d2028bc35275f5abb0495ec00b646746af0a
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"A593","status":"pending"},{"id":"O556","status":"closed"},{"id":"T913","status":"closed"},{"id":"F901","status":"pending"},{"id":"F865","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A593", "O556", "T913", "F865", "F901"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0431 · total · wording 3 · sameca3c2d44c9cdd4943995e755c9c05b6519ad84bd96df44d5604caec87b46cd92
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"J845","units":5,"price":4},{"id":"R908","units":15,"price":7},{"id":"P270","units":9,"price":10},{"id":"E540","units":11,"price":6},{"id":"I861","units":14,"price":8}]
Checked answer
{"total": 393}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0432 · sort · wording 3 · same4538c728dc4216aef4ee5fec7cecaa36a26c8a86fbd0052e01768d2ee0313bdc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"P366","priority":2},{"id":"R709","priority":2},{"id":"J387","priority":9},{"id":"H531","priority":8},{"id":"G873","priority":9}]
Checked answer
{"ids": ["G873", "J387"]}
Seed

invalid_json_object

[{"id":"P366"},{"id":"R709"},{"id":"J387"},{"id":"H531"},{"id":"G873"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["G873","J387"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G873","J387"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J387","G873"]}
16 tokens
0433 · lookup · wording 0 · samec3fb43b35d6d0bafb6847b5cb58173fdc5cd7781fd9ba98f42d63f85e8fb5351
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record P964. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"R986","city":"Lima","units":13},{"id":"W827","city":"Perth","units":6},{"id":"P964","city":"Bern","units":5},{"id":"W487","city":"Accra","units":4},{"id":"I698","city":"Oslo","units":14}]
Checked answer
{"city": "Bern", "units": 5}
Seed

Correct

{
  "city": "Bern",
  "units": 5
}
17 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":5}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":5}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":5}
10 tokens
0434 · filter · wording 0 · gain95240ceb3a43ac50ee749a27a7367f6239c0bd8ecc723c83c98dc4dfb989e635
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"T181","status":"ready"},{"id":"Y678","status":"pending"},{"id":"S726","status":"pending"},{"id":"X979","status":"ready"},{"id":"L400","status":"pending"}]
Checked answer
{"ids": ["T181", "X979"]}
Seed

wrong_schema_or_value

{"ids": ["X979", "L400", "T181", "S726", "Y678"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X979","T181"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X979","T181"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["T181","X979"]}
16 tokens
0435 · total · wording 0 · same1479b00e26b9e1baf3d3b6a5218c9da2ebd8a2ecab30f05d953bf3d826be4644
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"D270","units":14,"price":18},{"id":"Q506","units":4,"price":15},{"id":"F993","units":16,"price":1},{"id":"V671","units":4,"price":2},{"id":"N620","units":7,"price":6}]
Checked answer
{"total": 378}
Seed

invalid_json_object

{"total": 14 * 18 + 4 * 15 + 16 * 1 + 4 * 2 + 7 * 6}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0436 · sort · wording 0 · same1632f5322ae48268c7f65d52c53434f9729fbe3c86de80b8c15760e3c509e8d9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"I729","priority":9},{"id":"I163","priority":8},{"id":"V293","priority":7},{"id":"J521","priority":6},{"id":"M867","priority":4}]
Checked answer
{"ids": ["I729", "I163"]}
Seed

wrong_schema_or_value

{"ids": ["I729", "V293", "J521", "M867"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["I729","I163"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I729","I163"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I729","J521"]}
16 tokens
0437 · lookup · wording 1 · gain7e88fd168a5b1f932f9ff70b53c3ee816ed2f74d86b07b5783b03cebe58a056a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id W390 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"G799","city":"Oslo","units":1},{"id":"W390","city":"Riga","units":9},{"id":"E411","city":"Oslo","units":7},{"id":"G131","city":"Kyoto","units":4},{"id":"H936","city":"Kyoto","units":17}]
Checked answer
{"city": "Riga", "units": 9}
Seed

wrong_schema_or_value

{"city":"Oslo","units":1}
11 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":9}
11 tokens
0438 · filter · wording 1 · gain01aa07e99120854396146ed108f8071e56fdd4252e5970a5651031408525754e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"S488","status":"pending"},{"id":"D819","status":"closed"},{"id":"Q878","status":"pending"},{"id":"Z292","status":"closed"},{"id":"T692","status":"ready"}]
Checked answer
{"ids": ["T692"]}
Seed

wrong_schema_or_value

{
  "ids": ["S488", "D819", "Q878", "Z292", "T692"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["T692"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T692"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["T692"]}
11 tokens
0439 · total · wording 1 · same7723f9ca422b2917f117904894cb899d4379db5d85af1a8422be5aea6bf40bc9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"M384","units":19,"price":7},{"id":"V397","units":10,"price":8},{"id":"S506","units":18,"price":3},{"id":"D594","units":4,"price":9},{"id":"I654","units":15,"price":13}]
Checked answer
{"total": 498}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0440 · sort · wording 1 · gainf2a9cf56bb080f13c54e8124860cc2d4e375363ac4f54344bbf2f833ac31314e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Q641","priority":2},{"id":"Z565","priority":9},{"id":"G318","priority":1},{"id":"C642","priority":6},{"id":"R529","priority":5}]
Checked answer
{"ids": ["Z565", "C642"]}
Seed

invalid_json_object

[{"id":"Q641","priority":2},{"id":"G318","priority":1},{"id":"C642","priority":6},{"id":"R529","priority":5}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["Z565","C642"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Z565","C642"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z565","C642"]}
16 tokens
0441 · lookup · wording 2 · sameb5bc3059db8d543c6347a0993a1dda25b98de0da17a44329665b21d0ac271b98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"H999","city":"Accra","units":15},{"id":"K245","city":"Suva","units":3},{"id":"N421","city":"Suva","units":5},{"id":"U189","city":"Suva","units":12},{"id":"E804","city":"Accra","units":4}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0442 · filter · wording 2 · same6a4dca10c5756ecbc0d637ad13f4d205c0523fa0533a53ee7c443059cf192115
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"E760","status":"pending"},{"id":"O785","status":"pending"},{"id":"Y629","status":"closed"},{"id":"X668","status":"closed"},{"id":"F238","status":"pending"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"E760"},{"id":"O785"},{"id":"Y629"},{"id":"X668"},{"id":"F238"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

invalid_json_object

["ids":[]}
6 tokens
0443 · total · wording 2 · same50ea10a13a3def4d867c63b9148c4b6e450bfe5b8a0a9be4f75067d167666d94
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"E499","units":18,"price":11},{"id":"H866","units":10,"price":10},{"id":"R753","units":12,"price":7},{"id":"X724","units":8,"price":8},{"id":"K919","units":11,"price":16}]
Checked answer
{"total": 622}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0444 · sort · wording 2 · same32a046231f5f7b77cf7fbfb83a49156601a3780c0ef7dc8a72f5437974ffc9e9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"N996","priority":6},{"id":"S133","priority":7},{"id":"Q902","priority":2},{"id":"P736","priority":6},{"id":"G534","priority":6}]
Checked answer
{"ids": ["S133", "G534"]}
Seed

wrong_schema_or_value

{"ids": ["N996", "S133", "Q902", "P736", "G534"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["S133","N996"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["S133","N996"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S133","N996"]}
16 tokens
0445 · lookup · wording 3 · gain88752b96b8d670a73d3735a7ad9801577bd62487630754aa587826430410dd0d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=G839. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"G427","city":"Riga","units":7},{"id":"X925","city":"Accra","units":1},{"id":"G839","city":"Perth","units":11},{"id":"T806","city":"Suva","units":12},{"id":"X627","city":"Accra","units":8}]
Checked answer
{"city": "Perth", "units": 11}
Seed

wrong_schema_or_value

{"city":"Riga","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":11}
12 tokens
0446 · filter · wording 3 · gaind2b2321596f90d7309b47d4d7f06eaa8423a57371e63b6958ae9d75f1bf96886
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"S433","status":"pending"},{"id":"J732","status":"pending"},{"id":"J195","status":"ready"},{"id":"I934","status":"ready"},{"id":"N862","status":"pending"}]
Checked answer
{"ids": ["I934", "J195"]}
Seed

wrong_schema_or_value

{"ids": ["I934", "J195", "J732", "S433", "N862"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["I934","J195"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I934","J195"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I934","J195"]}
16 tokens
0447 · total · wording 3 · sameacc65b62a8b8c074db7bc9ead22c29c9e85ecf6a3c2515d77c0d126e52c7c2f3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"I287","units":2,"price":19},{"id":"X896","units":7,"price":13},{"id":"T586","units":15,"price":11},{"id":"M528","units":5,"price":11},{"id":"C729","units":14,"price":12}]
Checked answer
{"total": 517}
Seed

invalid_json_object

{
  "total": 19 + 13 + 15 + 11 + 11 + 12 + 14 + 12 = 99
}
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0448 · sort · wording 3 · same63d0da9607eb03277fc850eaeea95cc12fa11bcd7d288e4d65c3875a415a55b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"I792","priority":6},{"id":"E896","priority":5},{"id":"R967","priority":8},{"id":"V467","priority":8},{"id":"N816","priority":8}]
Checked answer
{"ids": ["N816", "R967"]}
Seed

invalid_json_object

[{"id":"I792","priority":6},{"id":"R967","priority":8},{"id":"V467","priority":8},{"id":"N816","priority":8}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["N816","R967"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["R967","N816"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N816","V467"]}
16 tokens
0449 · lookup · wording 0 · samec93ddcd594b5578dea1e838c2814d0b63de45494ba25dbef90737fdb297a4f14
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record A905. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"M563","city":"Lima","units":10},{"id":"M690","city":"Suva","units":8},{"id":"A905","city":"Accra","units":2},{"id":"W136","city":"Lima","units":0},{"id":"Z274","city":"Oslo","units":1}]
Checked answer
{"city": "Accra", "units": 2}
Seed

Correct

{
  "city": "Accra",
  "units": 2
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":2}
11 tokens
0450 · filter · wording 0 · gainf663f9715a0e6aa7a0114a74277b7976edc966e336f7fea0377e8d34b5139306
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"F639","status":"ready"},{"id":"G866","status":"closed"},{"id":"B330","status":"pending"},{"id":"K451","status":"pending"},{"id":"U434","status":"closed"}]
Checked answer
{"ids": ["F639"]}
Seed

wrong_schema_or_value

{"ids": ["B330", "F639", "K451", "U434", "G866"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F639"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["F639"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["F639"]}
11 tokens
0451 · total · wording 0 · same5744b7d2d79d48d4c7c519ecbc96b3ad1d4bb85bac3f7871ecfc7288b5f795fa
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"I494","units":6,"price":8},{"id":"X950","units":17,"price":7},{"id":"C626","units":16,"price":5},{"id":"Z161","units":0,"price":12},{"id":"V823","units":3,"price":16}]
Checked answer
{"total": 295}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0452 · sort · wording 0 · gain397bae5f088a5c34887778286322905a6edea57fa932df708ae7f2f528d690b4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z669","priority":5},{"id":"L807","priority":6},{"id":"Y814","priority":9},{"id":"T130","priority":2},{"id":"F245","priority":3}]
Checked answer
{"ids": ["Y814", "L807"]}
Seed

wrong_schema_or_value

{"ids": ["Y814", "T130", "F245", "L807", "Z669"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Y814","L807"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Y814","L807"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Y814","L807"]}
16 tokens
0453 · lookup · wording 1 · gain9d19ec914e65d010620d064dccac4465489936aa0d5375291d55eaf3b66ca309
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id R645 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"P635","city":"Perth","units":0},{"id":"C818","city":"Riga","units":8},{"id":"K964","city":"Riga","units":7},{"id":"R645","city":"Oslo","units":15},{"id":"T234","city":"Bern","units":16}]
Checked answer
{"city": "Oslo", "units": 15}
Seed

wrong_schema_or_value

{"city":"Perth","units":0}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":15}
12 tokens
0454 · filter · wording 1 · gain3995136380dc88f9a4f376fa2bbbe227a32de20ba48698f28418ca5ef4014b94
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"R607","status":"pending"},{"id":"S993","status":"pending"},{"id":"J367","status":"ready"},{"id":"Z504","status":"pending"},{"id":"I587","status":"closed"}]
Checked answer
{"ids": ["J367"]}
Seed

wrong_schema_or_value

{
  "ids": ["J367", "I587", "Z504"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":["J367"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["J367"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["J367"]}
11 tokens
0455 · total · wording 1 · same755192619b6f60b6f3ac0e9c2becb58b1a4d1103dce1813405b676d9bc8d7093
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L278","units":10,"price":10},{"id":"O685","units":3,"price":1},{"id":"A943","units":7,"price":15},{"id":"I211","units":12,"price":6},{"id":"D399","units":12,"price":18}]
Checked answer
{"total": 496}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":408}
8 tokens
0456 · sort · wording 1 · same4a5fc998d7b66626e76e788ad200224af292356dcc442047cab35bba89a0670c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J883","priority":2},{"id":"Z935","priority":9},{"id":"L587","priority":2},{"id":"B464","priority":5},{"id":"S671","priority":2}]
Checked answer
{"ids": ["Z935", "B464"]}
Seed

invalid_json_object

[{"id":"J883","priority":2},{"id":"L587","priority":2},{"id":"B464","priority":5},{"id":"S671","priority":2}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["Z935","B464"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Z935","J883"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z935","J883"]}
16 tokens
0457 · lookup · wording 2 · gain0902018fb5a060d2d3046a97d8893997a34f1d4ab566774409343cc9f7e1cc00
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to U618? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"U618","city":"Lima","units":8},{"id":"Z978","city":"Lima","units":1},{"id":"V466","city":"Riga","units":4},{"id":"J547","city":"Accra","units":14},{"id":"A846","city":"Riga","units":19}]
Checked answer
{"city": "Lima", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":8}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":8}
11 tokens
0458 · filter · wording 2 · sameaa8ddf2449557a252da8d12e00b8f593583c7d2f7d6a8fd3012a1962f1ae4744
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"O866","status":"ready"},{"id":"S494","status":"pending"},{"id":"O416","status":"pending"},{"id":"M512","status":"ready"},{"id":"C148","status":"closed"}]
Checked answer
{"ids": ["M512", "O866"]}
Seed

invalid_json_object

[{"id":"O866","status":"ready"},{"id":"M512","status":"ready"},{"id":"O416","status":"pending"},{"id":"C148","status":"closed"},{"id":"S494","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["M512","O866"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M512","O866"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["O416","M512"]]
16 tokens
0459 · total · wording 2 · same39849480d648584623a60fc85dd5a22f19b7ebb50ad6d0cc1fb0819114cc6225
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V949","units":3,"price":11},{"id":"W558","units":19,"price":9},{"id":"M875","units":7,"price":18},{"id":"R336","units":8,"price":15},{"id":"D678","units":14,"price":10}]
Checked answer
{"total": 590}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0460 · sort · wording 2 · same83ebec0d0ae3ac22d3a519447d95ee0db217cc4fe742dffeb66fdf397691599a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J210","priority":4},{"id":"C846","priority":1},{"id":"P743","priority":8},{"id":"Q105","priority":8},{"id":"C413","priority":6}]
Checked answer
{"ids": ["P743", "Q105"]}
Seed

wrong_schema_or_value

{"ids": ["C846", "P743"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["P743","Q105"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P743","Q105"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q105","J210"]}
16 tokens
0461 · lookup · wording 3 · gain36361676617e08ac4d5a1ea9d7da17ed93a9c23a2ebc02453fb8c1203c5c5433
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"W934","city":"Riga","units":13},{"id":"C677","city":"Perth","units":11},{"id":"V237","city":"Perth","units":7},{"id":"U298","city":"Oslo","units":15},{"id":"E139","city":"Lima","units":17}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0462 · filter · wording 3 · gainf264bac1041d78b433cfea69b5153612555cfe03abc0a401e19a97236e81437a
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"T693","status":"closed"},{"id":"K290","status":"closed"},{"id":"D761","status":"closed"},{"id":"M641","status":"closed"},{"id":"Q297","status":"ready"}]
Checked answer
{"ids": ["Q297"]}
Seed

wrong_schema_or_value

{"ids": ["D761", "M641", "Q297", "K290", "T693"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Q297"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Q297"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Q297"]}
11 tokens
0463 · total · wording 3 · same6293201b7acb9522e57e206fe40cc8bfd3adeb73dff1a689f95e10c2947781ab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"E244","units":7,"price":5},{"id":"F901","units":17,"price":12},{"id":"W547","units":19,"price":2},{"id":"I703","units":1,"price":2},{"id":"S573","units":19,"price":16}]
Checked answer
{"total": 583}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0464 · sort · wording 3 · gainbe21855ead5ce9606958b64a03d043fe56ae571e5ec1733727fdd0511177636c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"K577","priority":6},{"id":"T863","priority":3},{"id":"Y233","priority":4},{"id":"M216","priority":5},{"id":"B297","priority":8}]
Checked answer
{"ids": ["B297", "K577"]}
Seed

invalid_json_object

[{"id":"K577","priority":6},{"id":"T863","priority":3},{"id":"Y233","priority":4},{"id":"M216","priority":5},{"id":"B297","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["B297","K577"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B297","K577"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B297","K577"]}
16 tokens
0465 · lookup · wording 0 · same81eedf6e63a8fa380eaed868dfca3ba9fb1da1c69e4731f49869783388456e3e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record N797. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"D304","city":"Oslo","units":4},{"id":"B448","city":"Perth","units":4},{"id":"N797","city":"Lima","units":18},{"id":"L305","city":"Accra","units":14},{"id":"Z536","city":"Kyoto","units":11}]
Checked answer
{"city": "Lima", "units": 18}
Seed

Correct

{
  "city": "Lima",
  "units": 18
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":18}
12 tokens
0466 · filter · wording 0 · gain8ba8f53fa0863c6e855d5b4ccda84173e601acf52e51214d93830cb2a923f97c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P687","status":"pending"},{"id":"Y234","status":"closed"},{"id":"D730","status":"closed"},{"id":"A876","status":"ready"},{"id":"G343","status":"ready"}]
Checked answer
{"ids": ["A876", "G343"]}
Seed

wrong_schema_or_value

{"ids": ["A876", "G343", "P687", "Y234", "D730"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A876","G343"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A876","G343"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A876","G343"]}
16 tokens
0467 · total · wording 0 · same6e5b2c39b34e99a7f9b14bb96da46067d8feb3bf5b80f8fe6282d679c62718cb
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C884","units":11,"price":18},{"id":"O984","units":1,"price":17},{"id":"G817","units":14,"price":10},{"id":"V868","units":7,"price":9},{"id":"T203","units":15,"price":3}]
Checked answer
{"total": 463}
Seed

invalid_json_object

{"total": 11*18 + 1*17 + 14*10 + 7*9 + 15*3 = 205}
40 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0468 · sort · wording 0 · same1380aa73662251bb9f7b4a4232d49176ed19b05ee99494d556afe993c9ead571
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"P714","priority":3},{"id":"D308","priority":2},{"id":"F792","priority":7},{"id":"A742","priority":1},{"id":"R439","priority":2}]
Checked answer
{"ids": ["F792", "P714"]}
Seed

wrong_schema_or_value

{"ids": ["A742", "D308", "F792", "P714", "R439"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F792","P714"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F792","P714"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F792","D308"]}
16 tokens
0469 · lookup · wording 1 · samef0b8f985433177edc8ab673f682d6aa44e38d8da388978c814df4d26d27efb20
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id H848 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"X769","city":"Bern","units":6},{"id":"U126","city":"Riga","units":16},{"id":"Z702","city":"Riga","units":6},{"id":"H848","city":"Accra","units":12},{"id":"M985","city":"Oslo","units":15}]
Checked answer
{"city": "Accra", "units": 12}
Seed

Correct

{"city":"Accra","units":12}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":12}
12 tokens
0470 · filter · wording 1 · gaina04956286dcbfe3d77d3f7c30a39d558914771fe41190fe9e23b079ea6318572
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"U701","status":"ready"},{"id":"X827","status":"ready"},{"id":"J562","status":"closed"},{"id":"K187","status":"closed"},{"id":"V924","status":"closed"}]
Checked answer
{"ids": ["U701", "X827"]}
Seed

wrong_schema_or_value

{
  "ids": ["U701", "X827", "J562", "K187", "V924"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X827","U701"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X827","U701"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["U701","X827"]}
16 tokens
0471 · total · wording 1 · same59e06dcee64d7468dac4f3a8e7f471dc048c0d38ec6978d53dd2f7abb5be6da8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"S119","units":8,"price":4},{"id":"G428","units":10,"price":12},{"id":"D831","units":8,"price":16},{"id":"P487","units":6,"price":9},{"id":"I161","units":13,"price":4}]
Checked answer
{"total": 386}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0472 · sort · wording 1 · same2392d05446c4fa4cd2edea0aaafa08e57b86095cb86d851a8894bd76d09c058a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"C329","priority":7},{"id":"W937","priority":8},{"id":"R583","priority":2},{"id":"X979","priority":6},{"id":"W693","priority":8}]
Checked answer
{"ids": ["W693", "W937"]}
Seed

invalid_json_object

[{"id":"C329","priority":7},{"id":"W937","priority":8},{"id":"R583","priority":2},{"id":"X979","priority":6},{"id":"W693","priority":8}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["C329","W693"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W937","C329"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C329","W693"]}
16 tokens
0473 · lookup · wording 2 · gain42dc8cfb1e791030ceda2684836a7682265c8f2c4ceaa3c0a725d6f7fc710d24
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M543? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"G116","city":"Perth","units":10},{"id":"M543","city":"Riga","units":10},{"id":"C871","city":"Kyoto","units":6},{"id":"V746","city":"Oslo","units":2},{"id":"B577","city":"Suva","units":15}]
Checked answer
{"city": "Riga", "units": 10}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":10}
12 tokens
0474 · filter · wording 2 · sameb8653e349ec09e01014d2481ed9cd00acbe0a116d75720e8d4acc67bb23a2690
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"U836","status":"closed"},{"id":"F268","status":"pending"},{"id":"U538","status":"pending"},{"id":"L469","status":"ready"},{"id":"M598","status":"pending"}]
Checked answer
{"ids": ["L469"]}
Seed

invalid_json_object

[{"id":"L469","status":"ready"},{"id":"M598","status":"pending"}]
27 tokens
1 GPU · AdamW

invalid_json_object

["L469"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["L469"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["L469"]]
11 tokens
0475 · total · wording 2 · same943382caa2af1ff7e73313c2cf7e3ddda501d2204555730d37cf756ac5a309de
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"W256","units":0,"price":8},{"id":"C508","units":10,"price":4},{"id":"V180","units":16,"price":19},{"id":"H717","units":17,"price":2},{"id":"O946","units":2,"price":8}]
Checked answer
{"total": 394}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0476 · sort · wording 2 · sameb94f0639efbdec20a1f55cf41e3993fac95a4dacb9ce77138c31369f66788c51
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J905","priority":9},{"id":"K897","priority":6},{"id":"O537","priority":3},{"id":"W885","priority":4},{"id":"Y587","priority":5}]
Checked answer
{"ids": ["J905", "K897"]}
Seed

invalid_json_object

{"ids": ["J905", "O537", "W885", "Y587", "K897", "O537", "W885", "Y587", "K897", "O537", "W885", "Y587", "K897", "O537", "W885", "Y5
96 tokens
1 GPU · AdamW

Correct

{"ids":["J905","K897"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J905","K897"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J905","O537"]}
16 tokens
0477 · lookup · wording 3 · same7d0681d52eb7048a53e542cd8536a6d68d47daf1b06176ea3f71d2a7f2b43a5a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=F848. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"O560","city":"Bern","units":1},{"id":"J595","city":"Oslo","units":7},{"id":"F848","city":"Kyoto","units":6},{"id":"V257","city":"Kyoto","units":10},{"id":"D573","city":"Riga","units":16}]
Checked answer
{"city": "Kyoto", "units": 6}
Seed

Correct

{"city":"Kyoto","units":6}
12 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":6}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":6}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":6}
12 tokens
0478 · filter · wording 3 · lossbbb8215ffbde408cce861b25866316f6b2bbd109d25fed99808750338c62b4fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"L292","status":"ready"},{"id":"S729","status":"ready"},{"id":"T428","status":"ready"},{"id":"B325","status":"ready"},{"id":"X263","status":"ready"}]
Checked answer
{"ids": ["B325", "L292", "S729", "T428", "X263"]}
Seed

Correct

{"ids": ["B325", "L292", "S729", "T428", "X263"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["B325","C263","L292","S729","T428"]}
31 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["B325","C263","D428","E292","L292","S729","T428"]}
41 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["L292","S729","B325","X263"]}
26 tokens
0479 · total · wording 3 · same3fa5064d101f897a5fb691a6fd7164b8627b7dced95cc97bc0b68b085e4e2bd3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"A840","units":6,"price":15},{"id":"F750","units":18,"price":17},{"id":"A341","units":7,"price":17},{"id":"U867","units":6,"price":7},{"id":"I725","units":3,"price":12}]
Checked answer
{"total": 593}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0480 · sort · wording 3 · same46951b609b9cb271a9adea11e028359ec4ec340b6e50852409cd2533d5ff25c5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"O398","priority":6},{"id":"B692","priority":1},{"id":"P642","priority":4},{"id":"B697","priority":6},{"id":"U173","priority":2}]
Checked answer
{"ids": ["B697", "O398"]}
Seed

invalid_json_object

[{"id":"B697","priority":6},{"id":"U173","priority":2},{"id":"B692","priority":1},{"id":"O398","priority":6},{"id":"P642","priority":4}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["O398","B697"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["O398","B697"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["B692","O398"]}
16 tokens
0481 · lookup · wording 0 · gain30e689cfa042987193fb037af7bcb08bed683debb12b92665ca2180250dac53f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"B609","city":"Suva","units":8},{"id":"U982","city":"Riga","units":11},{"id":"R450","city":"Kyoto","units":14},{"id":"L439","city":"Perth","units":1},{"id":"N388","city":"Lima","units":7}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Suva","units":8},{"city":"Riga","units":11},{"city":"Kyoto","units":14},{"city":"Perth","units":1},{"city":"Lima","units":7}]
55 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0482 · filter · wording 0 · gain8399b5e8e3ded33a885fcf5059fa8bcc32ab1843b3474e8dba6d35df9309684c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L340","status":"pending"},{"id":"M545","status":"ready"},{"id":"N735","status":"closed"},{"id":"J766","status":"closed"},{"id":"X754","status":"pending"}]
Checked answer
{"ids": ["M545"]}
Seed

wrong_schema_or_value

{"ids": ["M545", "J766", "N735", "L340"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["M545"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M545"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M545"]}
11 tokens
0483 · total · wording 0 · same4d94bc35ec3623b0ae551d94016445e1c6dcf4af89ada5636ff7fee56fa7ac85
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Q852","units":15,"price":12},{"id":"F422","units":11,"price":4},{"id":"Y359","units":8,"price":4},{"id":"M276","units":4,"price":10},{"id":"Q567","units":5,"price":17}]
Checked answer
{"total": 381}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0484 · sort · wording 0 · gaindc869eabeba6b069bc193c98c1466378b9b7c3d781963ce6c6c27f2a2492f2fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"W542","priority":7},{"id":"F206","priority":1},{"id":"Q111","priority":1},{"id":"A217","priority":2},{"id":"M477","priority":7}]
Checked answer
{"ids": ["M477", "W542"]}
Seed

wrong_schema_or_value

{"ids": ["F206", "W542", "M477", "A217", "Q111"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M477","W542"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M477","W542"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M477","W542"]}
16 tokens
0485 · lookup · wording 1 · gainff258bd299f3d69bf5e02223128c43e34496a4b0c97db315d007eec12b595c5b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id I848 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"U430","city":"Riga","units":11},{"id":"B813","city":"Suva","units":13},{"id":"I759","city":"Oslo","units":4},{"id":"I848","city":"Kyoto","units":9},{"id":"F575","city":"Suva","units":0}]
Checked answer
{"city": "Kyoto", "units": 9}
Seed

wrong_schema_or_value

{"city":"Riga","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":9}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":9}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":9}
12 tokens
0486 · filter · wording 1 · gainb6cc566c6b731f15a674fc758a1de6f8611779d5667259bbab160205cede9351
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M673","status":"ready"},{"id":"P305","status":"closed"},{"id":"G899","status":"closed"},{"id":"C675","status":"pending"},{"id":"R532","status":"ready"}]
Checked answer
{"ids": ["M673", "R532"]}
Seed

wrong_schema_or_value

{
  "ids": ["C675", "G899", "M673", "P305", "R532"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["M673","R532"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M673","R532"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M673","R532"]}
16 tokens
0487 · total · wording 1 · sameb55190028db42859acad976c0473d235b85fdb804cf06f8bbcd8a1295cdb310d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"U417","units":9,"price":18},{"id":"W239","units":14,"price":1},{"id":"W308","units":3,"price":12},{"id":"L755","units":1,"price":7},{"id":"E761","units":0,"price":1}]
Checked answer
{"total": 219}
Seed

wrong_schema_or_value

{"total": 33}
8 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0488 · sort · wording 1 · gain9b96ec5c69b2c98d50c7c08d1425f6eab53671665ca27af4a7ac87367c1b2348
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"L725","priority":9},{"id":"C490","priority":6},{"id":"M510","priority":6},{"id":"K673","priority":9},{"id":"U930","priority":2}]
Checked answer
{"ids": ["K673", "L725"]}
Seed

invalid_json_object

[{"id":"L725","priority":9},{"id":"U930","priority":2}]
26 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["L725","K673"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L725","K673"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K673","L725"]}
16 tokens
0489 · lookup · wording 2 · gain28e92becec2321e21a2f8d37e664eec4f8a2a61a5cad15ad9cc7aaa9ab2e6e98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to B637? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"L920","city":"Suva","units":16},{"id":"B637","city":"Suva","units":8},{"id":"L960","city":"Kyoto","units":17},{"id":"D554","city":"Oslo","units":18},{"id":"W820","city":"Perth","units":19}]
Checked answer
{"city": "Suva", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":8}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":8}
11 tokens
0490 · filter · wording 2 · same99a91e7e256c72cfb06cc856f4ad06869c978a0989d52210fcbc0488fb5623a9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"U175","status":"ready"},{"id":"Y918","status":"closed"},{"id":"H722","status":"closed"},{"id":"L989","status":"closed"},{"id":"M395","status":"closed"}]
Checked answer
{"ids": ["U175"]}
Seed

invalid_json_object

[{"id":"U175"},{"id":"Y918"},{"id":"H722"},{"id":"L989"},{"id":"M395"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["U175"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["U175"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["U175"]]
11 tokens
0491 · total · wording 2 · samecbce88a8fc8c9649e768f9487927034ecd7d3cc1b0bf0d1ceb4cec34a0d505ab
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"C506","units":0,"price":5},{"id":"E333","units":3,"price":2},{"id":"X365","units":11,"price":8},{"id":"K581","units":16,"price":11},{"id":"Y505","units":11,"price":14}]
Checked answer
{"total": 424}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":388}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0492 · sort · wording 2 · samefcf36a1c6f055965ee09e83edd68e8baa6657682d13f559c4c1b64e3623f7919
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Q359","priority":6},{"id":"R927","priority":5},{"id":"T501","priority":4},{"id":"G244","priority":6},{"id":"U908","priority":6}]
Checked answer
{"ids": ["G244", "Q359"]}
Seed

wrong_schema_or_value

{"ids": ["U908", "G244", "T501", "R927", "Q359"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G244","Q359"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G244","Q359"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q359","U908"]}
16 tokens
0493 · lookup · wording 3 · same2af76bcfc9f6afb2a3d515cc3babebe23a33e2c526869a00d94e65293139343d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=E555. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Z602","city":"Oslo","units":17},{"id":"O913","city":"Perth","units":1},{"id":"H735","city":"Suva","units":17},{"id":"L304","city":"Suva","units":14},{"id":"E555","city":"Lima","units":5}]
Checked answer
{"city": "Lima", "units": 5}
Seed

Correct

{"city":"Lima","units":5}
11 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":5}
11 tokens
0494 · filter · wording 3 · gaina778844320bde852ed644f47d01f28080a86ced670bae6e096eb8c967cb82721
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"B833","status":"pending"},{"id":"S237","status":"pending"},{"id":"P767","status":"ready"},{"id":"N199","status":"closed"},{"id":"V732","status":"closed"}]
Checked answer
{"ids": ["P767"]}
Seed

wrong_schema_or_value

{"ids": ["B833", "S237", "P767", "N199", "V732"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P767"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["P767"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["P767"]}
11 tokens
0495 · total · wording 3 · samef02fdc4e91b1a66e4dbad42cd4c43d4c2dc5ee3844d8eef4bf7a08a85db534c1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"O278","units":18,"price":11},{"id":"Y100","units":18,"price":3},{"id":"M532","units":18,"price":11},{"id":"A617","units":6,"price":2},{"id":"W817","units":18,"price":5}]
Checked answer
{"total": 552}
Seed

invalid_json_object

{
  "total": 11 + 3 + 11 + 2 + 5 = 32
}
28 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0496 · sort · wording 3 · samef156c399de644bef5b492da5c5feda9cf3e7a46bcea24cf17c2ead8bbbea4b4d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J601","priority":1},{"id":"D727","priority":9},{"id":"C433","priority":2},{"id":"V546","priority":6},{"id":"P266","priority":8}]
Checked answer
{"ids": ["D727", "P266"]}
Seed

invalid_json_object

[{"id":"J601"},{"id":"D727"},{"id":"C433"},{"id":"V546"},{"id":"P266"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["D727","P266"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D727","P266"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D727","J601"]}
16 tokens
0497 · lookup · wording 0 · gain69df336ebc7eb2f203148c4349d25bb4e3a9ed657882e73b9fa7a03e6c283b98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record I868. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"N957","city":"Kyoto","units":14},{"id":"L926","city":"Accra","units":19},{"id":"V322","city":"Kyoto","units":6},{"id":"X278","city":"Oslo","units":8},{"id":"I868","city":"Perth","units":15}]
Checked answer
{"city": "Perth", "units": 15}
Seed

invalid_json_object

[{"city":"Perth","units":15}]
13 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":15}
12 tokens
0498 · filter · wording 0 · gainc860abfe137cad0724dd7c9e7a5c6184f05fcc25055613087bf691165b37e430
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Z489","status":"closed"},{"id":"W356","status":"closed"},{"id":"U440","status":"pending"},{"id":"F246","status":"pending"},{"id":"C160","status":"ready"}]
Checked answer
{"ids": ["C160"]}
Seed

wrong_schema_or_value

{"ids": ["C160", "F246", "U440", "Z489", "W356"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C160"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["C160"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["C160"]}
11 tokens
0499 · total · wording 0 · samecdcaa2c5d8096229083bec5e2b1cdd66cc84bb98a7978aab2c316e43df2030f9
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"L570","units":1,"price":18},{"id":"G999","units":17,"price":17},{"id":"D963","units":5,"price":6},{"id":"Q722","units":19,"price":9},{"id":"H495","units":19,"price":18}]
Checked answer
{"total": 850}
Seed

invalid_json_object

{"total": 18 + 17 + 6 + 9 + 18}
22 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":548}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0500 · sort · wording 0 · same16492f6bbb45d44b95fcc835e8185d605e8b720ff21bcfbc5a3eddd46aaf0018
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"F443","priority":8},{"id":"T191","priority":1},{"id":"H712","priority":6},{"id":"H131","priority":6},{"id":"Z714","priority":2}]
Checked answer
{"ids": ["F443", "H131"]}
Seed

wrong_schema_or_value

{"ids": ["F443", "T191", "H712", "H131", "Z714"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F443","H131"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["F443","H712"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["F443","H712"]}
16 tokens
0501 · lookup · wording 1 · gain1ce64a412833c3571616c97c7b9505ab0cd46a1f0a8df63e092e20becc52879e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"X483","city":"Bern","units":5},{"id":"O307","city":"Bern","units":6},{"id":"N626","city":"Bern","units":6},{"id":"P818","city":"Perth","units":6},{"id":"E440","city":"Suva","units":17}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Bern","units":5}
10 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0502 · filter · wording 1 · samece23b26146b0038b0940bfbe8229e8218d16b5e25d0cee05a53dcb36c9343754
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T233","status":"pending"},{"id":"K306","status":"ready"},{"id":"N661","status":"ready"},{"id":"H891","status":"closed"},{"id":"C637","status":"ready"}]
Checked answer
{"ids": ["C637", "K306", "N661"]}
Seed

wrong_schema_or_value

{
  "ids": ["C637", "H891", "K306", "N661", "T233"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["C637","K306","N661"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["C637","K306","N661"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K306","C637","N661"]}
21 tokens
0503 · total · wording 1 · same4e3af1b637514017bafdcb073d68a3de7e1d8797255a4d86982fa10d724551de
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L907","units":1,"price":7},{"id":"L157","units":17,"price":18},{"id":"B441","units":7,"price":13},{"id":"G593","units":14,"price":15},{"id":"A613","units":17,"price":2}]
Checked answer
{"total": 648}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0504 · sort · wording 1 · gaine7423f85ab8f7a985dd64906f82d15d8ceded839e62b8a5d2e96d0b711d64bc5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"U779","priority":3},{"id":"O557","priority":3},{"id":"Q376","priority":2},{"id":"B625","priority":6},{"id":"O843","priority":2}]
Checked answer
{"ids": ["B625", "O557"]}
Seed

invalid_json_object

[{"id":"O843","priority":2},{"id":"O557","priority":3},{"id":"Q376","priority":2},{"id":"U779","priority":3},{"id":"B625","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["B625","O557"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B625","O557"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B625","O557"]}
16 tokens
0505 · lookup · wording 2 · gain02b12f13ecbf8bfda108b76782f671657b4b913f649b748dcd1870c9e1e854b9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Y105? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"S355","city":"Accra","units":6},{"id":"F861","city":"Bern","units":3},{"id":"G445","city":"Accra","units":14},{"id":"Y105","city":"Suva","units":4},{"id":"D521","city":"Accra","units":13}]
Checked answer
{"city": "Suva", "units": 4}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":4}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":4}
11 tokens
0506 · filter · wording 2 · same4d0e55e8047068047090d218811b6f86feede7777d04653fcdd1cdacf6ccfeda
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"P862","status":"ready"},{"id":"C536","status":"closed"},{"id":"A526","status":"ready"},{"id":"O723","status":"ready"},{"id":"X748","status":"ready"}]
Checked answer
{"ids": ["A526", "O723", "P862", "X748"]}
Seed

invalid_json_object

[{"id":"P862","status":"ready"},{"id":"C536","status":"closed"},{"id":"A526","status":"ready"},{"id":"O723","status":"ready"},{"id":"X748","status":"ready"}]
63 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["A526","O723","X748"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["A526","O723","P862","X748"]}
26 tokens
4 GPUs · local windows

invalid_json_object

["ids":["A526","O723","X748"]]
21 tokens
0507 · total · wording 2 · same06c5aeb4a4584a77b8409edf251f341356cf67712ea2108c9461d11336f10db8
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Z342","units":4,"price":5},{"id":"G555","units":4,"price":19},{"id":"L209","units":4,"price":16},{"id":"E979","units":7,"price":16},{"id":"N295","units":11,"price":2}]
Checked answer
{"total": 294}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0508 · sort · wording 2 · same05a77003e53c2eee95599a6d8f47cf8ba1d59746b32a0bb76a09f23dfd35e5e9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"T400","priority":3},{"id":"X436","priority":2},{"id":"Q756","priority":5},{"id":"M666","priority":5},{"id":"M885","priority":3}]
Checked answer
{"ids": ["M666", "Q756"]}
Seed

wrong_schema_or_value

{"ids": ["T400", "X436", "Q756", "M666", "M885"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M666","Q756"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M666","Q756"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q756","M666"]}
16 tokens
0509 · lookup · wording 3 · samef4f37d5fd6251573dbfb83036477e43db6409b668310325b1842819ae8171afa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=A289. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"L344","city":"Perth","units":6},{"id":"B391","city":"Accra","units":16},{"id":"J804","city":"Riga","units":9},{"id":"H444","city":"Lima","units":4},{"id":"A289","city":"Riga","units":13}]
Checked answer
{"city": "Riga", "units": 13}
Seed

Correct

{"city":"Riga","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":13}
12 tokens
0510 · filter · wording 3 · same8dfdb2b674b8985ca74aabbc9661db2ddbae0d775550486675b0c4bb9a8f256f
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"R473","status":"pending"},{"id":"V480","status":"ready"},{"id":"L757","status":"closed"},{"id":"L778","status":"closed"},{"id":"Z316","status":"ready"}]
Checked answer
{"ids": ["V480", "Z316"]}
Seed

wrong_schema_or_value

{"ids": ["L757", "L778", "R473", "V480", "Z316"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Z316","V480"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Z316","V480"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z316","V480"]}
16 tokens
0511 · total · wording 3 · same408d7f492b50c334c668b7ec8a5b85f5d7c793f8ad517f5434ce10cce6ceb442
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"V957","units":13,"price":17},{"id":"H804","units":18,"price":17},{"id":"A733","units":5,"price":19},{"id":"G656","units":19,"price":7},{"id":"N873","units":4,"price":8}]
Checked answer
{"total": 787}
Seed

invalid_json_object

{
  "total": 17 + 17 + 19 + 7 + 8 = 69
}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0512 · sort · wording 3 · gain2143632c63ac7f203cf84ad3b721c3ff4bcc8a6a6bb80062ba4e65b06a997a4e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"M997","priority":2},{"id":"M788","priority":2},{"id":"K757","priority":2},{"id":"W612","priority":5},{"id":"A738","priority":5}]
Checked answer
{"ids": ["A738", "W612"]}
Seed

invalid_json_object

[{"id":"M997"},{"id":"M788"},{"id":"K757"},{"id":"W612"},{"id":"A738"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["A738","W612"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A738","W612"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A738","W612"]}
16 tokens
0513 · lookup · wording 0 · gainb4d22f48ad0cac4d6ffb811b62dadc5e0343e975d14c28e811ca6786e1fc5da5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Y827. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"M253","city":"Oslo","units":8},{"id":"X440","city":"Suva","units":12},{"id":"Y827","city":"Perth","units":5},{"id":"E131","city":"Kyoto","units":4},{"id":"A605","city":"Bern","units":8}]
Checked answer
{"city": "Perth", "units": 5}
Seed

invalid_json_object

[{"city":"Perth","units":5}]
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":5}
11 tokens
0514 · filter · wording 0 · gainc1cdde9ed120e2670dfea73419b899b227387063607da252d21f94f0f049ad78
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"R771","status":"ready"},{"id":"N536","status":"pending"},{"id":"Z420","status":"ready"},{"id":"Z484","status":"closed"},{"id":"Q279","status":"pending"}]
Checked answer
{"ids": ["R771", "Z420"]}
Seed

wrong_schema_or_value

{"ids": ["R771", "Z420", "Z484", "Q279"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["R771","Z420"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["R771","Z420"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["R771","Z420"]}
16 tokens
0515 · total · wording 0 · same888d3c7d048577982653b506e49ecd7405230479a025a96367d7263f882e5184
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"B169","units":16,"price":3},{"id":"Z173","units":6,"price":9},{"id":"Z884","units":10,"price":11},{"id":"I422","units":6,"price":14},{"id":"Y798","units":0,"price":14}]
Checked answer
{"total": 296}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0516 · sort · wording 0 · gain7177553d3d7969a37beb211e29ac72a64965de450d60dde33f0d7d399b5e0777
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"M244","priority":4},{"id":"K197","priority":6},{"id":"S745","priority":2},{"id":"I929","priority":8},{"id":"R999","priority":2}]
Checked answer
{"ids": ["I929", "K197"]}
Seed

wrong_schema_or_value

{"ids": ["M244", "S745"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["I929","K197"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I929","K197"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I929","K197"]}
16 tokens
0517 · lookup · wording 1 · same3efbbdb8eba9ab4277f35179bfa37d4d4f864d28cc909d6cbd4f87173e6736aa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id V454 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"S150","city":"Bern","units":11},{"id":"R801","city":"Suva","units":15},{"id":"V454","city":"Suva","units":11},{"id":"C216","city":"Perth","units":0},{"id":"T333","city":"Riga","units":17}]
Checked answer
{"city": "Suva", "units": 11}
Seed

Correct

{"city":"Suva","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":11}
12 tokens
0518 · filter · wording 1 · loss0eb2840d112a55ca050f749e8a53535e7085aa59e385137c6d5191e766f5b503
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"V206","status":"ready"},{"id":"W301","status":"closed"},{"id":"J272","status":"ready"},{"id":"M102","status":"pending"},{"id":"J699","status":"ready"}]
Checked answer
{"ids": ["J272", "J699", "V206"]}
Seed

Correct

{
  "ids": ["J272", "J699", "V206"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":["J272","J699","V206"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["J272","J699","V206"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["V206","J699","J272"]}
21 tokens
0519 · total · wording 1 · same612c09559b4de14d0848d556752e136a5eddfa6137ba8ed553ee9379e2f0a644
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"E152","units":2,"price":1},{"id":"B848","units":1,"price":6},{"id":"T666","units":14,"price":10},{"id":"D610","units":8,"price":15},{"id":"U524","units":8,"price":12}]
Checked answer
{"total": 364}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0520 · sort · wording 1 · gainac82ff34a6fad8ec795e548af34c6ab9d9b3f59a10d0b6a99d70eec42286fe45
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"E181","priority":2},{"id":"R345","priority":1},{"id":"F913","priority":1},{"id":"W368","priority":3},{"id":"C328","priority":4}]
Checked answer
{"ids": ["C328", "W368"]}
Seed

invalid_json_object

[{"id":"E181","priority":2},{"id":"R345","priority":1},{"id":"F913","priority":1},{"id":"W368","priority":3},{"id":"C328","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["C328","W368"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C328","W368"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C328","W368"]}
16 tokens
0521 · lookup · wording 2 · samea5a1f8e1323d357f2ba24fd97744f1da1f69ab0e28550bce4719eda9e94bf60a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Y128","city":"Riga","units":15},{"id":"G389","city":"Perth","units":4},{"id":"A862","city":"Oslo","units":10},{"id":"Z463","city":"Lima","units":6},{"id":"K464","city":"Lima","units":1}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0522 · filter · wording 2 · sameef470f4609b39b00117e7c06c5cb263ef4034d0e800f70eb80d37b04fa7fcd04
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"R920","status":"pending"},{"id":"X995","status":"closed"},{"id":"P739","status":"ready"},{"id":"I394","status":"closed"},{"id":"F124","status":"pending"}]
Checked answer
{"ids": ["P739"]}
Seed

invalid_json_object

[{"id":"P739"}]
11 tokens
1 GPU · AdamW

invalid_json_object

["P739"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["P739"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["P739"]]
11 tokens
0523 · total · wording 2 · same650e7250ad319d72a8c4f48c469f796e0ca8597eae87c89304138b00cd7bce1b
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"M738","units":0,"price":9},{"id":"E494","units":0,"price":18},{"id":"Z990","units":6,"price":5},{"id":"Y155","units":19,"price":18},{"id":"A414","units":2,"price":3}]
Checked answer
{"total": 378}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0524 · sort · wording 2 · samed7dd347f0eaba95044d91d16b29e77bc6658ce310aa49a624f65fe5a45e93e47
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"O880","priority":3},{"id":"Q931","priority":5},{"id":"U200","priority":2},{"id":"C115","priority":5},{"id":"I146","priority":3}]
Checked answer
{"ids": ["C115", "Q931"]}
Seed

wrong_schema_or_value

{"ids": ["U200", "C115", "I146", "O880", "Q931"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C115","Q931"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C115","Q931"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q931","C115"]}
16 tokens
0525 · lookup · wording 3 · same954320919b3204711b6b6bf48e1ee38832937e7a0816961a9363270a26f9ae73
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=B808. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E984","city":"Bern","units":17},{"id":"B808","city":"Suva","units":9},{"id":"M979","city":"Accra","units":14},{"id":"S842","city":"Accra","units":19},{"id":"T168","city":"Perth","units":17}]
Checked answer
{"city": "Suva", "units": 9}
Seed

Correct

{"city": "Suva", "units": 9}
14 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":9}
11 tokens
0526 · filter · wording 3 · gain9e4617ce8c84445be0758c7ce541fa282b274f19bab240094b06216f8f1dbdc7
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D767","status":"ready"},{"id":"B564","status":"closed"},{"id":"J141","status":"pending"},{"id":"X985","status":"closed"},{"id":"P737","status":"pending"}]
Checked answer
{"ids": ["D767"]}
Seed

wrong_schema_or_value

{"ids": ["B564", "D767", "J141", "P737", "X985"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D767"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["D767"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["D767"]}
11 tokens
0527 · total · wording 3 · same53551e6b1e788bce60ead2c28856478eccd6bf7aa0bf55ed6db1f4fe8be6770b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"X736","units":8,"price":14},{"id":"N214","units":12,"price":11},{"id":"H295","units":1,"price":5},{"id":"K870","units":14,"price":10},{"id":"N519","units":17,"price":14}]
Checked answer
{"total": 627}
Seed

invalid_json_object

{
  "total": 14 + 11 + 5 + 10 + 14 + 14 + 17 + 17 = 89
}
42 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0528 · sort · wording 3 · gainda91a598eb91df29f199d177d9dae4f4f0f2882f7b98967dcd2715f9adfdc7e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"W682","priority":4},{"id":"M261","priority":4},{"id":"D575","priority":2},{"id":"H783","priority":7},{"id":"J236","priority":2}]
Checked answer
{"ids": ["H783", "M261"]}
Seed

invalid_json_object

[{"id":"W682"},{"id":"M261"},{"id":"D575"},{"id":"H783"},{"id":"J236"}]
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H783","J236"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H783","W682"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H783","M261"]}
16 tokens
0529 · lookup · wording 0 · same2d36289270b92e342a060a8a2326d57b82449cfec85bc4e754f4e23e17e38505
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J273. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"O400","city":"Suva","units":13},{"id":"C335","city":"Kyoto","units":13},{"id":"C272","city":"Kyoto","units":14},{"id":"J273","city":"Perth","units":10},{"id":"M898","city":"Bern","units":7}]
Checked answer
{"city": "Perth", "units": 10}
Seed

Correct

{
  "city": "Perth",
  "units": 10
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":10}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":10}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":10}
12 tokens
0530 · filter · wording 0 · sameb692c23e85dd797596b241b0c91736001fe02e1e50eea3765dc9006921d6e046
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"A312","status":"closed"},{"id":"O807","status":"ready"},{"id":"J260","status":"pending"},{"id":"D356","status":"closed"},{"id":"U330","status":"ready"}]
Checked answer
{"ids": ["O807", "U330"]}
Seed

wrong_schema_or_value

{"ids": ["A312", "O807", "J260", "D356", "U330"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["A312","U330","O807"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D356","U330"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O807","A312"]}
16 tokens
0531 · total · wording 0 · samef2b5f0e8c3480ff40761de961f0dd695012a8a9ef19f89e18fb44f00323b3a29
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y196","units":8,"price":19},{"id":"C160","units":1,"price":4},{"id":"S222","units":9,"price":9},{"id":"D580","units":4,"price":13},{"id":"W468","units":0,"price":17}]
Checked answer
{"total": 289}
Seed

invalid_json_object

{"total": 19 + 4 + 9 + 13 + 17 = 64}
26 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0532 · sort · wording 0 · gain10acface79994c8567a7eb298d21b77a59d869511fdc0a4caf43ae6398cdfa18
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z300","priority":2},{"id":"G690","priority":6},{"id":"I934","priority":8},{"id":"N438","priority":9},{"id":"L129","priority":6}]
Checked answer
{"ids": ["N438", "I934"]}
Seed

wrong_schema_or_value

{"ids": ["G690", "I934", "L129", "Z300"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I934","N438"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["I934","N438"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["N438","I934"]}
16 tokens
0533 · lookup · wording 1 · same5eb7ae6b778f335f809ba644de1ce2ab009e024a021dc2ed60a3873815f3a4a2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z638 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W432","city":"Suva","units":2},{"id":"J458","city":"Kyoto","units":17},{"id":"P335","city":"Lima","units":9},{"id":"Z638","city":"Oslo","units":4},{"id":"S293","city":"Bern","units":0}]
Checked answer
{"city": "Oslo", "units": 4}
Seed

Correct

{"city":"Oslo","units":4}
11 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":4}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":4}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":4}
11 tokens
0534 · filter · wording 1 · gain53b54977ff9415ee8513891b0b7c6cb78587b7d8ffe95b624674011ffaa7b203
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T118","status":"ready"},{"id":"Z654","status":"closed"},{"id":"R303","status":"pending"},{"id":"X700","status":"ready"},{"id":"E107","status":"pending"}]
Checked answer
{"ids": ["T118", "X700"]}
Seed

wrong_schema_or_value

{
  "ids": ["R303", "X700", "T118", "Z654"]
}
32 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X700","T118"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E107","T118","X700"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["T118","X700"]}
16 tokens
0535 · total · wording 1 · same619be796bcd9031ec35f40e0f981c2c765125b2c189594b80015606908384fc7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"S737","units":9,"price":6},{"id":"L560","units":13,"price":11},{"id":"Q832","units":15,"price":18},{"id":"I256","units":19,"price":4},{"id":"H727","units":17,"price":6}]
Checked answer
{"total": 645}
Seed

wrong_schema_or_value

{"total": 230}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0536 · sort · wording 1 · gain57b5a476e639aa46592e812047fbc31c6a820272642b858fe610c42698d4c0d8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"M329","priority":7},{"id":"Y570","priority":4},{"id":"U530","priority":3},{"id":"G294","priority":4},{"id":"S971","priority":6}]
Checked answer
{"ids": ["M329", "S971"]}
Seed

invalid_json_object

[{"id":"M329","priority":7},{"id":"U530","priority":3},{"id":"G294","priority":4},{"id":"S971","priority":6}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["M329","S971"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M329","S971"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M329","S971"]}
16 tokens
0537 · lookup · wording 2 · gaineb298bfd05f1a6165ad8bc77216c533fd6da1c4f7155f23bfe1018fe42366db9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to P402? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"P402","city":"Suva","units":5},{"id":"B180","city":"Suva","units":5},{"id":"Q489","city":"Lima","units":5},{"id":"W976","city":"Perth","units":11},{"id":"C783","city":"Kyoto","units":1}]
Checked answer
{"city": "Suva", "units": 5}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":5}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":5}
11 tokens
0538 · filter · wording 2 · samef4c00e1b7a6c10876c53770ebe04a53bf73806ba02199447ff7787eac33ff266
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"U948","status":"closed"},{"id":"H579","status":"pending"},{"id":"S910","status":"pending"},{"id":"K479","status":"pending"},{"id":"E118","status":"ready"}]
Checked answer
{"ids": ["E118"]}
Seed

invalid_json_object

[{"id":"E118","status":"ready"}]
15 tokens
1 GPU · AdamW

invalid_json_object

["E118"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["E118"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["E118"]]
11 tokens
0539 · total · wording 2 · same2f5206b1e801cc20f6768a76c8c6cac3489b44e1135e8877badb27bebd0351ad
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Z697","units":0,"price":12},{"id":"Q291","units":15,"price":4},{"id":"O946","units":5,"price":17},{"id":"K476","units":5,"price":5},{"id":"X458","units":0,"price":13}]
Checked answer
{"total": 170}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":378}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0540 · sort · wording 2 · same281e0a2465e2c39b3d3614daef797116f28c36da008251ece35190727f99b53b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"U282","priority":4},{"id":"W225","priority":6},{"id":"U463","priority":8},{"id":"G220","priority":9},{"id":"E208","priority":6}]
Checked answer
{"ids": ["G220", "U463"]}
Seed

wrong_schema_or_value

{"ids": ["U282", "W225", "U463", "G220", "E208"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["G220","E208"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["G220","E208"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G220","E208"]}
16 tokens
0541 · lookup · wording 3 · gain94ddbef609f8bb0f03c1ffb9e8facd8b82d69b4a5108837ca494fec5a3a93ad9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"S265","city":"Oslo","units":14},{"id":"M669","city":"Lima","units":12},{"id":"R856","city":"Lima","units":5},{"id":"J563","city":"Lima","units":19},{"id":"N529","city":"Perth","units":10}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":14}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0542 · filter · wording 3 · same6d813ca692dfae6ce9e04b093fa92837c6660d83702c033629f5c97ba1b2babd
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"L364","status":"closed"},{"id":"D604","status":"ready"},{"id":"Q478","status":"closed"},{"id":"P506","status":"ready"},{"id":"D480","status":"ready"}]
Checked answer
{"ids": ["D480", "D604", "P506"]}
Seed

invalid_json_object

{"ids": ["D480", "D604", "D480", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D604", "D6
96 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D480","P506","D604"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D480","P506","D604"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D480","P506","D604"]}
21 tokens
0543 · total · wording 3 · sameaa0700994fde8f60ff423daa2387a2eda6ba9b7f0d7ec3f43c1363100219eca9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"R166","units":6,"price":15},{"id":"F662","units":7,"price":6},{"id":"G797","units":11,"price":16},{"id":"J540","units":11,"price":15},{"id":"R239","units":10,"price":10}]
Checked answer
{"total": 573}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":368}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":368}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0544 · sort · wording 3 · gain7b9663b93958170b039a6b242b9b9dc95662fb95963525b69022ce37d0177e14
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"V605","priority":2},{"id":"N395","priority":2},{"id":"V603","priority":3},{"id":"M529","priority":4},{"id":"Z339","priority":1}]
Checked answer
{"ids": ["M529", "V603"]}
Seed

invalid_json_object

[{"id":"V603","priority":3},{"id":"M529","priority":4},{"id":"V605","priority":2},{"id":"N395","priority":2},{"id":"Z339","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["M529","V603"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M529","V603"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M529","V603"]}
16 tokens
0545 · lookup · wording 0 · same16250dacdcf7c6ad0edbb9e6706981f8e2fa62809bf71cf2437126d15e653203
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record U883. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"I449","city":"Oslo","units":12},{"id":"U883","city":"Lima","units":16},{"id":"F982","city":"Accra","units":16},{"id":"H896","city":"Lima","units":9},{"id":"B654","city":"Accra","units":15}]
Checked answer
{"city": "Lima", "units": 16}
Seed

Correct

{
  "city": "Lima",
  "units": 16
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":16}
12 tokens
0546 · filter · wording 0 · gain124a8b113e9dff698d5a22b70ca643ae90132e5a300d0ae80d15bfda4b642e0a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"K810","status":"pending"},{"id":"E638","status":"pending"},{"id":"X157","status":"ready"},{"id":"C422","status":"closed"},{"id":"J570","status":"pending"}]
Checked answer
{"ids": ["X157"]}
Seed

wrong_schema_or_value

{"ids": ["X157", "J570", "K810", "C422"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["X157"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["X157"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["X157"]}
11 tokens
0547 · total · wording 0 · same9e9e99895e62da64e89031082061b6e091d634a8086a0afe88706d6216300bad
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"W710","units":13,"price":11},{"id":"T330","units":13,"price":4},{"id":"E606","units":17,"price":1},{"id":"E246","units":13,"price":4},{"id":"X374","units":13,"price":16}]
Checked answer
{"total": 472}
Seed

invalid_json_object

{"total": 11 + 4 + 1 + 4 + 16 = 36}
25 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":568}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0548 · sort · wording 0 · samec00f1143ad9ccd020f22d63ff7e04d7d97f502857f4eadb5fbd9331120224d82
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"C348","priority":6},{"id":"H962","priority":6},{"id":"L146","priority":9},{"id":"J620","priority":7},{"id":"B904","priority":6}]
Checked answer
{"ids": ["L146", "J620"]}
Seed

wrong_schema_or_value

{"ids": ["C348", "H962", "L146", "J620", "B904"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L146","J620"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L146","J620"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["L146","H962"]}
16 tokens
0549 · lookup · wording 1 · same89522460c75f5c8b417992d08c8acee0fd9d84392fcb3ecfe6b318eb7e37f731
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id E290 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"E290","city":"Kyoto","units":9},{"id":"R439","city":"Oslo","units":16},{"id":"N981","city":"Lima","units":1},{"id":"M412","city":"Suva","units":1},{"id":"B685","city":"Kyoto","units":0}]
Checked answer
{"city": "Kyoto", "units": 9}
Seed

Correct

{"city":"Kyoto","units":9}
12 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":9}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":9}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":9}
12 tokens
0550 · filter · wording 1 · gain31271b19e6935cc5af43f99b70e5e412b9498ce41e88c5b305a0e45e5b77f6aa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"Q291","status":"pending"},{"id":"M682","status":"closed"},{"id":"L455","status":"closed"},{"id":"P131","status":"ready"},{"id":"O562","status":"closed"}]
Checked answer
{"ids": ["P131"]}
Seed

wrong_schema_or_value

{
  "ids": ["P131", "O562", "L455", "M682", "Q291"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["P131"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["P131"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["P131"]}
11 tokens
0551 · total · wording 1 · same6a003586cd64c7b54c2bc4a01897416698de2d130ef7eef1bc6896b74d5dc7aa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D635","units":14,"price":14},{"id":"R560","units":6,"price":5},{"id":"T874","units":5,"price":15},{"id":"M688","units":8,"price":3},{"id":"E988","units":12,"price":13}]
Checked answer
{"total": 481}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0552 · sort · wording 1 · gain795550b5e30ece8d2debbc666129752a2c3e8ae102da8b341df5b829017f5c34
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"I766","priority":6},{"id":"S305","priority":5},{"id":"C566","priority":5},{"id":"P861","priority":9},{"id":"J954","priority":3}]
Checked answer
{"ids": ["P861", "I766"]}
Seed

invalid_json_object

[{"id":"I766","priority":6},{"id":"S305","priority":5},{"id":"C566","priority":5},{"id":"P861","priority":9},{"id":"J954","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["P861","I766"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P861","I766"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P861","I766"]}
16 tokens
0553 · lookup · wording 2 · gainba5dd4e6d50760c84b8cb787e09493e8b712db42ba824eca8936f8161779f47b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to A821? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"A821","city":"Bern","units":1},{"id":"P946","city":"Perth","units":10},{"id":"B835","city":"Lima","units":8},{"id":"G233","city":"Bern","units":7},{"id":"K918","city":"Suva","units":16}]
Checked answer
{"city": "Bern", "units": 1}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":1}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":1}
10 tokens
0554 · filter · wording 2 · same5cea9ecbe773a41a2c92fb2d24e21db6977bdf4c8f47a62b8c12319dd8441b86
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"R939","status":"pending"},{"id":"Z370","status":"pending"},{"id":"I168","status":"pending"},{"id":"Y964","status":"closed"},{"id":"L890","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"Z370","status":"pending"},{"id":"I168","status":"pending"},{"id":"Y964","status":"closed"},{"id":"L890","status":"closed"}]
51 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

invalid_json_object

["ids":[]}
6 tokens
0555 · total · wording 2 · same6ec07c3e66cfd08fc5c76a09c0a3b9ae4419fc162c831c3b99433d4d1e2d1f05
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V434","units":0,"price":17},{"id":"K768","units":9,"price":12},{"id":"L889","units":12,"price":15},{"id":"S828","units":15,"price":4},{"id":"Y688","units":8,"price":10}]
Checked answer
{"total": 428}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0556 · sort · wording 2 · gainbf5c7bdbfeaff7e64c31bfe0fcdff99d78fcf58a72efec6da0f5102bde6bad7d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K296","priority":5},{"id":"U737","priority":9},{"id":"F820","priority":6},{"id":"N162","priority":9},{"id":"H332","priority":3}]
Checked answer
{"ids": ["N162", "U737"]}
Seed

wrong_schema_or_value

{"ids": ["U737", "F820", "N162", "H332", "K296"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["N162","U737"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N162","K296"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["N162","U737"]}
16 tokens
0557 · lookup · wording 3 · gain0bf9be416114af235d0197edd5b1db43c3d931293bba0ebce2f8e89af73d6734
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=H582. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"F320","city":"Oslo","units":15},{"id":"D808","city":"Suva","units":19},{"id":"H582","city":"Lima","units":7},{"id":"N307","city":"Accra","units":5},{"id":"O875","city":"Oslo","units":1}]
Checked answer
{"city": "Lima", "units": 7}
Seed

wrong_schema_or_value

{"city":"Oslo","units":15}
12 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":7}
11 tokens
0558 · filter · wording 3 · sameb1c39840cd9b4582a7e6a8bea4d367547050601220ce3e52963afd3ce62a57c5
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Z274","status":"pending"},{"id":"N463","status":"ready"},{"id":"G482","status":"ready"},{"id":"A426","status":"pending"},{"id":"Z294","status":"pending"}]
Checked answer
{"ids": ["G482", "N463"]}
Seed

wrong_schema_or_value

{"ids": ["A426", "G482", "N463", "Z274", "Z294"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G482","N463"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G482","N463"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N463","G482"]}
16 tokens
0559 · total · wording 3 · same2d4425c1ac43662f13baca018819ea8f0e2bfc8c2ef7b10d62fd7785a9a28485
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"J232","units":3,"price":13},{"id":"Q854","units":10,"price":13},{"id":"O313","units":10,"price":9},{"id":"T321","units":18,"price":3},{"id":"O143","units":5,"price":15}]
Checked answer
{"total": 388}
Seed

invalid_json_object

{
  "total": 13 + 13 + 13 + 13 + 15 + 9 + 3 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 15 + 
96 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0560 · sort · wording 3 · same9d850371135caea74713a2e553fd45db416c6c56060f56b515e666c0705304e8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J252","priority":4},{"id":"G687","priority":7},{"id":"W819","priority":3},{"id":"E283","priority":3},{"id":"T573","priority":6}]
Checked answer
{"ids": ["G687", "T573"]}
Seed

invalid_json_object

[{"id":"J252"},{"id":"G687"},{"id":"W819"},{"id":"E283"},{"id":"T573"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["G687","T573"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G687","T573"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G687","J252"]}
16 tokens
0561 · lookup · wording 0 · gainc944497c4bed1f3ea6cf0796c551ec8689c6c56c16c2fbf93d141687392d998d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"M910","city":"Bern","units":5},{"id":"W658","city":"Perth","units":2},{"id":"D785","city":"Oslo","units":16},{"id":"E972","city":"Lima","units":16},{"id":"G348","city":"Kyoto","units":1}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Bern","units":5},{"city":"Perth","units":2},{"city":"Oslo","units":16},{"city":"Lima","units":16},{"city":"Kyoto","units":1}]
54 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0562 · filter · wording 0 · gain9cbd774f7305ecd91da4955facc2f7641a3a7abf98ac2a05662ff1591b59588c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"I777","status":"ready"},{"id":"N471","status":"ready"},{"id":"R358","status":"pending"},{"id":"N972","status":"ready"},{"id":"E654","status":"ready"}]
Checked answer
{"ids": ["E654", "I777", "N471", "N972"]}
Seed

wrong_schema_or_value

{"ids": ["I777", "N471", "N972", "E654"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I777","N471","N972","E654"]}
26 tokens
4 GPUs · DDP

Correct

{"ids":["E654","I777","N471","N972"]}
26 tokens
4 GPUs · local windows

Correct

{"ids":["E654","I777","N471","N972"]}
26 tokens
0563 · total · wording 0 · samec4d56fb1fd40bbf6910a5d91885350614d5ea0a73b72a20e2094e94880dc1acc
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"J811","units":10,"price":3},{"id":"D795","units":2,"price":9},{"id":"C647","units":8,"price":1},{"id":"B366","units":3,"price":13},{"id":"F149","units":16,"price":2}]
Checked answer
{"total": 127}
Seed

invalid_json_object

{"total": 10 * 3 + 2 * 9 + 8 * 1 + 3 * 13 + 16 * 2} = 30 + 18 + 8 + 39 + 32 = 127
61 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0564 · sort · wording 0 · gain3dfd61e94953a89fcd853c1ca4aa1b8c063540a914b24bfde7de8eb668e56771
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Q507","priority":5},{"id":"J383","priority":2},{"id":"H732","priority":9},{"id":"I345","priority":3},{"id":"J915","priority":2}]
Checked answer
{"ids": ["H732", "Q507"]}
Seed

wrong_schema_or_value

{"ids": ["J383", "J915"]}
17 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H732","I345"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H732","Q507"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H732","Q507"]}
16 tokens
0565 · lookup · wording 1 · gainc1a920663667242ac3417c37ec89862e585870c40fa164ea84e015d6d0a380ab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Y373 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"X786","city":"Bern","units":1},{"id":"Y373","city":"Kyoto","units":19},{"id":"J737","city":"Oslo","units":1},{"id":"B175","city":"Riga","units":16},{"id":"P411","city":"Perth","units":9}]
Checked answer
{"city": "Kyoto", "units": 19}
Seed

wrong_schema_or_value

{"city":"Bern","units":1}
10 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":19}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":19}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":19}
13 tokens
0566 · filter · wording 1 · gain1005138f5b01124102c3fd0ab2ce64516502b3b0940d022fb54f8b795909311b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"O615","status":"pending"},{"id":"W884","status":"ready"},{"id":"Q742","status":"pending"},{"id":"P916","status":"pending"},{"id":"P666","status":"pending"}]
Checked answer
{"ids": ["W884"]}
Seed

wrong_schema_or_value

{
  "ids": ["O615", "W884", "Q742", "P666", "P916"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["W884"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["W884"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["W884"]}
11 tokens
0567 · total · wording 1 · same9b2ad1ca3ae0ac6c24cd6989c3d58eec716f96da36e20312865b682e50202082
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"O437","units":19,"price":13},{"id":"S406","units":5,"price":11},{"id":"Q508","units":4,"price":6},{"id":"M860","units":9,"price":9},{"id":"V205","units":2,"price":17}]
Checked answer
{"total": 441}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0568 · sort · wording 1 · gain6426a54a2e77ff738d7de83cf9c2961d359a98a3c8db6e9f73ae911837d6525a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V178","priority":2},{"id":"Y846","priority":1},{"id":"L904","priority":9},{"id":"V663","priority":4},{"id":"F773","priority":3}]
Checked answer
{"ids": ["L904", "V663"]}
Seed

invalid_json_object

[{"id":"V178","priority":2},{"id":"Y846","priority":1},{"id":"L904","priority":9},{"id":"V663","priority":4},{"id":"F773","priority":3}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["L904","F773"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L904","V663"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L904","V663"]}
16 tokens
0569 · lookup · wording 2 · gaina221140baec2d9d2581c8288b3ae38120431908243a003bec53f68c887a42642
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to C311? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"M626","city":"Kyoto","units":15},{"id":"C311","city":"Kyoto","units":12},{"id":"B630","city":"Perth","units":19},{"id":"U812","city":"Accra","units":17},{"id":"Z832","city":"Accra","units":15}]
Checked answer
{"city": "Kyoto", "units": 12}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":12}
13 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":12}
13 tokens
0570 · filter · wording 2 · gain16b88afa415f7865402756ae36d8df924749e2dcb1f3bbaf20453f84126dfa91
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"P295","status":"closed"},{"id":"A242","status":"closed"},{"id":"M472","status":"pending"},{"id":"Q124","status":"pending"},{"id":"P923","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"A242","status":"closed"},{"id":"M472","status":"pending"},{"id":"P295","status":"closed"},{"id":"P923","status":"closed"},{"id":"Q124","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0571 · total · wording 2 · samebaf33b1712b146a4dca16cb93f142c44fdb2b13f56a4ca2212972680871e0d72
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"C857","units":18,"price":14},{"id":"S255","units":11,"price":7},{"id":"O126","units":15,"price":6},{"id":"Z595","units":2,"price":13},{"id":"A885","units":2,"price":8}]
Checked answer
{"total": 461}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0572 · sort · wording 2 · gain3b7b2df8a64bc45f2f4afaed79435703783562517fc1f5e022daf3868a2cfd6b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"V206","priority":9},{"id":"U705","priority":7},{"id":"J534","priority":3},{"id":"L673","priority":4},{"id":"C764","priority":7}]
Checked answer
{"ids": ["V206", "C764"]}
Seed

wrong_schema_or_value

{"ids": ["U705", "J534", "L673", "C764", "V206"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["V206","C764"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["V206","C764"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["V206","C764"]}
16 tokens
0573 · lookup · wording 3 · same5365c84c418e31ea220868efc968fbe0df733218a721d8cfa3f4444143864acf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=U945. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P741","city":"Perth","units":19},{"id":"U945","city":"Riga","units":16},{"id":"E420","city":"Perth","units":4},{"id":"D730","city":"Oslo","units":8},{"id":"H289","city":"Bern","units":10}]
Checked answer
{"city": "Riga", "units": 16}
Seed

Correct

{"city":"Riga","units":16}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":16}
12 tokens
0574 · filter · wording 3 · gaindc9de5f007445c46faebc30df24ce90b6591454176931b8a623d8ce18ddaf34a
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D608","status":"ready"},{"id":"S951","status":"closed"},{"id":"W237","status":"ready"},{"id":"D333","status":"pending"},{"id":"W195","status":"ready"}]
Checked answer
{"ids": ["D608", "W195", "W237"]}
Seed

wrong_schema_or_value

{"ids": ["D333", "D608", "D333", "D608", "S951", "W237", "W195", "W237"]}
53 tokens
1 GPU · AdamW

Correct

{"ids":["D608","W195","W237"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["D608","W195","W237"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["D608","W195","W237"]}
21 tokens
0575 · total · wording 3 · same31a135f1591f0a08f29b9b8736af787721d8785284ef3cbc83eb7b0eb1aef18e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"H805","units":4,"price":3},{"id":"L342","units":16,"price":12},{"id":"O276","units":18,"price":14},{"id":"H358","units":18,"price":9},{"id":"B822","units":7,"price":15}]
Checked answer
{"total": 723}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0576 · sort · wording 3 · gaina66e67f5835f01701bb6f6b5b997811bd607e7d42d65f1404566b7272e4ae5f4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"Z639","priority":9},{"id":"S275","priority":9},{"id":"Q284","priority":8},{"id":"Q902","priority":5},{"id":"X591","priority":6}]
Checked answer
{"ids": ["S275", "Z639"]}
Seed

invalid_json_object

[{"id":"S275","priority":9},{"id":"Q284","priority":8},{"id":"Q902","priority":5},{"id":"Z639","priority":9},{"id":"X591","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["S275","Z639"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S275","Z639"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S275","Z639"]}
16 tokens
0577 · lookup · wording 0 · same93179b7a93558a944963adc728548101223d74aedaea79dd8f18d585c146195e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record K189. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"M255","city":"Bern","units":4},{"id":"K189","city":"Perth","units":7},{"id":"W994","city":"Lima","units":11},{"id":"J669","city":"Accra","units":10},{"id":"L279","city":"Accra","units":9}]
Checked answer
{"city": "Perth", "units": 7}
Seed

Correct

{
  "city": "Perth",
  "units": 7
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":7}
11 tokens
0578 · filter · wording 0 · gain7f2a9034adcadb3f6046a4f44629fbcbfd192d977a103c2e599dccbd560f1afa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"C586","status":"ready"},{"id":"W381","status":"pending"},{"id":"A554","status":"closed"},{"id":"B905","status":"closed"},{"id":"G658","status":"ready"}]
Checked answer
{"ids": ["C586", "G658"]}
Seed

wrong_schema_or_value

{"ids": ["A554", "B905", "C586", "G658"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["C586","G658"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C586","G658"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C586","G658"]}
16 tokens
0579 · total · wording 0 · same0164ec48d30bb022f923a1912b00326ccbbd3f8ebe347d436887740a7a02ff96
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C333","units":17,"price":13},{"id":"Z232","units":18,"price":19},{"id":"I944","units":13,"price":3},{"id":"G936","units":7,"price":4},{"id":"D466","units":17,"price":18}]
Checked answer
{"total": 936}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0580 · sort · wording 0 · gain797edb8dc01df5e90e9fa1f3d3c74f397873b701abc7bfd19d7250c655c15838
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"T570","priority":7},{"id":"D775","priority":6},{"id":"N986","priority":4},{"id":"Y573","priority":2},{"id":"R671","priority":5}]
Checked answer
{"ids": ["T570", "D775"]}
Seed

wrong_schema_or_value

{"ids": ["Y573", "R671", "N986", "T570", "D775"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["T570","D775"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["T570","D775"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["T570","D775"]}
16 tokens
0581 · lookup · wording 1 · gainf421329247160cfd530517a15702b660013efcce2cf949f836632aac6b48b997
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"P110","city":"Bern","units":11},{"id":"H150","city":"Lima","units":13},{"id":"X185","city":"Riga","units":10},{"id":"I755","city":"Kyoto","units":0},{"id":"C389","city":"Riga","units":11}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Bern","units":11}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0582 · filter · wording 1 · gainfbe07b5b1ee76df76d62f75cb8894a56891943b9b1b88260a599fbe20c7c3c83
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T737","status":"pending"},{"id":"X418","status":"ready"},{"id":"J470","status":"ready"},{"id":"A673","status":"pending"},{"id":"Y713","status":"closed"}]
Checked answer
{"ids": ["J470", "X418"]}
Seed

wrong_schema_or_value

{
  "ids": ["A673", "J470", "X418", "T737", "Y713"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["J470","X418"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J470","X418"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J470","X418"]}
16 tokens
0583 · total · wording 1 · same4328ca07736f16bc0cbeab8d55b9e2ea0232deb6a0bfe14994f19a27dc075cfc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L569","units":16,"price":12},{"id":"T526","units":10,"price":18},{"id":"P663","units":1,"price":1},{"id":"A233","units":12,"price":7},{"id":"P158","units":14,"price":10}]
Checked answer
{"total": 597}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0584 · sort · wording 1 · same2b43a11e3998b2db578debaefaa2d6bb2ebf3ea366a84b0005441a69924cbc46
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"X897","priority":7},{"id":"V423","priority":8},{"id":"S917","priority":6},{"id":"M768","priority":7},{"id":"R297","priority":2}]
Checked answer
{"ids": ["V423", "M768"]}
Seed

invalid_json_object

[{"id":"X897","priority":7},{"id":"V423","priority":8},{"id":"S917","priority":6},{"id":"M768","priority":7},{"id":"R297","priority":2}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["V423","X897"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V423","X897"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["V423","X897"]}
16 tokens
0585 · lookup · wording 2 · gain60ffae7bd54609258472b69bcef0d8eb68415562fcac8cadb664f989b9647324
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to N677? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E509","city":"Lima","units":17},{"id":"N677","city":"Lima","units":19},{"id":"G148","city":"Lima","units":16},{"id":"S350","city":"Riga","units":19},{"id":"B268","city":"Accra","units":8}]
Checked answer
{"city": "Lima", "units": 19}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":19}
12 tokens
0586 · filter · wording 2 · same4675610f6fe11a002017f11401f7899b33f5355ecc2d6c1f58dfa87fc86d1106
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"F284","status":"pending"},{"id":"V975","status":"ready"},{"id":"Z294","status":"pending"},{"id":"I166","status":"pending"},{"id":"V516","status":"pending"}]
Checked answer
{"ids": ["V975"]}
Seed

invalid_json_object

[{"id":"V516","status":"ready"},{"id":"V975","status":"ready"},{"id":"F284","status":"pending"},{"id":"I166","status":"pending"},{"id":"Z294","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["V975"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["V975"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["V516"]]
11 tokens
0587 · total · wording 2 · same7680f090f458654c3f112fa75fb4f43eb3f38bc43536ecaae5ba548878fe8d5a
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"P312","units":2,"price":4},{"id":"P349","units":6,"price":9},{"id":"E650","units":10,"price":19},{"id":"C341","units":4,"price":12},{"id":"D621","units":6,"price":14}]
Checked answer
{"total": 384}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0588 · sort · wording 2 · same4d92d51ebe04fc9c9f34b11ba7708ba992cb5f1159f32ed35b2899490411281e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"E417","priority":1},{"id":"Z934","priority":9},{"id":"W678","priority":5},{"id":"F375","priority":1},{"id":"Q580","priority":3}]
Checked answer
{"ids": ["Z934", "W678"]}
Seed

wrong_schema_or_value

{"ids": ["E417", "Q580"]}
17 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Z934","E417"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Z934","E417"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z934","E417"]}
16 tokens
0589 · lookup · wording 3 · same0238598b04cda32467e7ab70a7bf96f679f52feb91b602b08f0f894c6adaf212
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Y437. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Z908","city":"Bern","units":9},{"id":"J146","city":"Lima","units":3},{"id":"C660","city":"Riga","units":2},{"id":"Q268","city":"Lima","units":6},{"id":"Y437","city":"Perth","units":9}]
Checked answer
{"city": "Perth", "units": 9}
Seed

Correct

{"city": "Perth", "units": 9}
14 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":9}
11 tokens
0590 · filter · wording 3 · gainf4ab8ee8270e65d17e485b411da4761594bd2f390498bd81cc56feb6e005fa17
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"X750","status":"ready"},{"id":"K382","status":"ready"},{"id":"C321","status":"pending"},{"id":"S333","status":"pending"},{"id":"E655","status":"closed"}]
Checked answer
{"ids": ["K382", "X750"]}
Seed

wrong_schema_or_value

{"ids": ["C321", "E655", "K382", "X750", "S333"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["K382","X750"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K382","X750"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K382","X750"]}
16 tokens
0591 · total · wording 3 · same178a8647fb64163473d8e23eda316df2659a7a3fe2703fa79af987522ea368b6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"C379","units":17,"price":5},{"id":"T454","units":6,"price":4},{"id":"T401","units":19,"price":4},{"id":"A691","units":2,"price":10},{"id":"V139","units":10,"price":4}]
Checked answer
{"total": 245}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0592 · sort · wording 3 · samefa5cc16aaca68d8cb005b0a2d196bb0f13cfcdbdf41d781412249bd896196043
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"H485","priority":2},{"id":"G564","priority":4},{"id":"D230","priority":2},{"id":"P817","priority":1},{"id":"M723","priority":3}]
Checked answer
{"ids": ["G564", "M723"]}
Seed

invalid_json_object

[{"id":"H485","priority":2},{"id":"G564","priority":4},{"id":"D230","priority":2},{"id":"P817","priority":1},{"id":"M723","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["G564","M723"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G564","M723"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G564","D230"]}
16 tokens
0593 · lookup · wording 0 · gain8037f251f4a0c991a4957f8f61bd737cbde41a45c50fda11abbfd946d1ebfd72
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z205. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Z205","city":"Perth","units":0},{"id":"X925","city":"Kyoto","units":4},{"id":"R737","city":"Kyoto","units":17},{"id":"S703","city":"Bern","units":14},{"id":"X471","city":"Lima","units":8}]
Checked answer
{"city": "Perth", "units": 0}
Seed

wrong_schema_or_value

{
  "city": "Perth",
  "units": null
}
17 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":0}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":0}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":0}
11 tokens
0594 · filter · wording 0 · gain42589e8a8d733d5b01e9b27bd06478429ba1c4b88a17eb082931630e34f7a690
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Z157","status":"closed"},{"id":"Z509","status":"pending"},{"id":"M260","status":"ready"},{"id":"H898","status":"pending"},{"id":"Y576","status":"pending"}]
Checked answer
{"ids": ["M260"]}
Seed

wrong_schema_or_value

{"ids": ["M260", "Y576", "Z157", "Z509", "H898"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M260"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M260"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M260"]}
11 tokens
0595 · total · wording 0 · same637b7711b435f890ecf957ececb43b78ec8c08895c0a078e8434bb03dc505fd2
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"E499","units":13,"price":3},{"id":"D636","units":16,"price":15},{"id":"K195","units":12,"price":17},{"id":"D904","units":14,"price":16},{"id":"J157","units":16,"price":1}]
Checked answer
{"total": 723}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0596 · sort · wording 0 · samedb425690c9b0d994f631265218634894e12b1e9c140e63ed4095c497cd507f42
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Q656","priority":8},{"id":"S960","priority":9},{"id":"B515","priority":9},{"id":"P452","priority":2},{"id":"F730","priority":7}]
Checked answer
{"ids": ["B515", "S960"]}
Seed

wrong_schema_or_value

{"ids": ["S960", "B515", "P452", "F730", "Q656"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B515","S960"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["S960","B515"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S960","B515"]}
16 tokens
0597 · lookup · wording 1 · gain9da5d778f278d3308a6c413c85046845252680c88de95da613b24fffc7c3b1dc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id F479 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W722","city":"Accra","units":16},{"id":"H284","city":"Kyoto","units":17},{"id":"F479","city":"Bern","units":12},{"id":"S697","city":"Kyoto","units":10},{"id":"N215","city":"Bern","units":19}]
Checked answer
{"city": "Bern", "units": 12}
Seed

wrong_schema_or_value

{"city":"Accra","units":16}
12 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":12}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":12}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":12}
11 tokens
0598 · filter · wording 1 · gainb690dbeb414932935be7bd38f2c79a0e23f29f9b4f5a34603257bfd547497625
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"E194","status":"pending"},{"id":"K335","status":"pending"},{"id":"X999","status":"closed"},{"id":"A308","status":"closed"},{"id":"Q640","status":"ready"}]
Checked answer
{"ids": ["Q640"]}
Seed

wrong_schema_or_value

{
  "ids": ["A308", "E194", "K335", "Q640"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["Q640"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Q640"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Q640"]}
11 tokens
0599 · total · wording 1 · same6979c03d9c24480a6efc38d1dd1bc735e12dbb7fddf9d9e7d10ee170a145a566
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"W535","units":17,"price":5},{"id":"V323","units":9,"price":4},{"id":"C644","units":1,"price":15},{"id":"H274","units":7,"price":8},{"id":"U428","units":15,"price":14}]
Checked answer
{"total": 402}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0600 · sort · wording 1 · gain6c8b79934907c4b3872d7633560e01d53f858eb5d946406f3c70f05396640047
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J342","priority":5},{"id":"J523","priority":2},{"id":"H661","priority":3},{"id":"M799","priority":4},{"id":"G584","priority":2}]
Checked answer
{"ids": ["J342", "M799"]}
Seed

invalid_json_object

[{"id":"J523","priority":2},{"id":"J342","priority":5},{"id":"H661","priority":3},{"id":"M799","priority":4},{"id":"G584","priority":2}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J342","M799"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J342","M799"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J342","M799"]}
16 tokens
0601 · lookup · wording 2 · same13a828e691dd7b14c69f2cc5de8b7172a8285ae15125db32f6ff8dba3f4ce9c8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Q891","city":"Riga","units":11},{"id":"J561","city":"Oslo","units":1},{"id":"L315","city":"Accra","units":18},{"id":"G966","city":"Perth","units":4},{"id":"K194","city":"Riga","units":12}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0602 · filter · wording 2 · samef5088dc8eae738381b4dfe3c01f27a05bb0a171bb8fd42001c17b81258dcacd1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"D885","status":"closed"},{"id":"O119","status":"closed"},{"id":"J909","status":"pending"},{"id":"K491","status":"ready"},{"id":"R115","status":"ready"}]
Checked answer
{"ids": ["K491", "R115"]}
Seed

invalid_json_object

[{"id":"K491"},{"id":"R115"}]
19 tokens
1 GPU · AdamW

invalid_json_object

["K491","R115"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["K491","R115"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["K491","R115"]]
16 tokens
0603 · total · wording 2 · same5e65c39bd3bb925f1d3837cdbd23802b81fea393c64594b6c95a881a88057d30
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"H852","units":17,"price":3},{"id":"Q851","units":14,"price":15},{"id":"O810","units":8,"price":5},{"id":"S557","units":2,"price":1},{"id":"H840","units":1,"price":9}]
Checked answer
{"total": 312}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0604 · sort · wording 2 · same19489b5df4d0851975df32a33bfc20eab0f3726080010de709fa4573f8dfb67c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"U181","priority":7},{"id":"O337","priority":4},{"id":"P597","priority":4},{"id":"U946","priority":8},{"id":"Q528","priority":1}]
Checked answer
{"ids": ["U946", "U181"]}
Seed

wrong_schema_or_value

{"ids": ["U181", "O337", "P597", "U946", "Q528"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["U181","U946"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["U181","U946"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["U181","O337"]}
16 tokens
0605 · lookup · wording 3 · gaina2b0c0391c00fed0106ff470007602b8d349f547f3569569279d97d8dade9be4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=X245. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"A885","city":"Riga","units":10},{"id":"G712","city":"Perth","units":17},{"id":"X245","city":"Accra","units":7},{"id":"R669","city":"Lima","units":6},{"id":"T894","city":"Suva","units":18}]
Checked answer
{"city": "Accra", "units": 7}
Seed

wrong_schema_or_value

{"city": "Riga", "units": 10}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":7}
11 tokens
0606 · filter · wording 3 · gaine0708f77603220e8dcf27ecd5651538252c72ca576adff29f30158dc05e7c4a5
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"E160","status":"closed"},{"id":"A448","status":"ready"},{"id":"M847","status":"closed"},{"id":"Q274","status":"closed"},{"id":"Q531","status":"pending"}]
Checked answer
{"ids": ["A448"]}
Seed

wrong_schema_or_value

{"ids": ["A448", "E160", "M847", "Q274", "Q531"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A448"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["A448"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["A448"]}
11 tokens
0607 · total · wording 3 · samea014bd5661da4eadeba127bdebd933587844cf37af5461b3ff009f58f1b74a3c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"X109","units":14,"price":9},{"id":"D113","units":8,"price":16},{"id":"N846","units":15,"price":2},{"id":"T160","units":5,"price":9},{"id":"S140","units":15,"price":6}]
Checked answer
{"total": 419}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0608 · sort · wording 3 · same3b5b9b3c9853aae03fd4a59961e2f0598199a149f7c50ed5c8bc77d41dbed18f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"A711","priority":4},{"id":"Z227","priority":1},{"id":"P160","priority":6},{"id":"U704","priority":2},{"id":"U982","priority":6}]
Checked answer
{"ids": ["P160", "U982"]}
Seed

invalid_json_object

[{"id":"A711","priority":4},{"id":"U704","priority":2},{"id":"U982","priority":6},{"id":"P160","priority":6},{"id":"Z227","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["P160","U982"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P160","A711"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["A711","P160"]}
16 tokens
0609 · lookup · wording 0 · same376ae4d0df319208144152fa37d00c5169ae5e3aec4cc31b46c22a968b92dac3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record F652. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"F652","city":"Kyoto","units":2},{"id":"W207","city":"Bern","units":1},{"id":"I174","city":"Bern","units":15},{"id":"G303","city":"Accra","units":2},{"id":"K830","city":"Oslo","units":8}]
Checked answer
{"city": "Kyoto", "units": 2}
Seed

Correct

{
  "city": "Kyoto",
  "units": 2
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":2}
12 tokens
0610 · filter · wording 0 · gaine5f2753c7112be2570cd6b2d77a12eaa3255687a95e091637c0c3f5fe68a4a12
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L956","status":"ready"},{"id":"P130","status":"closed"},{"id":"C659","status":"closed"},{"id":"F799","status":"pending"},{"id":"S953","status":"ready"}]
Checked answer
{"ids": ["L956", "S953"]}
Seed

wrong_schema_or_value

{"ids": ["C659", "F799", "L956", "S953", "P130"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L956","S953"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L956","S953"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L956","S953"]}
16 tokens
0611 · total · wording 0 · sameb621f308d81980be5a79d44c388995cb88fc53b4a3ca91d43e718b40e970383a
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"O988","units":6,"price":17},{"id":"K134","units":18,"price":17},{"id":"A302","units":9,"price":13},{"id":"M528","units":14,"price":10},{"id":"S100","units":16,"price":8}]
Checked answer
{"total": 793}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0612 · sort · wording 0 · gain0824dbb4e970eb4a605d1c5057837852d82ea18a0034d296545de815b1db5a13
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"M971","priority":2},{"id":"H722","priority":3},{"id":"U841","priority":2},{"id":"P609","priority":4},{"id":"O929","priority":2}]
Checked answer
{"ids": ["P609", "H722"]}
Seed

wrong_schema_or_value

{"ids": ["M971", "U841", "P609", "H722", "O929"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P609","H722"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H722","P609"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P609","H722"]}
16 tokens
0613 · lookup · wording 1 · gain2c5adab430118cedd4ce1babfed85c0125a92a48466ff545a0109b9a98355514
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id R510 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"I590","city":"Perth","units":3},{"id":"U653","city":"Lima","units":11},{"id":"R510","city":"Accra","units":0},{"id":"E159","city":"Perth","units":4},{"id":"O357","city":"Kyoto","units":19}]
Checked answer
{"city": "Accra", "units": 0}
Seed

wrong_schema_or_value

{"city":"Perth","units":3}
11 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":0}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":0}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":0}
11 tokens
0614 · filter · wording 1 · gainc9939dd6916a2db3260c043dfb8f53cf679d895713a38fed5dedf317b4bf1304
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"G783","status":"pending"},{"id":"G706","status":"pending"},{"id":"G638","status":"ready"},{"id":"C309","status":"pending"},{"id":"K503","status":"closed"}]
Checked answer
{"ids": ["G638"]}
Seed

wrong_schema_or_value

{
  "ids": ["G638", "G706", "G783", "K503"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["G638"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["G638"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["G638"]}
11 tokens
0615 · total · wording 1 · samea7442d90f8f836f267706080d01b805e9ca7c69d27810a3adf8c88667babcee1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"R836","units":1,"price":16},{"id":"K414","units":14,"price":3},{"id":"C956","units":14,"price":19},{"id":"Y307","units":5,"price":5},{"id":"K756","units":14,"price":13}]
Checked answer
{"total": 531}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0616 · sort · wording 1 · same16b3ca7cd6308c86c64fe2d01cf43cbadcf7060b6854f161887ecaa48f78eb97
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"S407","priority":2},{"id":"U102","priority":8},{"id":"Q613","priority":6},{"id":"K522","priority":6},{"id":"B769","priority":3}]
Checked answer
{"ids": ["U102", "K522"]}
Seed

invalid_json_object

[{"id":"U102","priority":8},{"id":"Q613","priority":6},{"id":"S407","priority":2},{"id":"K522","priority":6},{"id":"B769","priority":3}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["U102","Q613"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["U102","K522"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["U102","Q613"]}
16 tokens
0617 · lookup · wording 2 · gain1909d6c9b0286a78ebdaa3dca02e311494d8def3e6908f04409e555817948997
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to S192? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"S192","city":"Perth","units":6},{"id":"O819","city":"Lima","units":4},{"id":"Y816","city":"Oslo","units":16},{"id":"L782","city":"Oslo","units":2},{"id":"G701","city":"Perth","units":9}]
Checked answer
{"city": "Perth", "units": 6}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":6}
11 tokens
0618 · filter · wording 2 · samea9e46a44d07a66a4fdd58492a8666ab041975ac06a04153f6bca6516c598e1d1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"A237","status":"ready"},{"id":"C924","status":"ready"},{"id":"T655","status":"closed"},{"id":"J454","status":"pending"},{"id":"E969","status":"closed"}]
Checked answer
{"ids": ["A237", "C924"]}
Seed

invalid_json_object

[{"id":"A237"},{"id":"C924"},{"id":"E969"},{"id":"J454"},{"id":"T655"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["A237","C924"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A237","C924"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["C924","A237"]]
16 tokens
0619 · total · wording 2 · samed24a8d94ee7caa98cc4b1a65a0f16736d47646eb4911fcc52269575e9a2231c0
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"L655","units":13,"price":10},{"id":"F563","units":6,"price":4},{"id":"Y478","units":19,"price":18},{"id":"C738","units":9,"price":3},{"id":"F113","units":4,"price":1}]
Checked answer
{"total": 527}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0620 · sort · wording 2 · gain628c018efeda5c8a7d9efa57b37c34755f311e631338029d279ae3a9d744edc0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"B260","priority":2},{"id":"A300","priority":4},{"id":"Q546","priority":6},{"id":"J237","priority":1},{"id":"F177","priority":3}]
Checked answer
{"ids": ["Q546", "A300"]}
Seed

wrong_schema_or_value

{"ids": ["A300", "B260", "F177", "J237", "Q546"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Q546","A300"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A300","Q546"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q546","A300"]}
16 tokens
0621 · lookup · wording 3 · gaind6488bd46fed86720624a5e8e7ea5f0c532e66d362a25aaaea2a60320c0cfdd4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"X290","city":"Perth","units":18},{"id":"T948","city":"Lima","units":18},{"id":"U694","city":"Suva","units":12},{"id":"G524","city":"Lima","units":5},{"id":"G910","city":"Bern","units":5}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Perth","units":18}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0622 · filter · wording 3 · samee5e6b1b33f72ceab2438601887ee55f3d6bae7014001cc9c588382dfae5b3b78
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"M138","status":"ready"},{"id":"C398","status":"ready"},{"id":"W281","status":"pending"},{"id":"D751","status":"pending"},{"id":"G918","status":"ready"}]
Checked answer
{"ids": ["C398", "G918", "M138"]}
Seed

wrong_schema_or_value

{"ids": ["C398", "D751", "G918", "M138", "W281"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C398","G918","M138"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["C398","G918","M138"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C398","M138","G918"]}
21 tokens
0623 · total · wording 3 · samed24fa2ae22bad04f6c2a2fb2083f1cd315f44bb31a51c6a1ed40fcf780b5bbac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"M924","units":16,"price":12},{"id":"M710","units":14,"price":13},{"id":"T700","units":2,"price":11},{"id":"S482","units":5,"price":6},{"id":"Y444","units":4,"price":19}]
Checked answer
{"total": 502}
Seed

wrong_schema_or_value

{
  "total": 34
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":408}
8 tokens
0624 · sort · wording 3 · gainde9b4e2b781762b74bcbad0d19a598d52297603e97bc24ee52fa733f4533bcf3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"L675","priority":2},{"id":"U736","priority":3},{"id":"G268","priority":5},{"id":"O963","priority":3},{"id":"K213","priority":5}]
Checked answer
{"ids": ["G268", "K213"]}
Seed

invalid_json_object

[{"id":"U736","priority":3},{"id":"G268","priority":5},{"id":"L675","priority":2},{"id":"K213","priority":5}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["G268","K213"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G268","K213"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G268","K213"]}
16 tokens
0625 · lookup · wording 0 · gain98632625a09f5c96fc634453b0e07a68e5674c36a6f905d6025a17a8d8527878
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record D576. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y350","city":"Perth","units":10},{"id":"I570","city":"Oslo","units":10},{"id":"H490","city":"Kyoto","units":11},{"id":"N146","city":"Riga","units":19},{"id":"D576","city":"Lima","units":17}]
Checked answer
{"city": "Lima", "units": 17}
Seed

invalid_json_object

[{"city":"Lima","units":17}]
13 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":17}
12 tokens
0626 · filter · wording 0 · gain028908eaf66c0fc60b90f284f725220d253ee85f964b2f0d5917848aa5ea6fd2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"U140","status":"pending"},{"id":"J474","status":"closed"},{"id":"Q377","status":"pending"},{"id":"Z169","status":"pending"},{"id":"P351","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["J474", "P351", "Q377", "U140", "Z169"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0627 · total · wording 0 · samedc1c8ed3dde9fd73a95179a8d5acfd9dc5bb79a2ff2504f48ea5b60a61afe3d1
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P818","units":0,"price":8},{"id":"U704","units":10,"price":9},{"id":"G387","units":12,"price":7},{"id":"P380","units":3,"price":1},{"id":"Z922","units":13,"price":16}]
Checked answer
{"total": 385}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":368}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0628 · sort · wording 0 · gain61b751f7f4309ceff1387390b9cf271d9b3ab1c70a41e08af215fc39edd0f483
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"G645","priority":4},{"id":"H438","priority":6},{"id":"J570","priority":4},{"id":"F871","priority":3},{"id":"Y328","priority":1}]
Checked answer
{"ids": ["H438", "G645"]}
Seed

wrong_schema_or_value

{"ids": ["G645", "H438", "J570", "F871", "Y328"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H438","J570"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H438","G645"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H438","G645"]}
16 tokens
0629 · lookup · wording 1 · gain2fa4d8acc27f521bf8880838b7c95d76b64df9bdc134a8f8a954ecde80976bbc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id I580 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"G885","city":"Riga","units":13},{"id":"M173","city":"Kyoto","units":6},{"id":"Z738","city":"Suva","units":1},{"id":"R354","city":"Kyoto","units":13},{"id":"I580","city":"Bern","units":6}]
Checked answer
{"city": "Bern", "units": 6}
Seed

wrong_schema_or_value

{"city":"Riga","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":6}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":6}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":6}
10 tokens
0630 · filter · wording 1 · gain04dece1328fca0136cb871852c6295c8214b59f55224f669e84516f07d144855
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"G781","status":"pending"},{"id":"S524","status":"ready"},{"id":"B592","status":"pending"},{"id":"C989","status":"pending"},{"id":"V448","status":"ready"}]
Checked answer
{"ids": ["S524", "V448"]}
Seed

wrong_schema_or_value

{
  "ids": ["B592", "C989", "G781", "S524", "V448"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["V448","S524"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V448","S524"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S524","V448"]}
16 tokens
0631 · total · wording 1 · samef6b8cd5a3653e391e947d5fe9d88cbab630dcec4950b1b9b9b492933cf302527
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"I429","units":3,"price":1},{"id":"H697","units":19,"price":18},{"id":"X815","units":18,"price":14},{"id":"E892","units":13,"price":3},{"id":"T418","units":13,"price":15}]
Checked answer
{"total": 831}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0632 · sort · wording 1 · gainfde98db757531ba8dd147916eff6618709963dfdc55a1c899259c39d09549311
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Q895","priority":9},{"id":"I690","priority":9},{"id":"Q823","priority":4},{"id":"T860","priority":1},{"id":"Z869","priority":8}]
Checked answer
{"ids": ["I690", "Q895"]}
Seed

invalid_json_object

[{"id":"Q895","priority":9},{"id":"I690","priority":9},{"id":"T860","priority":1},{"id":"Z869","priority":8}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["I690","Q895"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I690","Q895"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I690","Q895"]}
16 tokens
0633 · lookup · wording 2 · gain83cc1cdf5b6be26abcad1ce8f0c877b1888fca802182829bff2bb3dbcf8273ac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to R910? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"B938","city":"Accra","units":5},{"id":"U782","city":"Suva","units":16},{"id":"R910","city":"Suva","units":13},{"id":"C605","city":"Suva","units":12},{"id":"L170","city":"Perth","units":4}]
Checked answer
{"city": "Suva", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":13}
12 tokens
0634 · filter · wording 2 · samee55246c975be67e0797be8ab38c25e61e9fb79348d6d10a996c7efaebbf7b0ec
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"B753","status":"ready"},{"id":"S178","status":"closed"},{"id":"I684","status":"pending"},{"id":"Y747","status":"ready"},{"id":"Y953","status":"ready"}]
Checked answer
{"ids": ["B753", "Y747", "Y953"]}
Seed

invalid_json_object

[{"id":"B753","status":"ready"},{"id":"Y747","status":"ready"},{"id":"Y953","status":"ready"}]
39 tokens
1 GPU · AdamW

Correct

{"ids":["B753","Y747","Y953"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["B753","Y747","Y953"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["B753","Y747","Y953"]]
21 tokens
0635 · total · wording 2 · same1b68dfd7dd4bb8d674f38364c9c30e058b1f8e3dd01fd94ce3d379d429eac0c2
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"U781","units":14,"price":1},{"id":"T544","units":14,"price":6},{"id":"U967","units":16,"price":2},{"id":"Z827","units":19,"price":9},{"id":"X986","units":16,"price":4}]
Checked answer
{"total": 365}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0636 · sort · wording 2 · gaind57283c54586a81fabf6cf1de82f46a012d68a1a5c88fe34c5535a411166f9cb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"K151","priority":5},{"id":"Z342","priority":3},{"id":"B444","priority":8},{"id":"G606","priority":7},{"id":"S853","priority":5}]
Checked answer
{"ids": ["B444", "G606"]}
Seed

wrong_schema_or_value

{"ids": ["K151", "Z342", "B444", "G606", "S853"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B444","G606"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B444","G606"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B444","G606"]}
16 tokens
0637 · lookup · wording 3 · sameded3d8c3a59809e3f88e0b657cfcf8a0001f2062e640f47f93bbc9166502a5c2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=J117. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T200","city":"Lima","units":2},{"id":"N361","city":"Oslo","units":1},{"id":"Z373","city":"Accra","units":8},{"id":"J117","city":"Kyoto","units":5},{"id":"B579","city":"Riga","units":7}]
Checked answer
{"city": "Kyoto", "units": 5}
Seed

Correct

{"city":"Kyoto","units":5}
12 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":5}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":5}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":5}
12 tokens
0638 · filter · wording 3 · gain849dc2a19d779246a9a5b3aeedcb37db8f90307bc421603f5da10ccef593f30e
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"G217","status":"ready"},{"id":"A711","status":"pending"},{"id":"I780","status":"ready"},{"id":"M998","status":"closed"},{"id":"X248","status":"ready"}]
Checked answer
{"ids": ["G217", "I780", "X248"]}
Seed

wrong_schema_or_value

{"ids": ["A711", "G217", "I780", "M998", "X248"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G217","I780","X248"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A711","G217","I780","X248"]}
26 tokens
4 GPUs · local windows

Correct

{"ids":["G217","I780","X248"]}
21 tokens
0639 · total · wording 3 · same656f7902252325355711149fbd97e216bd85691531f0b6f01416d8446bee9b01
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"W142","units":5,"price":12},{"id":"U963","units":5,"price":19},{"id":"F245","units":7,"price":17},{"id":"R247","units":16,"price":10},{"id":"Y168","units":16,"price":7}]
Checked answer
{"total": 546}
Seed

invalid_json_object

{
  "total": 12 + 19 + 17 + 10 + 7 = 67
}
30 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0640 · sort · wording 3 · gain76c6f34488d8f6cd0735b16ccac61b4d86b5f8eb199b4f5fc8ab88371f296372
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"T836","priority":6},{"id":"Z670","priority":8},{"id":"W476","priority":6},{"id":"Z434","priority":9},{"id":"Q549","priority":8}]
Checked answer
{"ids": ["Z434", "Q549"]}
Seed

invalid_json_object

[{"id":"Z670","priority":8},{"id":"W476","priority":6},{"id":"Z434","priority":9},{"id":"T836","priority":6}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["Z434","Q549"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Q549","Z434"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z434","Q549"]}
16 tokens
0641 · lookup · wording 0 · gain4356f88e07a5bc7db9d2cb19ff07048229b18665e1130954d70f48ad7a59cf62
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"L565","city":"Lima","units":16},{"id":"O897","city":"Suva","units":13},{"id":"U654","city":"Perth","units":13},{"id":"I174","city":"Kyoto","units":4},{"id":"D613","city":"Lima","units":18}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Lima","units":16},{"city":"Suva","units":13},{"city":"Perth","units":13},{"city":"Kyoto","units":4},{"city":"Lima","units":18}]
57 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0642 · filter · wording 0 · gainc16cf6aebb9fd9031063bc8728c9ffc854e78e30102ed67103f776a5675698bf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"N306","status":"closed"},{"id":"K689","status":"pending"},{"id":"N980","status":"closed"},{"id":"C840","status":"closed"},{"id":"J412","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["C840", "J412", "N306", "N980", "K689"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0643 · total · wording 0 · samead1147cca062ca5ad60c4d076090142448d54ddea4d4ac5ae014e309ae90153e
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P442","units":19,"price":7},{"id":"Q977","units":10,"price":9},{"id":"X773","units":7,"price":4},{"id":"J935","units":2,"price":5},{"id":"W311","units":4,"price":10}]
Checked answer
{"total": 301}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0644 · sort · wording 0 · same5466a78f0bcfd280ffe9823f1b695cfc5b43d78f11ee65999b3685989f881e1f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"X187","priority":1},{"id":"X635","priority":6},{"id":"B342","priority":5},{"id":"A659","priority":5},{"id":"O297","priority":4}]
Checked answer
{"ids": ["X635", "A659"]}
Seed

wrong_schema_or_value

{"ids": ["X187", "B342"]}
17 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["X635","B342"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["X635","B342"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["X635","B342"]}
16 tokens
0645 · lookup · wording 1 · sameae594630c47193a16a820d046566552988683d4872ddc4b45885f5c743801f4e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z340 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"H647","city":"Accra","units":0},{"id":"A151","city":"Bern","units":11},{"id":"F766","city":"Oslo","units":1},{"id":"Z340","city":"Perth","units":15},{"id":"C260","city":"Oslo","units":10}]
Checked answer
{"city": "Perth", "units": 15}
Seed

Correct

{"city":"Perth","units":15}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":15}
12 tokens
0646 · filter · wording 1 · gain476c3757175163b9132ac32b7f41231f219cdcdeaabb270e91b943f665870290
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"F341","status":"closed"},{"id":"L150","status":"pending"},{"id":"S426","status":"pending"},{"id":"C206","status":"ready"},{"id":"J396","status":"ready"}]
Checked answer
{"ids": ["C206", "J396"]}
Seed

wrong_schema_or_value

{
  "ids": ["C206", "F341", "J396", "L150", "S426"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["C206","J396"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C206","J396"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C206","J396"]}
16 tokens
0647 · total · wording 1 · same85a0f15f0b0ed4d2bf1d171c5c62b32db4713e1a54ce058b08686bec19f6dfe9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"G873","units":6,"price":14},{"id":"K725","units":18,"price":5},{"id":"G299","units":14,"price":12},{"id":"V568","units":5,"price":8},{"id":"B310","units":19,"price":18}]
Checked answer
{"total": 724}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0648 · sort · wording 1 · gain6d91f08b359f9abc1d30d32f2e37f6608b6c61376e03e33e49f127aa10a69076
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"O538","priority":3},{"id":"W661","priority":7},{"id":"H479","priority":3},{"id":"C420","priority":5},{"id":"C455","priority":5}]
Checked answer
{"ids": ["W661", "C420"]}
Seed

invalid_json_object

[{"id":"O538","priority":3},{"id":"H479","priority":3},{"id":"C420","priority":5},{"id":"C455","priority":5}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["W661","C420"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["W661","C420"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["W661","C420"]}
16 tokens
0649 · lookup · wording 2 · gain59af4f1693447b74ab91248f87653acd29654f4cc73192e2796b83ad4e327a21
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M313? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"P390","city":"Bern","units":9},{"id":"Z797","city":"Perth","units":10},{"id":"L396","city":"Lima","units":5},{"id":"K305","city":"Kyoto","units":8},{"id":"M313","city":"Bern","units":6}]
Checked answer
{"city": "Bern", "units": 6}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":6}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":6}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":6}
10 tokens
0650 · filter · wording 2 · same9bf6675f8ec13689d0a0e4d5dada24dd35198702897c17f39e685b084f4c4c2e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"C147","status":"ready"},{"id":"S669","status":"pending"},{"id":"B992","status":"closed"},{"id":"T822","status":"pending"},{"id":"N544","status":"ready"}]
Checked answer
{"ids": ["C147", "N544"]}
Seed

invalid_json_object

[{"id":"C147"},{"id":"N544"},{"id":"S669"},{"id":"T822"},{"id":"B992"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["C147","N544"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C147","N544"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["C147","N544"]]
16 tokens
0651 · total · wording 2 · samec7f1f0b9c2d99fb43ca420e5f849a1ca2c1987272fe4693fc4c916f3c92470b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"J992","units":17,"price":13},{"id":"U696","units":17,"price":2},{"id":"C353","units":4,"price":9},{"id":"E813","units":18,"price":8},{"id":"G409","units":15,"price":8}]
Checked answer
{"total": 555}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0652 · sort · wording 2 · same4284ab4223dde5b96ac846920c3aeca69e312b721ebcb3119301779fd48d81b4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"V442","priority":7},{"id":"J863","priority":1},{"id":"W572","priority":6},{"id":"R251","priority":8},{"id":"A646","priority":7}]
Checked answer
{"ids": ["R251", "A646"]}
Seed

wrong_schema_or_value

{"ids": ["V442", "J863", "W572", "R251", "A646"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["R251","A646"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["R251","V442"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["R251","V442"]}
16 tokens
0653 · lookup · wording 3 · gain68f3f5294ec31b513876606878dd827a0160e05752db3a56ff580d58da41b1ce
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T204. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"Y524","city":"Kyoto","units":15},{"id":"W903","city":"Perth","units":5},{"id":"J804","city":"Kyoto","units":16},{"id":"T204","city":"Riga","units":16},{"id":"F370","city":"Kyoto","units":0}]
Checked answer
{"city": "Riga", "units": 16}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":15}
13 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":16}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":16}
12 tokens
0654 · filter · wording 3 · gainf48fe4a76ca6ff0423eaa4b42853ec248234305b5d100c8a3ca542a50523a009
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"B289","status":"ready"},{"id":"P240","status":"closed"},{"id":"F334","status":"closed"},{"id":"I746","status":"pending"},{"id":"Q124","status":"pending"}]
Checked answer
{"ids": ["B289"]}
Seed

wrong_schema_or_value

{"ids": ["B289", "P240", "F334", "I746", "Q124"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B289"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["B289"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["B289"]}
11 tokens
0655 · total · wording 3 · same580c759f16e64dfc6da89236c62ca81a276eb9a3bcad4d0f3841c367ef9905a0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"O752","units":5,"price":3},{"id":"S200","units":8,"price":1},{"id":"A571","units":15,"price":9},{"id":"F803","units":1,"price":12},{"id":"G990","units":9,"price":12}]
Checked answer
{"total": 278}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0656 · sort · wording 3 · samedd5657a67f07be4a4b4e86d95561ba55883ee54c00639eff915ac2b72b7c3a44
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"B952","priority":5},{"id":"X238","priority":6},{"id":"I383","priority":7},{"id":"C965","priority":7},{"id":"Y256","priority":6}]
Checked answer
{"ids": ["C965", "I383"]}
Seed

invalid_json_object

[{"id":"B952","priority":5},{"id":"Y256","priority":6},{"id":"C965","priority":7},{"id":"I383","priority":7}]
50 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I383","C965"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["I383","C965"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["I383","C965"]}
16 tokens
0657 · lookup · wording 0 · gain2acaf3cb654f2cf6dd6254ed7ca0225ddea37e8083a151643aac27e1f2afe308
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record P160. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Q815","city":"Suva","units":11},{"id":"P160","city":"Bern","units":3},{"id":"F178","city":"Lima","units":7},{"id":"O385","city":"Perth","units":7},{"id":"V641","city":"Oslo","units":12}]
Checked answer
{"city": "Bern", "units": 3}
Seed

wrong_schema_or_value

{
  "city": "Suva",
  "units": 11
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":3}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":3}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":3}
10 tokens
0658 · filter · wording 0 · gain98618e9f57fc85601065f774a142d12b5d1efec75048011ece1af267dc6259c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"I312","status":"pending"},{"id":"D544","status":"ready"},{"id":"S508","status":"pending"},{"id":"B303","status":"pending"},{"id":"L531","status":"ready"}]
Checked answer
{"ids": ["D544", "L531"]}
Seed

wrong_schema_or_value

{"ids": ["B303", "D544", "I312", "L531", "S508"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D544","L531"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L531","D544"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D544","L531"]}
16 tokens
0659 · total · wording 0 · same6935663a168a139b6ad7010da2714f18bf4ad9c729d4b029fc7eb182da2f027a
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"U656","units":2,"price":7},{"id":"C807","units":4,"price":8},{"id":"B118","units":2,"price":15},{"id":"M345","units":5,"price":7},{"id":"H391","units":13,"price":19}]
Checked answer
{"total": 358}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0660 · sort · wording 0 · same0f951819169ab5fbe076243d2a839715045686e09065a1a195b9d62239d98480
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"N268","priority":9},{"id":"Z154","priority":7},{"id":"D537","priority":2},{"id":"M942","priority":9},{"id":"E863","priority":2}]
Checked answer
{"ids": ["M942", "N268"]}
Seed

wrong_schema_or_value

{"ids": ["M942", "D537", "N268", "Z154", "E863"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M942","N268"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M942","N268"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N268","M942"]}
16 tokens
0661 · lookup · wording 1 · gain9bac7d0dcb467f6c58039230ab21e6a573599fbbb52e1339b9e26885928949c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"K619","city":"Bern","units":11},{"id":"Q484","city":"Kyoto","units":2},{"id":"N336","city":"Lima","units":17},{"id":"E502","city":"Bern","units":14},{"id":"Z860","city":"Riga","units":6}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Bern","units":11}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0662 · filter · wording 1 · gain6cee8e0305f0d5a6aabe8f7842ebdc43ee3bd40bf3c82ee99bc1d63ffd31e34e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"Y767","status":"closed"},{"id":"Q423","status":"closed"},{"id":"J328","status":"pending"},{"id":"H330","status":"pending"},{"id":"R950","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["J328", "H330", "R950"]
}
26 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0663 · total · wording 1 · sameecf0f2fd871f8d42325e3655c6c3124a8262db498f8384ebd8510fd8073556d9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"U789","units":16,"price":17},{"id":"J535","units":16,"price":9},{"id":"L648","units":4,"price":17},{"id":"W766","units":7,"price":11},{"id":"G449","units":16,"price":14}]
Checked answer
{"total": 785}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0664 · sort · wording 1 · same516eed91fb0d8b06e290ba6907798e70fa93fa2b2fc3336c1ae1356c0c10757d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Q752","priority":8},{"id":"J226","priority":4},{"id":"K925","priority":4},{"id":"E941","priority":4},{"id":"O758","priority":2}]
Checked answer
{"ids": ["Q752", "E941"]}
Seed

invalid_json_object

[{"id":"Q752","priority":8},{"id":"J226","priority":4},{"id":"K925","priority":4},{"id":"E941","priority":4},{"id":"O758","priority":2}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Q752","J226"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Q752","J226"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q752","J226"]}
16 tokens
0665 · lookup · wording 2 · gaine6e79ed0a3fe5e4c3f4992be8c8c08ae1858a5a156fcfd4385666c0d0d12397a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to L273? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"N614","city":"Lima","units":0},{"id":"E714","city":"Oslo","units":1},{"id":"U315","city":"Riga","units":7},{"id":"N742","city":"Perth","units":3},{"id":"L273","city":"Suva","units":5}]
Checked answer
{"city": "Suva", "units": 5}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":5}
11 tokens
0666 · filter · wording 2 · same1d81daab4530ac7c047472aefda8585bedcef5adf226dd9c30c0d117eb08421d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"H360","status":"pending"},{"id":"G842","status":"closed"},{"id":"T210","status":"pending"},{"id":"O859","status":"closed"},{"id":"E876","status":"pending"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"H360"},{"id":"G842"},{"id":"T210"},{"id":"O859"},{"id":"E876"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

invalid_json_object

["ids":[]}
6 tokens
0667 · total · wording 2 · sameac7f2892c7109cb9511300aea5d841e0a6c7df084669baf5001c2423c0368da0
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Q978","units":14,"price":19},{"id":"O917","units":14,"price":19},{"id":"P126","units":6,"price":16},{"id":"F599","units":12,"price":12},{"id":"M857","units":13,"price":10}]
Checked answer
{"total": 902}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0668 · sort · wording 2 · same39c33d333222db5fa8ff837f8a43c28e942d291be6e2f8353fda4504433eb057
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S193","priority":4},{"id":"P550","priority":9},{"id":"C986","priority":8},{"id":"Y855","priority":7},{"id":"Z207","priority":2}]
Checked answer
{"ids": ["P550", "C986"]}
Seed

wrong_schema_or_value

{"ids": ["Y855", "Z207", "S193", "P550", "C986"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P550","C986"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P550","C986"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["P550","S193"]}
16 tokens
0669 · lookup · wording 3 · samee7e0add968fd13dbdfefd0019e6c8ed5cabac8340effd1a14ce5595becca7da7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=P476. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"A553","city":"Accra","units":17},{"id":"A991","city":"Kyoto","units":16},{"id":"R386","city":"Oslo","units":16},{"id":"P476","city":"Kyoto","units":16},{"id":"S444","city":"Kyoto","units":9}]
Checked answer
{"city": "Kyoto", "units": 16}
Seed

Correct

{"city": "Kyoto", "units": 16}
16 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":16}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":16}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":16}
13 tokens
0670 · filter · wording 3 · gain4525c37b57cfe145fe4dc0247b3303fe6810b835ec5dec813a164811778d325b
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D809","status":"pending"},{"id":"T751","status":"pending"},{"id":"D769","status":"pending"},{"id":"T254","status":"ready"},{"id":"N536","status":"pending"}]
Checked answer
{"ids": ["T254"]}
Seed

wrong_schema_or_value

{"ids": ["D769", "D809", "T254", "T751", "N536"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["T254"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T254"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["T254"]}
11 tokens
0671 · total · wording 3 · samea1781d4ce742fad4f589cc14b76acef589ed53de7673a5f7c2a600d4d113c455
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"D773","units":14,"price":2},{"id":"U803","units":16,"price":11},{"id":"H679","units":18,"price":16},{"id":"C139","units":17,"price":13},{"id":"E265","units":5,"price":6}]
Checked answer
{"total": 743}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0672 · sort · wording 3 · gaind9a3ca66054dd3a7c5ee78e99dab3b0876ee0f8c57b43df7634599c61009959b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"L930","priority":3},{"id":"X410","priority":7},{"id":"X846","priority":3},{"id":"P230","priority":8},{"id":"R829","priority":7}]
Checked answer
{"ids": ["P230", "R829"]}
Seed

invalid_json_object

[{"id":"L930","priority":3},{"id":"X846","priority":3},{"id":"X410","priority":7},{"id":"P230","priority":8}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["P230","R829"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P230","R829"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P230","R829"]}
16 tokens
0673 · lookup · wording 0 · gain3aba4cd91dba243ca02549eb25bcd6e8e6fb9b899ca84ed278240cf2c3cab878
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record G347. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"G347","city":"Lima","units":2},{"id":"E248","city":"Oslo","units":0},{"id":"Z390","city":"Perth","units":5},{"id":"S811","city":"Riga","units":2},{"id":"M101","city":"Lima","units":4}]
Checked answer
{"city": "Lima", "units": 2}
Seed

invalid_json_object

[{"city":"Lima","units":2}]
12 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":2}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":2}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":2}
11 tokens
0674 · filter · wording 0 · gainec20d49a231af6055e6f9a1db757391e8a4c949d1f39f1069252bf6af195050f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"U215","status":"pending"},{"id":"F598","status":"pending"},{"id":"K966","status":"closed"},{"id":"B688","status":"pending"},{"id":"M852","status":"ready"}]
Checked answer
{"ids": ["M852"]}
Seed

wrong_schema_or_value

{"ids": ["B688", "F598", "K966", "M852", "U215"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M852"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M852"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M852"]}
11 tokens
0675 · total · wording 0 · same0b9a42b3c0944ca7343f024ef9ee68544a68e7c9c1359efef15041547197aa96
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"X364","units":19,"price":17},{"id":"F459","units":0,"price":15},{"id":"F541","units":16,"price":15},{"id":"E457","units":16,"price":7},{"id":"Q468","units":11,"price":15}]
Checked answer
{"total": 840}
Seed

invalid_json_object

{"total": 17 * 19 + 15 * 0 + 15 * 16 + 7 * 16 + 15 * 11}
42 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0676 · sort · wording 0 · same1372c054b633084c2058d242c3085abedb90480ff3e9c11f8f1e11f3fc5ceb73
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"X881","priority":1},{"id":"U772","priority":7},{"id":"E876","priority":5},{"id":"U209","priority":5},{"id":"Z595","priority":3}]
Checked answer
{"ids": ["U772", "E876"]}
Seed

wrong_schema_or_value

{"ids": ["U772", "U209", "X881", "Z595", "E876"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["U772","E876"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E876","U209"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E772","U209"]}
16 tokens
0677 · lookup · wording 1 · samefd4428a18ee98b4b2e95087367ef49f8b275f210affe39ee87ce2782f41b8f4b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Y893 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"C243","city":"Kyoto","units":15},{"id":"I896","city":"Suva","units":0},{"id":"W101","city":"Suva","units":7},{"id":"Y893","city":"Perth","units":12},{"id":"L328","city":"Lima","units":5}]
Checked answer
{"city": "Perth", "units": 12}
Seed

Correct

{"city":"Perth","units":12}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":12}
12 tokens
0678 · filter · wording 1 · samedf0c895be5f1dab99dcdc5983dc81292f6faf2dd3db6176a87da402e90f0135e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"V178","status":"pending"},{"id":"M682","status":"pending"},{"id":"E570","status":"ready"},{"id":"I740","status":"ready"},{"id":"S576","status":"ready"}]
Checked answer
{"ids": ["E570", "I740", "S576"]}
Seed

wrong_schema_or_value

{
  "ids": ["I740", "E570", "S576", "V178", "M682"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I740","S576","E570"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["E570","I740","S576"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E570","S576","I740"]}
21 tokens
0679 · total · wording 1 · same0d1c6585f3837d9d3e49e756005b8f1b399b54f0b8074cc5de802b0c726b27ae
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"H157","units":15,"price":19},{"id":"X416","units":7,"price":17},{"id":"X824","units":11,"price":5},{"id":"V129","units":15,"price":19},{"id":"C709","units":16,"price":7}]
Checked answer
{"total": 856}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0680 · sort · wording 1 · samece1200bc31154b054ba040358469210af8c76e21825a57d68b84d8eb76c98031
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V778","priority":5},{"id":"Y851","priority":3},{"id":"Q616","priority":9},{"id":"M808","priority":9},{"id":"G370","priority":2}]
Checked answer
{"ids": ["M808", "Q616"]}
Seed

invalid_json_object

[{"id":"V778","priority":5},{"id":"Y851","priority":3},{"id":"Q616","priority":9},{"id":"M808","priority":9},{"id":"G370","priority":2}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["M808","Q616"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M808","Q616"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q616","M808"]}
16 tokens
0681 · lookup · wording 2 · same03b421c16886ea110f5dfb206df951e62dd316f846751d894e88e76ee4e6f38a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"K545","city":"Perth","units":18},{"id":"N564","city":"Riga","units":15},{"id":"Q959","city":"Oslo","units":1},{"id":"Z916","city":"Lima","units":6},{"id":"W746","city":"Lima","units":18}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0682 · filter · wording 2 · same6ad89ecbb803c98ec54a04ca3bd900faf9265a8ab138daadfb1c70c086a8eccb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"B193","status":"closed"},{"id":"A299","status":"closed"},{"id":"V659","status":"ready"},{"id":"R880","status":"ready"},{"id":"L731","status":"pending"}]
Checked answer
{"ids": ["R880", "V659"]}
Seed

invalid_json_object

[{"id":"V659"}]
11 tokens
1 GPU · AdamW

Correct

{"ids":["R880","V659"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["R880","V659"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["R880","V659"]]
16 tokens
0683 · total · wording 2 · same6c548d3a8a3d69b6e02e92515160a086b9605137bb9cc456e422592af62ddcb3
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Q341","units":17,"price":8},{"id":"P882","units":10,"price":18},{"id":"O124","units":13,"price":17},{"id":"V143","units":3,"price":14},{"id":"Q335","units":2,"price":5}]
Checked answer
{"total": 589}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0684 · sort · wording 2 · gain80dc53f772ef83438ded6e7fc2b4b4ccb4543886f1ecc973c2c4a8a4ff9ffc0e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"V694","priority":2},{"id":"S869","priority":5},{"id":"M166","priority":7},{"id":"W428","priority":4},{"id":"N578","priority":2}]
Checked answer
{"ids": ["M166", "S869"]}
Seed

wrong_schema_or_value

{"ids": ["V694", "S869", "M166", "W428", "N578"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M166","S869"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M166","S869"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M166","S869"]}
16 tokens
0685 · lookup · wording 3 · sameecb3c9e3bb05993be15a1b02f5338fdcba1f14980a801b3ee79559a62adcebc6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=O121. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"J401","city":"Kyoto","units":5},{"id":"X253","city":"Lima","units":6},{"id":"O905","city":"Perth","units":19},{"id":"O121","city":"Kyoto","units":11},{"id":"I832","city":"Oslo","units":7}]
Checked answer
{"city": "Kyoto", "units": 11}
Seed

Correct

{"city":"Kyoto","units":11}
13 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":11}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":11}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":11}
13 tokens
0686 · filter · wording 3 · gain52e5308f3ae3793a48077619e0f63138f9d165b55013cbc87f5a985696c501ff
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"F278","status":"closed"},{"id":"B734","status":"closed"},{"id":"M443","status":"pending"},{"id":"F991","status":"closed"},{"id":"N947","status":"ready"}]
Checked answer
{"ids": ["N947"]}
Seed

wrong_schema_or_value

{"ids": ["F278", "B734", "M443", "F991", "N947"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["N947"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["N947"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["N947"]}
11 tokens
0687 · total · wording 3 · samef85e3b3c619cb275381f78d238a97dac2642e957fad2262da4a8e74a1712bd0b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"C323","units":0,"price":5},{"id":"T414","units":11,"price":19},{"id":"R547","units":11,"price":18},{"id":"N192","units":19,"price":17},{"id":"I645","units":16,"price":12}]
Checked answer
{"total": 922}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0688 · sort · wording 3 · gaind39468d18a6e90fd0edc6a2148b28b8e5d6ba4bbad095b4d029a96645a899333
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"S187","priority":5},{"id":"D752","priority":1},{"id":"C335","priority":3},{"id":"H973","priority":5},{"id":"W921","priority":1}]
Checked answer
{"ids": ["H973", "S187"]}
Seed

invalid_json_object

[{"id":"S187"},{"id":"D752"},{"id":"C335"},{"id":"H973"},{"id":"W921"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["H973","S187"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H973","S187"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H973","S187"]}
16 tokens
0689 · lookup · wording 0 · samea9922102fc25d0c9ec9fa1d8f2bed39287d648e4779a2fe94777232977d9d701
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record A871. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"U778","city":"Oslo","units":7},{"id":"N909","city":"Perth","units":15},{"id":"A871","city":"Oslo","units":15},{"id":"Q929","city":"Riga","units":5},{"id":"S938","city":"Lima","units":11}]
Checked answer
{"city": "Oslo", "units": 15}
Seed

Correct

{
  "city": "Oslo",
  "units": 15
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":15}
12 tokens
0690 · filter · wording 0 · gain8c731a32712d6c08152f16b17438b69a8bd1794934ac64c838c32534dc7b0010
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"T628","status":"closed"},{"id":"O782","status":"closed"},{"id":"S281","status":"closed"},{"id":"B348","status":"closed"},{"id":"O475","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["O475", "O782", "S281", "T628", "B348"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0691 · total · wording 0 · same9ee67435686074a27c07cce9cb121def5b666ba1d18efa2274debd0702d64f58
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"T485","units":0,"price":18},{"id":"X388","units":15,"price":16},{"id":"F418","units":13,"price":12},{"id":"F470","units":8,"price":6},{"id":"G364","units":12,"price":9}]
Checked answer
{"total": 552}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0692 · sort · wording 0 · gainbe9c9f47c11bd0a51de789fd74e371ec1b393f831d48a1f138912077228fed81
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z698","priority":8},{"id":"G956","priority":3},{"id":"N334","priority":3},{"id":"S434","priority":1},{"id":"C515","priority":7}]
Checked answer
{"ids": ["Z698", "C515"]}
Seed

wrong_schema_or_value

{"ids": ["G956", "N334", "S434", "C515", "Z698"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Z698","C515"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Z698","C515"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z698","C515"]}
16 tokens
0693 · lookup · wording 1 · gain59040069a1511a7e89c2eb88c66e18427631569d25cc35ec70f82879eeae69dc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M962 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"L979","city":"Lima","units":13},{"id":"D611","city":"Perth","units":8},{"id":"G632","city":"Kyoto","units":6},{"id":"H350","city":"Lima","units":10},{"id":"M962","city":"Perth","units":3}]
Checked answer
{"city": "Perth", "units": 3}
Seed

wrong_schema_or_value

{"city":"Lima","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":3}
11 tokens
0694 · filter · wording 1 · gain695f610f9f44d7c49ea9346c04e29c15ead0d3a86893b59d9a60493c2de06949
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"A791","status":"pending"},{"id":"X884","status":"ready"},{"id":"Z283","status":"closed"},{"id":"N126","status":"ready"},{"id":"P864","status":"pending"}]
Checked answer
{"ids": ["N126", "X884"]}
Seed

wrong_schema_or_value

{
  "ids": ["A791", "X884", "Z283", "N126", "P864"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["N126","X884"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A791","N126","N126"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["N126","X884"]}
16 tokens
0695 · total · wording 1 · samebbbd33084f21ffa220b26f000b2f80063a4dca72146b88800a80c5572799b7e6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D710","units":0,"price":8},{"id":"C263","units":9,"price":11},{"id":"L506","units":10,"price":9},{"id":"P766","units":7,"price":11},{"id":"O554","units":3,"price":8}]
Checked answer
{"total": 290}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0696 · sort · wording 1 · gainc79851a4722c5249ee80e98dadb9cdefa29d7f7f6212c3cfef13829889234fad
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"N960","priority":8},{"id":"F306","priority":2},{"id":"W444","priority":1},{"id":"J965","priority":8},{"id":"Y147","priority":5}]
Checked answer
{"ids": ["J965", "N960"]}
Seed

invalid_json_object

[{"id":"N960","priority":8},{"id":"F306","priority":2},{"id":"W444","priority":1},{"id":"J965","priority":8},{"id":"Y147","priority":5}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["N960","J965"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N960","J965"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J965","N960"]}
16 tokens
0697 · lookup · wording 2 · gain507e6c932b5584ad75631004ee0f3ca8a8a52af4721c7ee3c3db1eac1b9f978f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to D638? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"F499","city":"Perth","units":1},{"id":"H881","city":"Oslo","units":7},{"id":"R144","city":"Accra","units":12},{"id":"U906","city":"Riga","units":9},{"id":"D638","city":"Lima","units":15}]
Checked answer
{"city": "Lima", "units": 15}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": 15
}
17 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":15}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":15}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":15}
12 tokens
0698 · filter · wording 2 · same78977bb14ed8dd1f26772245875a4fb05fb1a2d534d12c04f443773b1a2fe0ef
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"D536","status":"ready"},{"id":"P407","status":"pending"},{"id":"W196","status":"pending"},{"id":"D146","status":"pending"},{"id":"O307","status":"ready"}]
Checked answer
{"ids": ["D536", "O307"]}
Seed

invalid_json_object

[{"id":"D536"},{"id":"O307"},{"id":"P407"},{"id":"W196"},{"id":"D146"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["D536","O307"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D536","O307"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["D536","O307"]]
16 tokens
0699 · total · wording 2 · same42674b8269f999bf6a27e40179951aa7b7865e75ecd063e877bcc2e80dcecf24
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"S915","units":18,"price":11},{"id":"V193","units":6,"price":5},{"id":"R380","units":18,"price":11},{"id":"D466","units":13,"price":4},{"id":"N366","units":0,"price":4}]
Checked answer
{"total": 478}
Seed

wrong_schema_or_value

{"total": 110}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":588}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0700 · sort · wording 2 · samef58d867c81d80499d9db32d1729b97c010494cfa28cc223f51e0d094d6cd79f0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"X588","priority":7},{"id":"R766","priority":5},{"id":"P941","priority":4},{"id":"J337","priority":5},{"id":"N160","priority":5}]
Checked answer
{"ids": ["X588", "J337"]}
Seed

wrong_schema_or_value

{"ids": ["X588", "R766", "P941", "J337", "N160"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["X588","J337"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["X588","J337"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["X588","R766"]}
16 tokens
0701 · lookup · wording 3 · gain3bdd2dd562b7a3d91447b05c853fff7ae3dd781048ba74df340db20cd1ec8b2d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"R600","city":"Suva","units":19},{"id":"C880","city":"Lima","units":17},{"id":"W228","city":"Suva","units":18},{"id":"B269","city":"Perth","units":13},{"id":"O598","city":"Bern","units":3}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Suva","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0702 · filter · wording 3 · gainbc7c5ddff70a484d7ebf579b8cdce9ee80e03d98e35572ce700d75cb011b3857
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"S394","status":"ready"},{"id":"M936","status":"closed"},{"id":"L704","status":"closed"},{"id":"C562","status":"pending"},{"id":"P679","status":"closed"}]
Checked answer
{"ids": ["S394"]}
Seed

wrong_schema_or_value

{"ids": ["C562", "L704", "M936", "S394", "P679"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["S394"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["S394"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["S394"]}
11 tokens
0703 · total · wording 3 · samee44fb048568d90cadec68d81c8c44e63e4831d7cc757eceedf89f25249b8730c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"F737","units":11,"price":13},{"id":"W625","units":9,"price":2},{"id":"T573","units":17,"price":3},{"id":"H894","units":11,"price":11},{"id":"I942","units":14,"price":8}]
Checked answer
{"total": 445}
Seed

invalid_json_object

{
  "total": 13 + 2 + 3 + 11 + 8 = 31
}
28 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0704 · sort · wording 3 · sameab7ca12f0321df45bbdfed079e2be7f1b606d48a5e9132d731662c0adc096188
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"Z872","priority":7},{"id":"M596","priority":5},{"id":"A963","priority":9},{"id":"G378","priority":6},{"id":"P687","priority":8}]
Checked answer
{"ids": ["A963", "P687"]}
Seed

invalid_json_object

[{"id":"Z872","priority":7},{"id":"M596","priority":5},{"id":"A963","priority":9},{"id":"G378","priority":6},{"id":"P687","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A963","P687"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A963","P687"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["A963","Z872"]}
16 tokens
0705 · lookup · wording 0 · samea783e4b7804b68c0f8e3680ab383f9c1cf2c5172d046d6a9413bf42fa743df30
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record F114. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"B104","city":"Perth","units":1},{"id":"L711","city":"Perth","units":7},{"id":"W928","city":"Accra","units":6},{"id":"F114","city":"Kyoto","units":4},{"id":"F263","city":"Riga","units":12}]
Checked answer
{"city": "Kyoto", "units": 4}
Seed

Correct

{
  "city": "Kyoto",
  "units": 4
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":4}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":4}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":4}
12 tokens
0706 · filter · wording 0 · gaind61d8ef9a69795a3757c136056259b20eccc10f540f02c575f7227c56217118e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L238","status":"ready"},{"id":"E257","status":"pending"},{"id":"F501","status":"closed"},{"id":"D522","status":"pending"},{"id":"B746","status":"pending"}]
Checked answer
{"ids": ["L238"]}
Seed

wrong_schema_or_value

{"ids": ["E257", "F501", "D522", "L238"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["L238"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["L238"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["L238"]}
11 tokens
0707 · total · wording 0 · same973235a0acf76e220405026729527cff7784b4d3d36b577d97b27f69db00196e
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C279","units":18,"price":15},{"id":"Q998","units":15,"price":5},{"id":"R775","units":5,"price":17},{"id":"W714","units":3,"price":17},{"id":"O259","units":5,"price":3}]
Checked answer
{"total": 496}
Seed

wrong_schema_or_value

{"total": 150}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0708 · sort · wording 0 · gaind2ebd02463060158e936dfbfc881bf09e61b99ea54bc83c3ca6b6cb4b5a3cec1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E142","priority":9},{"id":"I869","priority":8},{"id":"I282","priority":6},{"id":"I720","priority":7},{"id":"B842","priority":2}]
Checked answer
{"ids": ["E142", "I869"]}
Seed

wrong_schema_or_value

{"ids": ["I869", "I282", "I720", "E142"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["E142","I869"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E142","I869"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E142","I869"]}
16 tokens
0709 · lookup · wording 1 · samed9884abe780d830b335061588a86aae2aabddb4bd03ef19e6e037077deedf23d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id E880 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M896","city":"Suva","units":3},{"id":"Q807","city":"Lima","units":1},{"id":"N929","city":"Riga","units":4},{"id":"E880","city":"Suva","units":2},{"id":"I973","city":"Bern","units":0}]
Checked answer
{"city": "Suva", "units": 2}
Seed

Correct

{"city":"Suva","units":2}
11 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":2}
11 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":2}
11 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":2}
11 tokens
0710 · filter · wording 1 · gain4cf382574451411f0ad7a985369bb1fbfaf58ee9cc2f93302f7ef1ca72eee96a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"U411","status":"closed"},{"id":"E626","status":"closed"},{"id":"Y475","status":"pending"},{"id":"L652","status":"ready"},{"id":"A220","status":"ready"}]
Checked answer
{"ids": ["A220", "L652"]}
Seed

wrong_schema_or_value

{
  "ids": ["A220", "E626", "L652", "U411"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["A220","L652"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A220","L652"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A220","L652"]}
16 tokens
0711 · total · wording 1 · samea5772b133a1ded6c7ebcc1ca48bbe3d399df2e9b08311f874545c9ff1481ccfb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"H581","units":11,"price":11},{"id":"A916","units":1,"price":1},{"id":"Z638","units":11,"price":15},{"id":"C998","units":2,"price":9},{"id":"B164","units":19,"price":17}]
Checked answer
{"total": 628}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0712 · sort · wording 1 · gainf7d8f5e0452bba2be7ae39726ced8a821f01d43c24e2103cf13b8bfb74c7fee2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"J443","priority":6},{"id":"L164","priority":9},{"id":"A347","priority":3},{"id":"A894","priority":7},{"id":"D868","priority":9}]
Checked answer
{"ids": ["D868", "L164"]}
Seed

invalid_json_object

[{"id":"L164","priority":9},{"id":"A347","priority":3},{"id":"A894","priority":7},{"id":"J443","priority":6}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["D868","L164"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D868","L164"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D868","L164"]}
16 tokens
0713 · lookup · wording 2 · gain4f0f72d5b82f07fd8d01a7097b6cdd6e2e7fa6d2e568d57c4cd4454cbbf29b8d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to S838? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"B491","city":"Bern","units":14},{"id":"S838","city":"Suva","units":19},{"id":"Z543","city":"Lima","units":11},{"id":"Y103","city":"Lima","units":0},{"id":"V110","city":"Suva","units":6}]
Checked answer
{"city": "Suva", "units": 19}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":19}
12 tokens
0714 · filter · wording 2 · samefe5ad55bb86b5bbf08958c77756c301af05132a4c847e76d09273e4310759705
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"J590","status":"closed"},{"id":"L759","status":"pending"},{"id":"O860","status":"ready"},{"id":"G182","status":"pending"},{"id":"N407","status":"ready"}]
Checked answer
{"ids": ["N407", "O860"]}
Seed

invalid_json_object

[{"id":"O860","status":"ready"},{"id":"G182","status":"pending"},{"id":"N407","status":"ready"},{"id":"J590","status":"closed"},{"id":"L759","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["N407","O860"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N407","O860"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["O860","N407"]]
16 tokens
0715 · total · wording 2 · same44ccbb7c0eecae1b34d7b3ed92dcdf07d577f6a674687ae9f3bc461aa5014d85
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y278","units":17,"price":11},{"id":"S560","units":14,"price":14},{"id":"D659","units":12,"price":13},{"id":"I391","units":9,"price":8},{"id":"F534","units":13,"price":5}]
Checked answer
{"total": 676}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0716 · sort · wording 2 · same1b0bed0c897a54a2e1791ec4686b92aed264a9a52886c13fef03e9216fe6b0fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Z798","priority":7},{"id":"S388","priority":8},{"id":"K517","priority":1},{"id":"O855","priority":2},{"id":"O578","priority":7}]
Checked answer
{"ids": ["S388", "O578"]}
Seed

wrong_schema_or_value

{"ids": ["O855", "O578", "K517", "S388", "Z798"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["S388","Z798"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["S388","Z798"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S388","Z798"]}
16 tokens
0717 · lookup · wording 3 · gain5a00a6c91aef9199f0c06190978390789201b6e73d78cc965b86d8b6ccc71592
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=T638. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"T554","city":"Suva","units":10},{"id":"T638","city":"Perth","units":3},{"id":"X343","city":"Suva","units":6},{"id":"V141","city":"Bern","units":12},{"id":"O222","city":"Suva","units":8}]
Checked answer
{"city": "Perth", "units": 3}
Seed

wrong_schema_or_value

{"city":"Suva","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":3}
11 tokens
0718 · filter · wording 3 · gain1f734b9350fee98fa902320b0c71e83af3a85504cc7f00469d6a4f2aa724dca8
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"G384","status":"ready"},{"id":"C471","status":"closed"},{"id":"D204","status":"ready"},{"id":"U425","status":"closed"},{"id":"C956","status":"closed"}]
Checked answer
{"ids": ["D204", "G384"]}
Seed

wrong_schema_or_value

{"ids": ["C471", "C956", "D204", "G384", "U425"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D204","G384"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D204","G384"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D204","G384"]}
16 tokens
0719 · total · wording 3 · same189f628352edff7a75e7feacf490ee7fecd1e07845dc5ef9cf9f64acc682ebd9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"E826","units":5,"price":10},{"id":"S842","units":4,"price":7},{"id":"Z329","units":10,"price":2},{"id":"X150","units":1,"price":8},{"id":"W141","units":16,"price":5}]
Checked answer
{"total": 186}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":288}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0720 · sort · wording 3 · same3924f700431304d7d753a7c53907d483b7f622f01c71c346353bfe7e6894d8be
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"C933","priority":3},{"id":"G488","priority":5},{"id":"D602","priority":7},{"id":"Z877","priority":6},{"id":"U289","priority":4}]
Checked answer
{"ids": ["D602", "Z877"]}
Seed

invalid_json_object

[{"id":"G488","priority":5},{"id":"U289","priority":4},{"id":"C933","priority":3},{"id":"D602","priority":7},{"id":"Z877","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["D602","Z877"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D602","G488"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D602","C933"]}
16 tokens
0721 · lookup · wording 0 · gain592a0552e66e5e3ea372ae7788f2860f1b42d59f915bd1c9bc6f0a36f1f79a35
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y434","city":"Oslo","units":2},{"id":"S806","city":"Kyoto","units":14},{"id":"I427","city":"Suva","units":15},{"id":"D177","city":"Riga","units":7},{"id":"K914","city":"Bern","units":12}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Oslo",
  "units": 2
}
18 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0722 · filter · wording 0 · gain606714e9da3ba53d00552eb24b907f3d2f338be20bc1a42d7525a7fba81a749f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Y478","status":"closed"},{"id":"P760","status":"pending"},{"id":"P323","status":"pending"},{"id":"G864","status":"pending"},{"id":"P830","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["P323", "P760", "P830", "Y478"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0723 · total · wording 0 · same152f2ad9ea6040a554b67dd190bdaac94fe5f6fd546e508ac897948c779c1d00
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"U287","units":17,"price":9},{"id":"X699","units":0,"price":5},{"id":"F543","units":17,"price":6},{"id":"D965","units":15,"price":19},{"id":"H133","units":4,"price":3}]
Checked answer
{"total": 552}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":548}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0724 · sort · wording 0 · same1594e6822b2e2e35474fa46519e1ae0e5add33d26ef37605ca62d814c6ba8931
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U653","priority":6},{"id":"A629","priority":1},{"id":"U170","priority":1},{"id":"L630","priority":3},{"id":"U581","priority":4}]
Checked answer
{"ids": ["U653", "U581"]}
Seed

wrong_schema_or_value

{"ids": ["U653", "A629", "U170", "L630", "U581"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["U653","L630"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["U653","L630"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["U653","A629"]}
16 tokens
0725 · lookup · wording 1 · gain25ce7f6f5ce75204915d7ff9c77060fc3a569f4d20506a541b7a16e5179da88a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id K216 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"U601","city":"Kyoto","units":4},{"id":"Z984","city":"Riga","units":5},{"id":"B496","city":"Lima","units":12},{"id":"W403","city":"Riga","units":14},{"id":"K216","city":"Riga","units":14}]
Checked answer
{"city": "Riga", "units": 14}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":4}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":14}
12 tokens
0726 · filter · wording 1 · gain1adb1c4045000d2110b5875f24808a7db4ec0113ec4bbfb6745567ff707d40a8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"O961","status":"ready"},{"id":"K352","status":"pending"},{"id":"R482","status":"pending"},{"id":"T878","status":"pending"},{"id":"V128","status":"closed"}]
Checked answer
{"ids": ["O961"]}
Seed

wrong_schema_or_value

{
  "ids": ["O961", "K352", "R482", "T878", "V128"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["O961"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["O961"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["O961"]}
11 tokens
0727 · total · wording 1 · same488846cd8f1207a29e4fa6fbba70c2d88cf9227ed24186cf107d12d3f1549dce
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"X589","units":8,"price":8},{"id":"S345","units":18,"price":4},{"id":"Y771","units":15,"price":10},{"id":"Z446","units":13,"price":17},{"id":"V956","units":4,"price":1}]
Checked answer
{"total": 511}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0728 · sort · wording 1 · gaind18438d9357ec37a6a8e05e4582de0a8a514a42ccaf6481ffbb2977a20d264b0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"W538","priority":3},{"id":"S553","priority":1},{"id":"G580","priority":8},{"id":"I866","priority":2},{"id":"D557","priority":4}]
Checked answer
{"ids": ["G580", "D557"]}
Seed

invalid_json_object

[{"id":"W538","priority":3},{"id":"S553","priority":1},{"id":"G580","priority":8},{"id":"I866","priority":2},{"id":"D557","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["G580","D557"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["G580","W538"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G580","D557"]}
16 tokens
0729 · lookup · wording 2 · gain5f252e46fc3a874551588c48b2ab792e1c893855b40322ce4be311452489e67b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to K883? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"W814","city":"Riga","units":15},{"id":"O602","city":"Bern","units":4},{"id":"F680","city":"Suva","units":10},{"id":"K883","city":"Kyoto","units":13},{"id":"P838","city":"Riga","units":16}]
Checked answer
{"city": "Kyoto", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":13}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":13}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":13}
13 tokens
0730 · filter · wording 2 · gain1f1919c76c678ad7cdb986a6880f5caefa5834b9ab43197ef38395fb2f7b1d28
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"L366","status":"closed"},{"id":"E286","status":"pending"},{"id":"G796","status":"closed"},{"id":"M651","status":"pending"},{"id":"X935","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"E286"},{"id":"G796"},{"id":"M651"},{"id":"X935"}]
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0731 · total · wording 2 · same27a44163c8a3517739787c0a65800fd77aa34b1492e8d803a614f8148e44b6ca
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"M282","units":19,"price":3},{"id":"I232","units":9,"price":18},{"id":"H540","units":6,"price":10},{"id":"U130","units":18,"price":16},{"id":"P228","units":2,"price":17}]
Checked answer
{"total": 601}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":678}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0732 · sort · wording 2 · gainc3b0787720d0cd4efa67c628fec5a9267f2907162ef4b4dd057db23a5f61481e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Q224","priority":8},{"id":"L907","priority":6},{"id":"O734","priority":6},{"id":"O484","priority":9},{"id":"J685","priority":1}]
Checked answer
{"ids": ["O484", "Q224"]}
Seed

wrong_schema_or_value

{"ids": ["O484", "O734", "J685", "Q224", "L907"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["O484","Q224"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O484","Q224"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["O484","Q224"]}
16 tokens
0733 · lookup · wording 3 · gain9c42ba2073f8d7f8e22a9bab9c554b0bab30c315b192e3afbffb25342d798c37
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=X306. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"W459","city":"Perth","units":11},{"id":"H626","city":"Accra","units":17},{"id":"X306","city":"Accra","units":15},{"id":"G955","city":"Oslo","units":8},{"id":"A700","city":"Lima","units":10}]
Checked answer
{"city": "Accra", "units": 15}
Seed

wrong_schema_or_value

{"city":"Perth","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":15}
12 tokens
0734 · filter · wording 3 · gain1567d08ce0c596844fda4cfc521df2b3c7eda0ff7599c9cac5aa8ced233263e0
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"C209","status":"pending"},{"id":"N969","status":"pending"},{"id":"Z255","status":"closed"},{"id":"T920","status":"ready"},{"id":"P559","status":"ready"}]
Checked answer
{"ids": ["P559", "T920"]}
Seed

wrong_schema_or_value

{"ids": ["C209", "N969", "Z255", "T920", "P559"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P559","T920"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P559","T920"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P559","T920"]}
16 tokens
0735 · total · wording 3 · same9b156f7d807ebb2f89111e84ad9cd90f8e2c47f7a28567ccb6911e2a959748c6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"V148","units":11,"price":7},{"id":"Z615","units":13,"price":11},{"id":"L654","units":14,"price":17},{"id":"D840","units":7,"price":19},{"id":"N293","units":13,"price":18}]
Checked answer
{"total": 825}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0736 · sort · wording 3 · gain1c33c05eaa090eee28498749c72c59b7c48d669509114bfb3977b94355298c69
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"R329","priority":5},{"id":"D655","priority":3},{"id":"M522","priority":1},{"id":"R476","priority":9},{"id":"T222","priority":9}]
Checked answer
{"ids": ["R476", "T222"]}
Seed

invalid_json_object

[{"id":"R329","priority":5},{"id":"D655","priority":3},{"id":"M522","priority":1},{"id":"R476","priority":9},{"id":"T222","priority":9}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["T222","R476"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["R476","T222"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["R476","T222"]}
16 tokens
0737 · lookup · wording 0 · gain965849443f4fade22a64d10cb782b89a66be65a0d4e6a7e77d1b933fb5f04416
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record X754. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Y644","city":"Perth","units":17},{"id":"I689","city":"Accra","units":2},{"id":"L153","city":"Lima","units":3},{"id":"W662","city":"Bern","units":10},{"id":"X754","city":"Perth","units":14}]
Checked answer
{"city": "Perth", "units": 14}
Seed

invalid_json_object

[{"city":"Perth","units":17}]
13 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":14}
12 tokens
0738 · filter · wording 0 · gainba72270c2213dd0bf7594896e4f8f0de487e880ed3541a230c43a8ba156fbe02
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"F401","status":"pending"},{"id":"M668","status":"pending"},{"id":"B303","status":"ready"},{"id":"Z470","status":"pending"},{"id":"L188","status":"pending"}]
Checked answer
{"ids": ["B303"]}
Seed

wrong_schema_or_value

{"ids": ["B303", "L188", "M668", "F401", "Z470"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B303"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["B303"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["B303"]}
11 tokens
0739 · total · wording 0 · samee70bbf677bd8d4acedf58767304d3067a474561aceaef8ea01f787616da135c2
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"L125","units":2,"price":17},{"id":"R204","units":8,"price":9},{"id":"Q585","units":2,"price":18},{"id":"S566","units":15,"price":8},{"id":"O718","units":10,"price":2}]
Checked answer
{"total": 282}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0740 · sort · wording 0 · sameda2a8f80965d95d4edbe303f505c3dccac963d4fbadefdc39a05ae6d2ca077e4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"S164","priority":9},{"id":"J255","priority":3},{"id":"F394","priority":4},{"id":"Q367","priority":9},{"id":"J634","priority":4}]
Checked answer
{"ids": ["Q367", "S164"]}
Seed

wrong_schema_or_value

{"ids": ["J255", "S164", "J634", "Q367"]}
29 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["S164","Q367"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["S164","Q367"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S164","Q367"]}
16 tokens
0741 · lookup · wording 1 · gaine423cab9bc679a719efaab19623e3c07c54b1c0df65e520ba264d28ca2bcbb5e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"T178","city":"Oslo","units":12},{"id":"D872","city":"Bern","units":16},{"id":"P565","city":"Kyoto","units":11},{"id":"Q808","city":"Accra","units":5},{"id":"X446","city":"Oslo","units":1}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":12}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0742 · filter · wording 1 · gain9b1b50ef7b630b174bde57fea0779db8d9b3276559471e105ea026db8fcf9456
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"N298","status":"ready"},{"id":"C576","status":"closed"},{"id":"S378","status":"pending"},{"id":"W331","status":"ready"},{"id":"Z932","status":"ready"}]
Checked answer
{"ids": ["N298", "W331", "Z932"]}
Seed

wrong_schema_or_value

{
  "ids": ["N298", "S378", "W331", "Z932"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["N298","W331","Z932"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["C576","N298","W331","Z932"]}
26 tokens
4 GPUs · local windows

Correct

{"ids":["N298","W331","Z932"]}
21 tokens
0743 · total · wording 1 · same405f3e5408ef73a6199b1a085812453e661a8aacd0e04a224a9ec6a528870ac0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"E189","units":3,"price":6},{"id":"F454","units":3,"price":7},{"id":"Z247","units":5,"price":2},{"id":"I758","units":11,"price":16},{"id":"W764","units":0,"price":8}]
Checked answer
{"total": 225}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":27}
7 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0744 · sort · wording 1 · same559d782bdb6ec3a2171216b810139d0980b9b56362e96be111d2231e05aa9e55
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"I908","priority":3},{"id":"C825","priority":2},{"id":"U667","priority":6},{"id":"J198","priority":8},{"id":"S933","priority":4}]
Checked answer
{"ids": ["J198", "U667"]}
Seed

invalid_json_object

[{"id":"I908","priority":3},{"id":"C825","priority":2},{"id":"J198","priority":8},{"id":"S933","priority":4}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["J198","U667"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J198","U667"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J198","I908"]}
16 tokens
0745 · lookup · wording 2 · gainca15f372e56ba6225ce841f12c524e0c36770ab070b64fde85da8d6a6931ef06
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to M554? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"M200","city":"Kyoto","units":18},{"id":"M554","city":"Kyoto","units":0},{"id":"H183","city":"Perth","units":0},{"id":"W770","city":"Bern","units":6},{"id":"A369","city":"Bern","units":3}]
Checked answer
{"city": "Kyoto", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":0}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":0}
12 tokens
0746 · filter · wording 2 · samefc1cc79added76bc1012905354815d5bb5576cf33097c26bcc2b2109aeeb0989
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S249","status":"closed"},{"id":"Z679","status":"closed"},{"id":"N519","status":"ready"},{"id":"D903","status":"ready"},{"id":"I263","status":"ready"}]
Checked answer
{"ids": ["D903", "I263", "N519"]}
Seed

invalid_json_object

[{"id":"N519"},{"id":"I263"},{"id":"D903"},{"id":"S249"},{"id":"Z679"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["D903","I263","N519"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["D903","I263","N519"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["I263","N519","D903"]]
21 tokens
0747 · total · wording 2 · same3a98c4e27ae5b61210ef68d24b3a3ced0a0a6a5e45a4e3e448f5844b54aaf4f7
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"K511","units":8,"price":4},{"id":"K865","units":3,"price":18},{"id":"P914","units":17,"price":10},{"id":"Q909","units":2,"price":12},{"id":"K647","units":8,"price":18}]
Checked answer
{"total": 424}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0748 · sort · wording 2 · gaincc34e7a76ec9ab0f864d25889302c694a9d3e3c643441b9bb5b14d5a7b5cf0c4
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"I552","priority":4},{"id":"R504","priority":9},{"id":"S676","priority":5},{"id":"V624","priority":9},{"id":"Z417","priority":1}]
Checked answer
{"ids": ["R504", "V624"]}
Seed

invalid_json_object

{"ids": ["I552", "V624", "S676", "Z417", "R504", "I552", "V624", "S676", "Z417", "R504", "I552", "V624", "S676", "Z417", "R504", "I5
96 tokens
1 GPU · AdamW

Correct

{"ids":["R504","V624"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V624","R504"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["R504","V624"]}
16 tokens
0749 · lookup · wording 3 · gainc62626834fec0321558e9a797d03320ec8d28ca2cae86fd30d28c233bf8f687d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=N850. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"X702","city":"Riga","units":14},{"id":"C556","city":"Oslo","units":10},{"id":"N850","city":"Perth","units":13},{"id":"X333","city":"Riga","units":17},{"id":"Z123","city":"Oslo","units":6}]
Checked answer
{"city": "Perth", "units": 13}
Seed

wrong_schema_or_value

{"city":"Riga","units":14}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":13}
12 tokens
0750 · filter · wording 3 · gain3f4b8ca6d98dd923a2d83fdd720084f56e28db51d9c0bc5d23c5cefe04056c36
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"Z720","status":"pending"},{"id":"S780","status":"closed"},{"id":"L719","status":"pending"},{"id":"A990","status":"ready"},{"id":"U714","status":"ready"}]
Checked answer
{"ids": ["A990", "U714"]}
Seed

wrong_schema_or_value

{"ids": ["A990", "L719", "S780", "U714", "Z720"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A990","U714"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A990","U714"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A990","U714"]}
16 tokens
0751 · total · wording 3 · same3c519fdaef18cbec1c8664f4e37622ecd23573b3b45283b0d5e60e2172b8e579
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"A863","units":15,"price":6},{"id":"X847","units":12,"price":17},{"id":"H813","units":5,"price":1},{"id":"I231","units":5,"price":5},{"id":"G444","units":17,"price":19}]
Checked answer
{"total": 647}
Seed

wrong_schema_or_value

{
  "total": 103
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0752 · sort · wording 3 · same7be309a3c7311b17f2034c820dc68d34c8a38aa1182822e2ee3c5f783cf206fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"R356","priority":7},{"id":"S947","priority":5},{"id":"R131","priority":4},{"id":"G228","priority":2},{"id":"M834","priority":7}]
Checked answer
{"ids": ["M834", "R356"]}
Seed

invalid_json_object

[{"id":"R356"},{"id":"S947"},{"id":"R131"},{"id":"G228"},{"id":"M834"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["M834","R356"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M834","R356"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["R356","M834"]}
16 tokens
0753 · lookup · wording 0 · same8e5287bc43bba9fc149863982b696fe990f7164d59763046f19a71766329b788
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record O150. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"C340","city":"Bern","units":0},{"id":"O150","city":"Lima","units":14},{"id":"W741","city":"Suva","units":12},{"id":"M679","city":"Oslo","units":17},{"id":"Z480","city":"Lima","units":4}]
Checked answer
{"city": "Lima", "units": 14}
Seed

Correct

{
  "city": "Lima",
  "units": 14
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":14}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":14}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":14}
12 tokens
0754 · filter · wording 0 · gaindbe50626439ded1a6b00afe812a5232b3ba23e90e1d070ede39cfca912c7ad69
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"H573","status":"closed"},{"id":"C679","status":"closed"},{"id":"J494","status":"ready"},{"id":"V782","status":"ready"},{"id":"M373","status":"ready"}]
Checked answer
{"ids": ["J494", "M373", "V782"]}
Seed

wrong_schema_or_value

{"ids": ["J494", "M373", "V782", "H573", "C679"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J494","M373","V782"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["J494","M373","V782"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["J494","M373","V782"]}
21 tokens
0755 · total · wording 0 · same8390020e3446899fede90b329e9fa0d291807e51613db64a216e138a865fed62
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"M738","units":19,"price":13},{"id":"H777","units":4,"price":14},{"id":"L373","units":3,"price":11},{"id":"M348","units":4,"price":12},{"id":"X470","units":6,"price":7}]
Checked answer
{"total": 426}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0756 · sort · wording 0 · gainaaa5690c7513e07141652b1e9dde20ebd6a08c3a6e560ba1d7c812c4087fb11e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U296","priority":3},{"id":"E694","priority":7},{"id":"M576","priority":7},{"id":"V508","priority":6},{"id":"H419","priority":8}]
Checked answer
{"ids": ["H419", "E694"]}
Seed

wrong_schema_or_value

{"ids": ["U296", "E694", "M576", "V508", "H419"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H419","M576"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H419","M576"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H419","E694"]}
16 tokens
0757 · lookup · wording 1 · gain8739e6ee2731fff85b41b2be11aa204353b2d0dd2b27c5f254a1eb40f0e0b6bb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id X240 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W894","city":"Suva","units":11},{"id":"I861","city":"Suva","units":4},{"id":"Q784","city":"Lima","units":4},{"id":"X240","city":"Oslo","units":9},{"id":"L371","city":"Accra","units":1}]
Checked answer
{"city": "Oslo", "units": 9}
Seed

wrong_schema_or_value

{"city":"Suva","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":9}
11 tokens
0758 · filter · wording 1 · gain666c2de8cb90f0bf7cb522e1cf4a10b7ee901784c2b74bda34f1be0cbd39773d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"G903","status":"ready"},{"id":"U576","status":"closed"},{"id":"O132","status":"ready"},{"id":"O925","status":"pending"},{"id":"X173","status":"pending"}]
Checked answer
{"ids": ["G903", "O132"]}
Seed

wrong_schema_or_value

{
  "ids": ["G903", "O132", "O925", "X173"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["G903","O132"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G903","O132"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G903","O132"]}
16 tokens
0759 · total · wording 1 · samed8d675f89b15dc019f0b5c4d6861816389624f9fb329e70e33d1f89329c32b26
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D148","units":19,"price":13},{"id":"H264","units":1,"price":13},{"id":"J261","units":16,"price":7},{"id":"D968","units":9,"price":14},{"id":"K649","units":18,"price":19}]
Checked answer
{"total": 840}
Seed

wrong_schema_or_value

{"total": 102}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0760 · sort · wording 1 · gain5af5bd4f51168f5e4fac57775407d2da36847331c1d97dbd7797add66e076db3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"M897","priority":6},{"id":"W634","priority":4},{"id":"C619","priority":6},{"id":"L851","priority":7},{"id":"F655","priority":6}]
Checked answer
{"ids": ["L851", "C619"]}
Seed

invalid_json_object

[{"id":"M897","priority":6},{"id":"L851","priority":7}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["L851","C619"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L851","M897"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L851","C619"]}
16 tokens
0761 · lookup · wording 2 · samef933be671c5bddbb940a6bf98fe6e1de4cfb71bb9c75db2b673481a582d1e26d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"B822","city":"Suva","units":8},{"id":"O133","city":"Perth","units":11},{"id":"H520","city":"Accra","units":2},{"id":"X790","city":"Bern","units":13},{"id":"G130","city":"Perth","units":6}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0762 · filter · wording 2 · same8ff54eb6d66b21ea8c2ea4b57b66c82dd0a57eba5a60c64ea76c2b303258dcc7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Z133","status":"pending"},{"id":"Q460","status":"closed"},{"id":"O351","status":"ready"},{"id":"P287","status":"ready"},{"id":"D859","status":"closed"}]
Checked answer
{"ids": ["O351", "P287"]}
Seed

invalid_json_object

[{"id":"O351","status":"ready"},{"id":"P287","status":"ready"},{"id":"Z133","status":"pending"},{"id":"Q460","status":"closed"},{"id":"D859","status":"closed"}]
63 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P287","O351"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O351","P287"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["O351","P287"]]
16 tokens
0763 · total · wording 2 · same42f9fc75d645114e6fd7b5ac1ba78f25b693f15ebf113e0c750fc21df8fe3f52
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"B774","units":6,"price":11},{"id":"S814","units":4,"price":16},{"id":"P735","units":11,"price":8},{"id":"E122","units":3,"price":18},{"id":"K171","units":13,"price":4}]
Checked answer
{"total": 324}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0764 · sort · wording 2 · same3297cd92fb5a503dd8e79bfd2aae95123a6e46cdc0f128531f94f6ef398cac12
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"E815","priority":8},{"id":"Y181","priority":9},{"id":"J987","priority":4},{"id":"P131","priority":9},{"id":"K646","priority":6}]
Checked answer
{"ids": ["P131", "Y181"]}
Seed

wrong_schema_or_value

{"ids": ["E815", "Y181", "J987", "P131", "K646"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["E815","P131"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["E815","P131"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E815","P131"]}
16 tokens
0765 · lookup · wording 3 · samef7181a5beafdd057f9c925bbbaa724c1c5fd0423496e7fec4cf3084119405290
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Y587. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"X443","city":"Oslo","units":8},{"id":"Y587","city":"Riga","units":1},{"id":"R888","city":"Riga","units":5},{"id":"D593","city":"Perth","units":5},{"id":"U942","city":"Kyoto","units":19}]
Checked answer
{"city": "Riga", "units": 1}
Seed

Correct

{"city":"Riga","units":1}
11 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":1}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":1}
11 tokens
0766 · filter · wording 3 · gainc69bbbc31a3a3ffc95c632eb3cc315076d1dda160952e29d0b968f060e274208
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"B705","status":"pending"},{"id":"A714","status":"closed"},{"id":"Y205","status":"pending"},{"id":"T222","status":"closed"},{"id":"L145","status":"ready"}]
Checked answer
{"ids": ["L145"]}
Seed

wrong_schema_or_value

{"ids": ["A714", "B705", "L145", "T222", "Y205"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L145"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["L145"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["L145"]}
11 tokens
0767 · total · wording 3 · same90022cc59fe19a3c37ff8c2f07f09c270680193b0af7f881e7f55c7c19cf6f17
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"F340","units":11,"price":13},{"id":"A439","units":13,"price":18},{"id":"F646","units":15,"price":14},{"id":"A603","units":8,"price":8},{"id":"O712","units":17,"price":5}]
Checked answer
{"total": 736}
Seed

wrong_schema_or_value

{
  "total": 42
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0768 · sort · wording 3 · same11a90876c52dd747ea52e06a3ab05df300edb5c9faa5bbeeeb046f59055dabb9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"J894","priority":2},{"id":"S965","priority":6},{"id":"J763","priority":3},{"id":"V629","priority":8},{"id":"I751","priority":1}]
Checked answer
{"ids": ["V629", "S965"]}
Seed

invalid_json_object

[{"id":"J894"},{"id":"S965"},{"id":"J763"},{"id":"V629"},{"id":"I751"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["V629","S965"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["V629","S965"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S965","V629"]}
16 tokens
0769 · lookup · wording 0 · samec089cfb87bfeefd13bce5ee96f4a59f357c13c1f20522f831149fdd12cb8c66a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record H727. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H727","city":"Bern","units":10},{"id":"B492","city":"Bern","units":9},{"id":"V785","city":"Suva","units":3},{"id":"T608","city":"Oslo","units":15},{"id":"Z444","city":"Oslo","units":12}]
Checked answer
{"city": "Bern", "units": 10}
Seed

Correct

{
  "city": "Bern",
  "units": 10
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":10}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":10}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":10}
11 tokens
0770 · filter · wording 0 · gain191f7d2b3b087773da3e4fb21f6b4e8c2b8613ad0443a33025fb7972af26cfaf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Z803","status":"ready"},{"id":"C998","status":"closed"},{"id":"L706","status":"ready"},{"id":"P334","status":"pending"},{"id":"S679","status":"pending"}]
Checked answer
{"ids": ["L706", "Z803"]}
Seed

wrong_schema_or_value

{"ids": ["C998", "L706", "P334", "Z803"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["L706","Z803"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L706","Z803"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L706","Z803"]}
16 tokens
0771 · total · wording 0 · same69b0231a15fcec3b15358eb845cf392c88b58daa2b9a86bc3d55ef5a39ffbaeb
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"O291","units":19,"price":4},{"id":"G588","units":4,"price":3},{"id":"L539","units":17,"price":17},{"id":"U660","units":12,"price":9},{"id":"A565","units":7,"price":15}]
Checked answer
{"total": 590}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0772 · sort · wording 0 · gain02e673f03c9ba72f3a273b5b6169a530908e4515c610844fe09bc8809d771157
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"O492","priority":3},{"id":"M351","priority":4},{"id":"E674","priority":9},{"id":"R127","priority":2},{"id":"Y922","priority":9}]
Checked answer
{"ids": ["E674", "Y922"]}
Seed

wrong_schema_or_value

{"ids": ["O492", "M351", "R127", "Y922", "E674"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E674","Y922"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E674","Y922"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E674","Y922"]}
16 tokens
0773 · lookup · wording 1 · samed810dc941dd22e383f8a7049074b51a8f5d60b8a1198cae7bc0608a3be59915d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id T280 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D298","city":"Riga","units":15},{"id":"Y687","city":"Perth","units":11},{"id":"T280","city":"Suva","units":12},{"id":"F488","city":"Kyoto","units":13},{"id":"S826","city":"Accra","units":10}]
Checked answer
{"city": "Suva", "units": 12}
Seed

Correct

{"city":"Suva","units":12}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":12}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":12}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":12}
12 tokens
0774 · filter · wording 1 · same23529b39f6abfa87e58790f7c1d0e31410dca059185819f27ae8b1eecdcbf361
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"A716","status":"ready"},{"id":"S991","status":"ready"},{"id":"U866","status":"ready"},{"id":"C837","status":"pending"},{"id":"Q407","status":"ready"}]
Checked answer
{"ids": ["A716", "Q407", "S991", "U866"]}
Seed

wrong_schema_or_value

{
  "ids": ["A716", "S991", "U866", "C837", "Q407"]
}
38 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["A716","S991","Q407"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["A716","S991","Q407","U866"]}
26 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["A716","S991","Q407"]}
21 tokens
0775 · total · wording 1 · same0c1b7b128f393505fd5075ec6388a49ddf47f7088f7d9f716c825d6d188c3fb6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L298","units":19,"price":1},{"id":"N730","units":6,"price":3},{"id":"P149","units":16,"price":2},{"id":"Z361","units":5,"price":6},{"id":"O735","units":16,"price":16}]
Checked answer
{"total": 355}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0776 · sort · wording 1 · gain8fb11165030d6b04d6070d5b3e08b3730d8404601a1421170ef0ab9b2485e270
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"A315","priority":1},{"id":"P758","priority":3},{"id":"Y137","priority":7},{"id":"M883","priority":9},{"id":"Y312","priority":1}]
Checked answer
{"ids": ["M883", "Y137"]}
Seed

invalid_json_object

[{"id":"A315","priority":1},{"id":"P758","priority":3},{"id":"Y137","priority":7},{"id":"M883","priority":9},{"id":"Y312","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["M883","Y137"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M883","Y137"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M883","Y137"]}
16 tokens
0777 · lookup · wording 2 · gain526960867d348febb39d8ece19a19a4315f6cc101f72df62034aa8904ce017af
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to C572? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"C572","city":"Oslo","units":18},{"id":"R228","city":"Suva","units":11},{"id":"I870","city":"Bern","units":7},{"id":"C914","city":"Suva","units":9},{"id":"I788","city":"Lima","units":12}]
Checked answer
{"city": "Oslo", "units": 18}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":18}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":18}
12 tokens
0778 · filter · wording 2 · sameaf69d500d51656765232566348563c2bfee79b4674da85e8ba6c517fd8ab0194
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"K503","status":"ready"},{"id":"S810","status":"pending"},{"id":"B319","status":"pending"},{"id":"J940","status":"pending"},{"id":"X213","status":"pending"}]
Checked answer
{"ids": ["K503"]}
Seed

invalid_json_object

[{"id":"K503"},{"id":"S810"},{"id":"B319"},{"id":"J940"},{"id":"X213"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["K503"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["K503"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["K503"]]
11 tokens
0779 · total · wording 2 · samed9316c885244d75dee040074deef2e10e31586f883fe6596c37c473004e1d6db
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"D878","units":0,"price":3},{"id":"I154","units":9,"price":16},{"id":"Q186","units":17,"price":14},{"id":"X373","units":2,"price":17},{"id":"D758","units":14,"price":3}]
Checked answer
{"total": 458}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0780 · sort · wording 2 · gain770832c945bc93ebdc54ae12e687aed6dcadcf7160fedf653752174ed7beb203
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"B233","priority":2},{"id":"Q107","priority":5},{"id":"P854","priority":2},{"id":"D759","priority":6},{"id":"R782","priority":8}]
Checked answer
{"ids": ["R782", "D759"]}
Seed

wrong_schema_or_value

{"ids": ["B233", "P854", "R782", "D759", "Q107", "P854"]}
41 tokens
1 GPU · AdamW

Correct

{"ids":["R782","D759"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["R782","D759"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["R782","D759"]}
16 tokens
0781 · lookup · wording 3 · gain101b1c0a596c1b1e92083982ed9c97c6c34771e966c589c3c451acc086b90205
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"U758","city":"Kyoto","units":19},{"id":"A902","city":"Kyoto","units":13},{"id":"E344","city":"Oslo","units":2},{"id":"S130","city":"Kyoto","units":13},{"id":"F778","city":"Lima","units":11}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":19}
13 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0782 · filter · wording 3 · gain199989e4fd1646882e974e2816ef89b8d500749bf7fa5acfd0b833bcb070fe37
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"H312","status":"closed"},{"id":"E304","status":"ready"},{"id":"A954","status":"closed"},{"id":"R285","status":"closed"},{"id":"T455","status":"ready"}]
Checked answer
{"ids": ["E304", "T455"]}
Seed

wrong_schema_or_value

{"ids": ["A954", "E304", "H312", "R285", "T455"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E304","T455"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E304","T455"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E304","T455"]}
16 tokens
0783 · total · wording 3 · samed1baba23c0a05a00f5d43ca5bcd2419e09a41f1e2f6e61f64d8daff7978281f1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"M450","units":15,"price":11},{"id":"J327","units":9,"price":14},{"id":"U532","units":4,"price":13},{"id":"A731","units":6,"price":11},{"id":"Q499","units":9,"price":6}]
Checked answer
{"total": 463}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0784 · sort · wording 3 · gain5bb43d17887aa243af9c0c3325fab65ea9594cf90cf7c9e0253a793c8e797c48
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"I230","priority":6},{"id":"Y409","priority":9},{"id":"I395","priority":5},{"id":"N273","priority":1},{"id":"J238","priority":9}]
Checked answer
{"ids": ["J238", "Y409"]}
Seed

invalid_json_object

[{"id":"I230","priority":6},{"id":"Y409","priority":9},{"id":"I395","priority":5},{"id":"N273","priority":1},{"id":"J238","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["J238","Y409"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J238","Y409"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J238","Y409"]}
16 tokens
0785 · lookup · wording 0 · same85ba42ca67f1cfad238df38a21fc635536730a2cbc5401743dcd8e5ff030b573
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record X915. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W472","city":"Bern","units":11},{"id":"N505","city":"Perth","units":11},{"id":"M366","city":"Accra","units":11},{"id":"X915","city":"Kyoto","units":10},{"id":"T854","city":"Accra","units":3}]
Checked answer
{"city": "Kyoto", "units": 10}
Seed

Correct

{
  "city": "Kyoto",
  "units": 10
}
20 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":10}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":10}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":10}
13 tokens
0786 · filter · wording 0 · gain1cceacb58818eea598a2eb7c88969daf4f3e71f61622c07307cd1682654dc59a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"D563","status":"pending"},{"id":"F812","status":"pending"},{"id":"G793","status":"closed"},{"id":"S237","status":"closed"},{"id":"L717","status":"closed"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["D563", "F812", "G793", "S237", "L717"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0787 · total · wording 0 · same051f1601a9867b946e556e5ac17d8dae8fdcf5721a2d8d92185be279d6c8ef87
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"L823","units":16,"price":12},{"id":"V115","units":6,"price":17},{"id":"T802","units":1,"price":16},{"id":"U184","units":4,"price":18},{"id":"S720","units":5,"price":7}]
Checked answer
{"total": 417}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0788 · sort · wording 0 · gaine2e02f93b345bcd5d4edec624027edd25df7fb6f8d839794cfdc826488ace4e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"F809","priority":7},{"id":"F174","priority":5},{"id":"I438","priority":1},{"id":"Z858","priority":7},{"id":"L232","priority":5}]
Checked answer
{"ids": ["F809", "Z858"]}
Seed

wrong_schema_or_value

{"ids": ["F809", "I438", "Z858", "L232"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["F809","Z858"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F809","Z858"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F809","Z858"]}
16 tokens
0789 · lookup · wording 1 · gain5c6073a09995bd46b19dbf8c66bc27f9328e70b15e00c3038cdd0bf4717ded20
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id N110 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D499","city":"Kyoto","units":13},{"id":"G593","city":"Kyoto","units":0},{"id":"N110","city":"Kyoto","units":10},{"id":"F459","city":"Lima","units":18},{"id":"E150","city":"Suva","units":14}]
Checked answer
{"city": "Kyoto", "units": 10}
Seed

wrong_schema_or_value

{"city":"Kyoto","units":13}
13 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":10}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":10}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":10}
13 tokens
0790 · filter · wording 1 · gainf9b071e526c988cb24eb85f3435963b112faad6cbd8656308b635fadaa24bdc0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"P666","status":"closed"},{"id":"K751","status":"ready"},{"id":"A995","status":"pending"},{"id":"W190","status":"closed"},{"id":"Q852","status":"pending"}]
Checked answer
{"ids": ["K751"]}
Seed

wrong_schema_or_value

{
  "ids": ["A995", "K751", "Q852", "P666", "W190"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["K751"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["K751"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["K751"]}
11 tokens
0791 · total · wording 1 · same4b7b05cdaed5d705fcf64ef2e7da6f66f2a4e63f617320cfaa25849e9c8c497a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"S977","units":0,"price":16},{"id":"A382","units":6,"price":5},{"id":"E696","units":2,"price":7},{"id":"N651","units":1,"price":1},{"id":"N528","units":4,"price":6}]
Checked answer
{"total": 69}
Seed

wrong_schema_or_value

{"total": 23}
8 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":17}
7 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":17}
7 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0792 · sort · wording 1 · sameebc71b6a97ace60f27dc8f4267bf487a3eb8e2185ff83ac6ac12105376349806
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"Y527","priority":1},{"id":"G989","priority":7},{"id":"Y689","priority":4},{"id":"Z587","priority":6},{"id":"C780","priority":4}]
Checked answer
{"ids": ["G989", "Z587"]}
Seed

invalid_json_object

[{"id":"Y527","priority":1},{"id":"G989","priority":7},{"id":"Y689","priority":4},{"id":"Z587","priority":6},{"id":"C780","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["G989","Z587"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["G989","C780"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G989","C780"]}
16 tokens
0793 · lookup · wording 2 · gain31b4867ac92840da29cddad88f972b4158c3c818701c02ae719078bf58c5a6e1
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to B857? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"J909","city":"Riga","units":5},{"id":"S177","city":"Riga","units":12},{"id":"P917","city":"Accra","units":10},{"id":"J573","city":"Kyoto","units":2},{"id":"B857","city":"Kyoto","units":14}]
Checked answer
{"city": "Kyoto", "units": 14}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":14}
13 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":14}
11 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":14}
13 tokens
0794 · filter · wording 2 · sameb7b314cbeec6405e1a5e0d9dc1997ea94c1b13f86973f7fa028f49af8ad965e2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"E480","status":"pending"},{"id":"K747","status":"ready"},{"id":"G552","status":"closed"},{"id":"L582","status":"pending"},{"id":"I383","status":"pending"}]
Checked answer
{"ids": ["K747"]}
Seed

invalid_json_object

[{"id":"K747","status":"ready"},{"id":"L582","status":"pending"},{"id":"I383","status":"pending"},{"id":"E480","status":"pending"},{"id":"G552","status":"closed"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["K747"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["K747"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["K747"]]
11 tokens
0795 · total · wording 2 · same60fb9dc8230ca6bd9d9643fd3025fc9e4325b8a3a3dfbde117b3718133c9d223
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"V228","units":5,"price":5},{"id":"A468","units":16,"price":7},{"id":"L181","units":17,"price":8},{"id":"Y501","units":16,"price":19},{"id":"R450","units":12,"price":14}]
Checked answer
{"total": 745}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0796 · sort · wording 2 · gain9339c72fc79a3ca43a988aa1ce0ebc29cf1e4e2d1a1aa80876276e9e810f9b35
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"W554","priority":8},{"id":"H979","priority":5},{"id":"H123","priority":9},{"id":"V946","priority":8},{"id":"U819","priority":7}]
Checked answer
{"ids": ["H123", "V946"]}
Seed

wrong_schema_or_value

{"ids": ["H123", "V946", "U819", "W554", "H979"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H123","W554"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H123","W554"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H123","V946"]}
16 tokens
0797 · lookup · wording 3 · samef21664061e3d7eef8c4fa5f85878cc1a3af2af360e8f84cb4ad987d8f41b022c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=P652. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"M833","city":"Bern","units":6},{"id":"A556","city":"Bern","units":18},{"id":"I202","city":"Riga","units":6},{"id":"U592","city":"Riga","units":11},{"id":"P652","city":"Kyoto","units":18}]
Checked answer
{"city": "Kyoto", "units": 18}
Seed

Correct

{"city":"Kyoto","units":18}
13 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":18}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":18}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":18}
13 tokens
0798 · filter · wording 3 · gaina8bb3099b3f89393d2752276063489adfe9d947fcd1afb618f1224c967d0e67c
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"J487","status":"closed"},{"id":"C965","status":"pending"},{"id":"V130","status":"ready"},{"id":"I772","status":"pending"},{"id":"C448","status":"closed"}]
Checked answer
{"ids": ["V130"]}
Seed

wrong_schema_or_value

{"ids": ["C448", "C965", "J487", "I772", "V130"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["V130"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["V130"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["V130"]}
11 tokens
0799 · total · wording 3 · same428f634a0cd2dc3fbac0752d5d216c10fa9a0efeaff8c54a1e4f1cbc1d0ee8ab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"K568","units":11,"price":8},{"id":"L565","units":8,"price":13},{"id":"E524","units":18,"price":11},{"id":"L657","units":6,"price":13},{"id":"C442","units":2,"price":2}]
Checked answer
{"total": 472}
Seed

wrong_schema_or_value

{
  "total": 120
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0800 · sort · wording 3 · sameef8d7b8ca886efe79fbef793d7f6d8105a24debbc3f7a20571c9c7fccf37b45d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"N603","priority":5},{"id":"U619","priority":9},{"id":"H292","priority":6},{"id":"V566","priority":2},{"id":"Y701","priority":9}]
Checked answer
{"ids": ["U619", "Y701"]}
Seed

invalid_json_object

[{"id":"N603","priority":5},{"id":"U619","priority":9},{"id":"H292","priority":6},{"id":"V566","priority":2},{"id":"Y701","priority":9}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Y701","H292"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H292","Y701"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["H292","N603"]}
16 tokens
0801 · lookup · wording 0 · gain48396b1ab0cf30507dbfa91708b1ff4b0e07a273cbe5be9b6bd414753901bd0e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"K966","city":"Perth","units":8},{"id":"A684","city":"Suva","units":6},{"id":"N901","city":"Oslo","units":6},{"id":"I342","city":"Lima","units":6},{"id":"C448","city":"Riga","units":19}]
Checked answer
{"city": null, "units": null}
Seed

invalid_json_object

[{"city":"Perth","units":8},{"city":"Suva","units":6},{"city":"Oslo","units":6},{"city":"Lima","units":6},{"city":"Riga","units":19}]
53 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0802 · filter · wording 0 · gain4025264315175ebce5606478a9ae0c12df2951926c2af190bb275972c02e04b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"B111","status":"ready"},{"id":"E565","status":"pending"},{"id":"Y860","status":"closed"},{"id":"J886","status":"ready"},{"id":"Q823","status":"closed"}]
Checked answer
{"ids": ["B111", "J886"]}
Seed

wrong_schema_or_value

{"ids": ["B111", "E565", "J886", "Q823"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["B111","J886"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B111","J886"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B111","J886"]}
16 tokens
0803 · total · wording 0 · same25e0742d1bf7268ac92bbdc29dee6c1fd60dd927d650e599367a5144af5e2f7e
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"F302","units":7,"price":9},{"id":"C865","units":0,"price":5},{"id":"A290","units":8,"price":4},{"id":"U303","units":18,"price":4},{"id":"T376","units":8,"price":13}]
Checked answer
{"total": 271}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0804 · sort · wording 0 · gain8a5eaa88ea057c21166f2d7a1e99f00fbed4e27d0704bedb2acada5f32f0348f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"K636","priority":7},{"id":"V156","priority":7},{"id":"G556","priority":9},{"id":"E687","priority":2},{"id":"K197","priority":8}]
Checked answer
{"ids": ["G556", "K197"]}
Seed

wrong_schema_or_value

{"ids": ["K636", "V156", "G556", "K197", "E687"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["G556","K197"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["G556","K197"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["G556","K197"]}
16 tokens
0805 · lookup · wording 1 · samed48c5a8e58b966b1d1139618ec739d19195414c1c2b3237c9838890f108b8413
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id W365 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W365","city":"Perth","units":17},{"id":"F422","city":"Oslo","units":17},{"id":"L213","city":"Oslo","units":11},{"id":"F413","city":"Accra","units":3},{"id":"R959","city":"Perth","units":11}]
Checked answer
{"city": "Perth", "units": 17}
Seed

Correct

{"city":"Perth","units":17}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":17}
12 tokens
0806 · filter · wording 1 · gain2811af892ae42c3565efffe1a451d256ceea57110c6b800580afb4b9c559b80e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M761","status":"closed"},{"id":"N530","status":"closed"},{"id":"A928","status":"pending"},{"id":"M293","status":"closed"},{"id":"E949","status":"ready"}]
Checked answer
{"ids": ["E949"]}
Seed

wrong_schema_or_value

{
  "ids": ["A928", "E949", "M293", "M761", "N530"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["E949"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["E949"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["E949"]}
11 tokens
0807 · total · wording 1 · same9fbd9d11e692b11d2f15ecc5f34554f8c6cf58b7a628b82166245c15e5b97ce6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"H707","units":5,"price":7},{"id":"V885","units":11,"price":11},{"id":"Z902","units":11,"price":19},{"id":"Q230","units":1,"price":6},{"id":"I955","units":4,"price":2}]
Checked answer
{"total": 379}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":378}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0808 · sort · wording 1 · gain77fa984f91b44742b93caef01c43519b0ef82e764d99942f07f711710f853b39
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"D674","priority":8},{"id":"H976","priority":9},{"id":"Y931","priority":7},{"id":"K669","priority":2},{"id":"N886","priority":3}]
Checked answer
{"ids": ["H976", "D674"]}
Seed

invalid_json_object

[{"id":"D674","priority":8},{"id":"H976","priority":9},{"id":"Y931","priority":7},{"id":"K669","priority":2},{"id":"N886","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["H976","D674"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H976","D674"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H976","D674"]}
16 tokens
0809 · lookup · wording 2 · gaind6e05fcbf3e502c4ebad4a9a6419f881576b4f5b356488a855f37de0a358fb0b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to B528? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"K805","city":"Suva","units":18},{"id":"B528","city":"Lima","units":1},{"id":"T938","city":"Bern","units":2},{"id":"X990","city":"Kyoto","units":14},{"id":"B920","city":"Oslo","units":9}]
Checked answer
{"city": "Lima", "units": 1}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":1}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":1}
11 tokens
0810 · filter · wording 2 · samee6e2ca2ecb53ad29fc9c0f1e9679fc8e7ceeeb7b3472f8f3c01f1ac9a1f859c3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"B143","status":"pending"},{"id":"D734","status":"ready"},{"id":"N962","status":"closed"},{"id":"D890","status":"pending"},{"id":"F209","status":"ready"}]
Checked answer
{"ids": ["D734", "F209"]}
Seed

invalid_json_object

[{"id":"D890","status":"pending"},{"id":"D734","status":"ready"},{"id":"F209","status":"ready"},{"id":"N962","status":"closed"},{"id":"B143","status":"pending"}]
63 tokens
1 GPU · AdamW

invalid_json_object

["D734","F209"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["D734","F209"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["D734","F209"]]
16 tokens
0811 · total · wording 2 · same0aa46504ab7f5391f58d8f0ed70cb0e421299cec2171aaab463e2fbcae3e66e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"Y722","units":15,"price":19},{"id":"R695","units":16,"price":5},{"id":"F660","units":8,"price":9},{"id":"R654","units":8,"price":9},{"id":"Y265","units":10,"price":15}]
Checked answer
{"total": 659}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0812 · sort · wording 2 · gainb389cd561998ce4e42432e15b11109b091982d75223a44b859119200dbb41c85
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"U622","priority":1},{"id":"F508","priority":2},{"id":"P119","priority":5},{"id":"T875","priority":6},{"id":"B540","priority":1}]
Checked answer
{"ids": ["T875", "P119"]}
Seed

wrong_schema_or_value

{"ids": ["U622", "F508", "P119", "T875", "B540"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P119","T875"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P119","T875"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["T875","P119"]}
16 tokens
0813 · lookup · wording 3 · gainc3af71fd6cffa9bbeb0020c01f5f977eaed5836d42f11adc9f0ffb1ab796cae7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Y316. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"H153","city":"Bern","units":19},{"id":"D604","city":"Accra","units":16},{"id":"Y316","city":"Accra","units":18},{"id":"Q261","city":"Bern","units":4},{"id":"Z387","city":"Bern","units":1}]
Checked answer
{"city": "Accra", "units": 18}
Seed

wrong_schema_or_value

{"city":"Bern","units":19}
11 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":18}
12 tokens
0814 · filter · wording 3 · gaina35cbf16a5e23f5e208078e839807c3b683e08e049d686960d00130d836c2286
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"R573","status":"closed"},{"id":"D891","status":"pending"},{"id":"R932","status":"pending"},{"id":"X490","status":"ready"},{"id":"H975","status":"closed"}]
Checked answer
{"ids": ["X490"]}
Seed

wrong_schema_or_value

{"ids": ["X490", "H975", "R573", "D891", "R932"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["X490"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["X490"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["X490"]}
11 tokens
0815 · total · wording 3 · same7a439df1798881c6aedfcfb05d1c50eb55fd332f6cf173d41776e6446e139e7d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"O498","units":13,"price":15},{"id":"S285","units":19,"price":1},{"id":"K520","units":1,"price":6},{"id":"G443","units":9,"price":4},{"id":"C654","units":15,"price":11}]
Checked answer
{"total": 421}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0816 · sort · wording 3 · gainbac78517f48e2203e6eb8c531428974c26a2db4ab5f372c0856eb1dd2cb6b4c5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"G391","priority":2},{"id":"S497","priority":7},{"id":"I917","priority":4},{"id":"K603","priority":9},{"id":"S806","priority":2}]
Checked answer
{"ids": ["K603", "S497"]}
Seed

invalid_json_object

[{"id":"S806","priority":2},{"id":"S497","priority":7},{"id":"G391","priority":2},{"id":"I917","priority":4},{"id":"K603","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["K603","S497"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K603","S497"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K603","S497"]}
16 tokens
0817 · lookup · wording 0 · same63a9692a4dd913b6492c302589a087415864e8abf22f70a711797bfd3a938a63
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record N376. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"C363","city":"Kyoto","units":2},{"id":"X384","city":"Bern","units":16},{"id":"C580","city":"Suva","units":16},{"id":"L325","city":"Lima","units":19},{"id":"N376","city":"Lima","units":3}]
Checked answer
{"city": "Lima", "units": 3}
Seed

Correct

{
  "city": "Lima",
  "units": 3
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":3}
11 tokens
0818 · filter · wording 0 · gaince255bbb42fbc0e7000a11a78b1c48b08a8cd0550b96ad236534b79f824258e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"F175","status":"closed"},{"id":"B281","status":"ready"},{"id":"C330","status":"closed"},{"id":"O323","status":"ready"},{"id":"P613","status":"closed"}]
Checked answer
{"ids": ["B281", "O323"]}
Seed

wrong_schema_or_value

{"ids": ["B281", "C330", "O323", "P613"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["B281","O323"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B281","O323"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B281","O323"]}
16 tokens
0819 · total · wording 0 · samef1ee9ac386ba27a3e55ce54533760c92dbbb78ba5b8cea3bd411a866929194f9
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"F146","units":0,"price":14},{"id":"D154","units":12,"price":7},{"id":"U213","units":4,"price":17},{"id":"P978","units":4,"price":4},{"id":"A569","units":10,"price":13}]
Checked answer
{"total": 298}
Seed

wrong_schema_or_value

{"total": 140}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":308}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0820 · sort · wording 0 · gaine83506d658dbe5247b89b213845230441cd7a23c8b2658af346ed10b09ebfdca
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"T324","priority":7},{"id":"V680","priority":4},{"id":"A667","priority":2},{"id":"Q749","priority":8},{"id":"I825","priority":6}]
Checked answer
{"ids": ["Q749", "T324"]}
Seed

wrong_schema_or_value

{"ids": ["V680", "A667", "I825", "Q749", "T324"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Q749","T324"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Q749","T324"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q749","T324"]}
16 tokens
0821 · lookup · wording 1 · gain2fae99d6673bc018627a6d74592f871af70c2bc3d103519086146cd1ca8aff9d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"U832","city":"Riga","units":10},{"id":"C630","city":"Perth","units":2},{"id":"Q931","city":"Bern","units":6},{"id":"N434","city":"Oslo","units":8},{"id":"Z422","city":"Perth","units":13}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":10}
12 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0822 · filter · wording 1 · gainda4945693bdfa34cd7ec8d285afdc31435b53bb5f641861160823917037d6d83
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"T492","status":"closed"},{"id":"W570","status":"pending"},{"id":"Y288","status":"ready"},{"id":"Z353","status":"closed"},{"id":"M396","status":"ready"}]
Checked answer
{"ids": ["M396", "Y288"]}
Seed

wrong_schema_or_value

{
  "ids": ["M396", "Y288", "Z353", "T492"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["M396","Y288"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M396","Y288"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M396","Y288"]}
16 tokens
0823 · total · wording 1 · same837db5a8346f326f574be466db4f38f48ec3fb28eed729cbf61b239e8eb3a729
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"O539","units":19,"price":7},{"id":"T949","units":14,"price":8},{"id":"V150","units":11,"price":1},{"id":"B261","units":6,"price":11},{"id":"F994","units":14,"price":8}]
Checked answer
{"total": 434}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0824 · sort · wording 1 · gain30341d87eff2d468a4a28e50116a1f6539c8b477f1d6d3ed9eb9128009ed671a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"O199","priority":2},{"id":"X174","priority":2},{"id":"J151","priority":2},{"id":"C279","priority":9},{"id":"W665","priority":6}]
Checked answer
{"ids": ["C279", "W665"]}
Seed

invalid_json_object

[{"id":"O199","priority":2},{"id":"X174","priority":2},{"id":"J151","priority":2},{"id":"C279","priority":9},{"id":"W665","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["C279","W665"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C279","W665"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C279","W665"]}
16 tokens
0825 · lookup · wording 2 · gain1274aae96b1e0a71213dc6b79fea148c1d9f813381b0dd05eee312b86cb0ad7d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to L921? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"L921","city":"Lima","units":8},{"id":"A115","city":"Bern","units":0},{"id":"W919","city":"Lima","units":12},{"id":"K247","city":"Oslo","units":4},{"id":"F745","city":"Accra","units":14}]
Checked answer
{"city": "Lima", "units": 8}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":8}
11 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":8}
11 tokens
0826 · filter · wording 2 · same73a42045c0cb52412edbe93f6cfe1ac0a488d013a14e0317957cb432055785ce
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"W602","status":"closed"},{"id":"F932","status":"pending"},{"id":"N941","status":"pending"},{"id":"G365","status":"ready"},{"id":"Q422","status":"pending"}]
Checked answer
{"ids": ["G365"]}
Seed

invalid_json_object

[{"id":"G365"}]
11 tokens
1 GPU · AdamW

invalid_json_object

["G365"]
7 tokens
4 GPUs · DDP

Correct

{"ids":["G365"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G365"]]
11 tokens
0827 · total · wording 2 · same4c8543e9c186ae1e3eed5b741f52c9db4a8f1fc6d461a89b0dbe2ed75fcf2bbb
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"F398","units":0,"price":2},{"id":"T657","units":13,"price":12},{"id":"Z855","units":1,"price":11},{"id":"P882","units":11,"price":15},{"id":"W189","units":5,"price":11}]
Checked answer
{"total": 387}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0828 · sort · wording 2 · sameefe07be8152088133fc4600330ffcf072578fae5c7b667febe91576c31bd459e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"I138","priority":1},{"id":"Z640","priority":8},{"id":"W875","priority":2},{"id":"V174","priority":5},{"id":"K455","priority":1}]
Checked answer
{"ids": ["Z640", "V174"]}
Seed

wrong_schema_or_value

{"ids": ["I138", "Z640", "W875", "V174", "K455"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Z640","W875"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["Z640","I138"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z640","I138"]}
16 tokens
0829 · lookup · wording 3 · gainf97169b264ce08cbb403dbdfcf108cec5e0133bab8a485cfcf5960dafdd6697c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=M214. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"D556","city":"Riga","units":9},{"id":"Y261","city":"Kyoto","units":19},{"id":"P769","city":"Kyoto","units":6},{"id":"B321","city":"Kyoto","units":18},{"id":"M214","city":"Riga","units":15}]
Checked answer
{"city": "Riga", "units": 15}
Seed

wrong_schema_or_value

{"city":"Riga","units":9}
11 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":15}
12 tokens
0830 · filter · wording 3 · gaina74dfefae0fef88990dfe059d43b02cb9e41f81f7dac3d1c660d3a54c221029b
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"E431","status":"closed"},{"id":"Z661","status":"closed"},{"id":"K275","status":"closed"},{"id":"K986","status":"ready"},{"id":"Z965","status":"closed"}]
Checked answer
{"ids": ["K986"]}
Seed

wrong_schema_or_value

{"ids": ["K275", "K986", "Z661", "Z965", "E431"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["K986"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["K986"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["K986"]}
11 tokens
0831 · total · wording 3 · same1fb674d86760a7b7cc0976f69c74f1b919400b5a4a23478756d7513f94ee54d7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"K349","units":12,"price":18},{"id":"B803","units":5,"price":9},{"id":"H687","units":5,"price":19},{"id":"B430","units":14,"price":5},{"id":"L745","units":7,"price":1}]
Checked answer
{"total": 433}
Seed

invalid_json_object

{
  "total": 12 * 18 + 5 * 9 + 5 * 19 + 14 * 5 + 7 * 1 = 216 + 45 + 95 + 70 + 7 = 522
}
66 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0832 · sort · wording 3 · gain028fae400c0585a7d96490f312df2f47fddfe64b22a77e2706b4d6b43839fd06
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"T748","priority":9},{"id":"Q109","priority":9},{"id":"G675","priority":6},{"id":"N759","priority":2},{"id":"C120","priority":8}]
Checked answer
{"ids": ["Q109", "T748"]}
Seed

invalid_json_object

[{"id":"T748","priority":9},{"id":"G675","priority":6},{"id":"N759","priority":2},{"id":"C120","priority":8}]
50 tokens
1 GPU · AdamW

Correct

{"ids":["Q109","T748"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["T748","Q109"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Q109","T748"]}
16 tokens
0833 · lookup · wording 0 · gaindd8ff4f5b105808b37f53af7d8791a7ba66e8bdf05ccc3dfb3426ab1dd3f3401
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J428. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"A726","city":"Lima","units":16},{"id":"J428","city":"Accra","units":19},{"id":"V421","city":"Accra","units":6},{"id":"H883","city":"Oslo","units":3},{"id":"Q488","city":"Oslo","units":13}]
Checked answer
{"city": "Accra", "units": 19}
Seed

wrong_schema_or_value

{
  "city": "Lima",
  "units": 16
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":19}
12 tokens
0834 · filter · wording 0 · samed390035fa34e1ded7ab46bbdfb8b0510ab0be3eaebf68d81bfd12d87d4fde223
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"G325","status":"ready"},{"id":"S282","status":"ready"},{"id":"Y623","status":"pending"},{"id":"F985","status":"pending"},{"id":"F383","status":"ready"}]
Checked answer
{"ids": ["F383", "G325", "S282"]}
Seed

wrong_schema_or_value

{"ids": ["G325", "S282", "F383", "F985", "Y623"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["G325","F383","S282"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["G325","F383"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G325","F383","S282"]}
21 tokens
0835 · total · wording 0 · same2a2680d678a712fc6f10c89d759c00b9f870345d3b57e1b5ddb4373b30b37a4e
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Z561","units":15,"price":12},{"id":"Z376","units":16,"price":17},{"id":"I560","units":1,"price":15},{"id":"K556","units":7,"price":15},{"id":"M590","units":17,"price":13}]
Checked answer
{"total": 793}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0836 · sort · wording 0 · samecb25c3494a3dfab3e25a054561ed8fb8b37dfee2ec36bbbdd677e43541e2279d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"H139","priority":4},{"id":"U611","priority":3},{"id":"D486","priority":1},{"id":"R368","priority":5},{"id":"O787","priority":9}]
Checked answer
{"ids": ["O787", "R368"]}
Seed

wrong_schema_or_value

{"ids": ["U611", "D486", "R368", "H139", "O787"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["O787","R368"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O787","R368"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O787","H139"]}
16 tokens
0837 · lookup · wording 1 · samebbfc5a7255fdb09e225a29bd744c3d7790181c03c1d71ae630ad20c579b97153
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id W755 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W755","city":"Suva","units":13},{"id":"V627","city":"Riga","units":2},{"id":"R752","city":"Perth","units":5},{"id":"Z646","city":"Suva","units":12},{"id":"N556","city":"Oslo","units":9}]
Checked answer
{"city": "Suva", "units": 13}
Seed

Correct

{"city":"Suva","units":13}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":13}
12 tokens
0838 · filter · wording 1 · samef65bc4bfdec3ded50bb8fa9e6f25592ad22bc848969ff34f2fa32379a68a0d74
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"F854","status":"closed"},{"id":"G755","status":"ready"},{"id":"X809","status":"pending"},{"id":"U782","status":"pending"},{"id":"L922","status":"ready"}]
Checked answer
{"ids": ["G755", "L922"]}
Seed

wrong_schema_or_value

{
  "ids": ["F854", "G755", "U782", "L922"]
}
32 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["L922","G755"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["L922","G755"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["L922","G755"]}
16 tokens
0839 · total · wording 1 · same80d538b82b696d89ef58016f5e6f1ac527c3423460c8bf055cae51bead2702f6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L771","units":10,"price":14},{"id":"E108","units":7,"price":1},{"id":"N306","units":13,"price":4},{"id":"A130","units":13,"price":17},{"id":"P148","units":11,"price":19}]
Checked answer
{"total": 629}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0840 · sort · wording 1 · gainc4c6498fe25741cfbb4c5316cd57b18d5f17229351d31e0e5eb25274ea029dd2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"R673","priority":2},{"id":"Z703","priority":5},{"id":"E157","priority":8},{"id":"M255","priority":3},{"id":"R122","priority":3}]
Checked answer
{"ids": ["E157", "Z703"]}
Seed

invalid_json_object

[{"id":"R673","priority":2},{"id":"Z703","priority":5},{"id":"E157","priority":8},{"id":"M255","priority":3},{"id":"R122","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["E157","Z703"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E157","Z703"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E157","Z703"]}
16 tokens
0841 · lookup · wording 2 · samee59ed92abfb34fe447d6ec2cc5801e8246635de6a2be0c94cf84aa9afc7e9f5c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"O454","city":"Riga","units":12},{"id":"R995","city":"Accra","units":8},{"id":"H375","city":"Oslo","units":0},{"id":"M694","city":"Accra","units":3},{"id":"N192","city":"Bern","units":12}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0842 · filter · wording 2 · same42022a7bd3f9ffaab68fa01f8123198878b3af292d1abd71ccb81b6def0cddba
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"G633","status":"ready"},{"id":"Y691","status":"ready"},{"id":"B215","status":"ready"},{"id":"T843","status":"ready"},{"id":"T426","status":"pending"}]
Checked answer
{"ids": ["B215", "G633", "T843", "Y691"]}
Seed

invalid_json_object

[{"id":"G633","status":"ready"},{"id":"Y691","status":"ready"},{"id":"B215","status":"ready"},{"id":"T843","status":"ready"},{"id":"T426","status":"pending"}]
63 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["B215","G633","Y691","T843"]}
26 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["B215","G633","Y691","T843"]}
26 tokens
4 GPUs · local windows

invalid_json_object

["ids":["G633","B215","T843"]]
21 tokens
0843 · total · wording 2 · samef446e64095b4896a248e68e02c473ecbfc2220d606a6cf76ee2e55531a6d6d84
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"C704","units":7,"price":9},{"id":"W271","units":7,"price":4},{"id":"F628","units":15,"price":19},{"id":"R939","units":13,"price":19},{"id":"A532","units":14,"price":15}]
Checked answer
{"total": 833}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0844 · sort · wording 2 · samed57e3e95e7ee12c06bdac598937d0a2a3a11f42ec2c307947e72f9bb985c0747
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"E397","priority":4},{"id":"V984","priority":8},{"id":"B307","priority":1},{"id":"V403","priority":5},{"id":"D550","priority":1}]
Checked answer
{"ids": ["V984", "V403"]}
Seed

wrong_schema_or_value

{"ids": ["V403", "V984", "B307", "E397", "D550"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["V984","E397"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V984","E397"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["V403","E397"]}
16 tokens
0845 · lookup · wording 3 · gain7da1c0d180b6463ac44caa98bfb6409c84c2d8f5ee9b9226b6465b5c3ce4dee6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=V135. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"N444","city":"Suva","units":11},{"id":"M137","city":"Accra","units":3},{"id":"V135","city":"Perth","units":7},{"id":"L468","city":"Suva","units":13},{"id":"D518","city":"Suva","units":10}]
Checked answer
{"city": "Perth", "units": 7}
Seed

wrong_schema_or_value

{"city":"Suva","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":7}
11 tokens
0846 · filter · wording 3 · same8f225805a1c1f42bfe5021a9da15aff9650ae5c8643d63e95fa16efaa04543b8
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D184","status":"closed"},{"id":"U779","status":"ready"},{"id":"E775","status":"pending"},{"id":"P626","status":"pending"},{"id":"V700","status":"ready"}]
Checked answer
{"ids": ["U779", "V700"]}
Seed

wrong_schema_or_value

{"ids": ["U779", "D184", "P626", "E775", "V700"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D184","V700","U779"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D184","V700","U779"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D184","V700","U779"]}
21 tokens
0847 · total · wording 3 · sameef6d9ddab61443d7afbebbd9073640cf7db726ec87265b410dcdcb29bca606fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"W176","units":0,"price":1},{"id":"D428","units":14,"price":14},{"id":"O740","units":1,"price":7},{"id":"H926","units":9,"price":12},{"id":"M742","units":7,"price":19}]
Checked answer
{"total": 444}
Seed

wrong_schema_or_value

{
  "total": 40
}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":408}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0848 · sort · wording 3 · same2d4db3ed264646a5e226c0ebe2e5ad110623e7184b71146ff696f14cc4350ebd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"U789","priority":2},{"id":"W577","priority":9},{"id":"G117","priority":3},{"id":"A652","priority":1},{"id":"P673","priority":4}]
Checked answer
{"ids": ["W577", "P673"]}
Seed

invalid_json_object

[{"id":"U789"},{"id":"W577"},{"id":"G117"},{"id":"A652"},{"id":"P673"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["W577","P673"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W577","G117"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["W577","A652"]}
16 tokens
0849 · lookup · wording 0 · sameb67b53cf130d6d379d493aa64df7096d19dad2431584e2694a2985cee9824985
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J781. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"J781","city":"Accra","units":18},{"id":"M646","city":"Bern","units":13},{"id":"D716","city":"Riga","units":0},{"id":"N486","city":"Riga","units":19},{"id":"W510","city":"Bern","units":17}]
Checked answer
{"city": "Accra", "units": 18}
Seed

Correct

{
  "city": "Accra",
  "units": 18
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":18}
12 tokens
0850 · filter · wording 0 · gain2846efbf2549267a7993ba45b540892cf27da80a602778299799575e00419beb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Z398","status":"closed"},{"id":"P481","status":"pending"},{"id":"Z921","status":"closed"},{"id":"I243","status":"closed"},{"id":"X325","status":"ready"}]
Checked answer
{"ids": ["X325"]}
Seed

wrong_schema_or_value

{"ids": ["I243", "X325"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["X325"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["X325"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["X325"]}
11 tokens
0851 · total · wording 0 · same3e9e354123f55c0dc92109c8c88493bbb173da885304f1ec2737d197c0058c6a
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"S145","units":6,"price":7},{"id":"I759","units":12,"price":12},{"id":"O172","units":18,"price":13},{"id":"K696","units":18,"price":10},{"id":"V441","units":13,"price":6}]
Checked answer
{"total": 678}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0852 · sort · wording 0 · gainbed4723ca1753a03bddd3acde14aba9bbe8ffbff2524840e0516028a9cdb4613
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"P900","priority":8},{"id":"W817","priority":3},{"id":"Q657","priority":7},{"id":"Z633","priority":3},{"id":"Y988","priority":9}]
Checked answer
{"ids": ["Y988", "P900"]}
Seed

wrong_schema_or_value

{"ids": ["P900", "W817", "Q657", "Z633", "Y988"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Y988","P900"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Y988","P900"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Y988","P900"]}
16 tokens
0853 · lookup · wording 1 · gainfd3c33b51f970c34a3ee3d23a2de375133a0b24f73de4f7d5ff5fd2399062f15
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id V994 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"Z770","city":"Accra","units":15},{"id":"V994","city":"Bern","units":0},{"id":"D545","city":"Accra","units":11},{"id":"M535","city":"Accra","units":4},{"id":"T297","city":"Perth","units":18}]
Checked answer
{"city": "Bern", "units": 0}
Seed

wrong_schema_or_value

{"city":"Accra","units":15}
12 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":0}
10 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":0}
10 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":0}
10 tokens
0854 · filter · wording 1 · gain0ff1ce5f0dbf8de629c56aac3c684dfab9aa88e683ac4187a43e7b0c41c48ab5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"Z734","status":"closed"},{"id":"F654","status":"pending"},{"id":"E636","status":"ready"},{"id":"H115","status":"ready"},{"id":"G824","status":"pending"}]
Checked answer
{"ids": ["E636", "H115"]}
Seed

wrong_schema_or_value

{
  "ids": ["E636", "F654", "G824", "H115"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["E636","H115"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E636","H115"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E636","H115"]}
16 tokens
0855 · total · wording 1 · samea4917987e5a0e062270e21dcbe49a4a704271fa955b3edb9f762b356f16b4e40
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"P155","units":10,"price":3},{"id":"E176","units":17,"price":1},{"id":"M179","units":19,"price":3},{"id":"R566","units":16,"price":8},{"id":"Y813","units":19,"price":14}]
Checked answer
{"total": 498}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0856 · sort · wording 1 · samec91038d479c84646d0f091744992da4a7a311217fc5f6d0e6392b4918ef62aa2
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"K501","priority":5},{"id":"X342","priority":1},{"id":"J540","priority":8},{"id":"Y284","priority":7},{"id":"B987","priority":8}]
Checked answer
{"ids": ["B987", "J540"]}
Seed

invalid_json_object

[{"id":"K501","priority":5},{"id":"X342","priority":1},{"id":"J540","priority":8},{"id":"Y284","priority":7},{"id":"B987","priority":8}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["J540","B987"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["J540","B987"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J540","B987"]}
16 tokens
0857 · lookup · wording 2 · gaine90ca7471d22f83dee15b1e280c653cd9ee83d539c1e4a017a303f3d9b472c39
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to V703? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T485","city":"Bern","units":18},{"id":"N582","city":"Suva","units":10},{"id":"V703","city":"Accra","units":4},{"id":"E442","city":"Suva","units":1},{"id":"X554","city":"Accra","units":2}]
Checked answer
{"city": "Accra", "units": 4}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":4}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":4}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":4}
11 tokens
0858 · filter · wording 2 · samede1ba9e21b5ca9b0c7062b690ee96f639e614e5f1b06e4db80022d031b5c78fe
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"I634","status":"ready"},{"id":"O874","status":"closed"},{"id":"Y419","status":"ready"},{"id":"M252","status":"closed"},{"id":"I854","status":"pending"}]
Checked answer
{"ids": ["I634", "Y419"]}
Seed

invalid_json_object

[{"id":"I634"},{"id":"O874"},{"id":"Y419"},{"id":"M252"},{"id":"I854"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["I634","Y419"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I634","Y419"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["I634","Y419"]]
16 tokens
0859 · total · wording 2 · same0c4c20dacaa875a44e8364f8c0d123277245ee3cfe74356ad9e607720daeb7be
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"W253","units":3,"price":4},{"id":"Q290","units":5,"price":7},{"id":"R215","units":8,"price":14},{"id":"E163","units":0,"price":19},{"id":"Y411","units":5,"price":11}]
Checked answer
{"total": 214}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":268}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":268}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0860 · sort · wording 2 · gain9cabc7416efe00e2b501d645401f4055e83ac830c0206646a1648748584c31ad
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Y149","priority":5},{"id":"P789","priority":6},{"id":"G783","priority":3},{"id":"E326","priority":5},{"id":"O778","priority":3}]
Checked answer
{"ids": ["P789", "E326"]}
Seed

wrong_schema_or_value

{"ids": ["Y149", "P789", "G783", "E326", "O778"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P789","Y149"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["P789","Y149"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P789","E326"]}
16 tokens
0861 · lookup · wording 3 · gain1495a09a41538c0a3b79bd67273b113d727e98e447e3bf6ab81b1d1e93d7fc46
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E806","city":"Oslo","units":2},{"id":"R253","city":"Perth","units":4},{"id":"X600","city":"Lima","units":1},{"id":"O400","city":"Riga","units":15},{"id":"K650","city":"Perth","units":18}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Oslo","units":2}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0862 · filter · wording 3 · gain124a38b569f3522a98e8c6e1801447838fee18297969f3f6296a7f2d1fb512a3
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"J162","status":"closed"},{"id":"Q755","status":"ready"},{"id":"P260","status":"ready"},{"id":"T704","status":"pending"},{"id":"O986","status":"closed"}]
Checked answer
{"ids": ["P260", "Q755"]}
Seed

wrong_schema_or_value

{"ids": ["Q755", "P260", "T704", "J162", "O986"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["P260","Q755"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P260","Q755"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P260","Q755"]}
16 tokens
0863 · total · wording 3 · same31f5f5ea77ad48f76a444fcc37ff9854bb70f6575f219d408cbd93f59b386fb3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"H860","units":8,"price":1},{"id":"C268","units":6,"price":15},{"id":"L713","units":8,"price":11},{"id":"C233","units":18,"price":8},{"id":"E221","units":15,"price":19}]
Checked answer
{"total": 615}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0864 · sort · wording 3 · gainef7df038c42a4c272af9aa7284c3dbd46c1de2109e60faaaf250639e75ffd3bf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"H633","priority":6},{"id":"Z763","priority":1},{"id":"K549","priority":5},{"id":"F157","priority":6},{"id":"N714","priority":3}]
Checked answer
{"ids": ["F157", "H633"]}
Seed

invalid_json_object

[{"id":"H633","priority":6},{"id":"Z763","priority":1},{"id":"K549","priority":5},{"id":"F157","priority":6},{"id":"N714","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["F157","H633"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["H633","F157"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F157","H633"]}
16 tokens
0865 · lookup · wording 0 · same8ae00556a9134c8dba9c639c582885f033f342b1e61e8d6bf3c2aea77aefe0d5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record W977. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"W977","city":"Riga","units":1},{"id":"L878","city":"Accra","units":0},{"id":"B613","city":"Suva","units":19},{"id":"J757","city":"Perth","units":14},{"id":"L924","city":"Bern","units":19}]
Checked answer
{"city": "Riga", "units": 1}
Seed

Correct

{
  "city": "Riga",
  "units": 1
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":1}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":1}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":1}
11 tokens
0866 · filter · wording 0 · gaince1c2475a7770962004eac8b96de83dcdb4c268f015aa27841525fa882b27384
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"L613","status":"pending"},{"id":"D960","status":"ready"},{"id":"L471","status":"pending"},{"id":"A214","status":"closed"},{"id":"P169","status":"closed"}]
Checked answer
{"ids": ["D960"]}
Seed

wrong_schema_or_value

{"ids": ["D960", "L471", "L613", "P169"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["D960"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["D960"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["D960"]}
11 tokens
0867 · total · wording 0 · samed1540e2112b8b3c51f7d4a7b997380efb0a71e97d41bb61c698045b650dad25d
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"M277","units":7,"price":2},{"id":"F765","units":10,"price":11},{"id":"M152","units":5,"price":14},{"id":"C791","units":5,"price":10},{"id":"W383","units":11,"price":3}]
Checked answer
{"total": 277}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":318}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0868 · sort · wording 0 · same82e6c9f4cd83a2e86f77f62f85c77c5c89df1967b74025e5fd8eeebd3197f048
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"O828","priority":7},{"id":"G559","priority":7},{"id":"E998","priority":5},{"id":"B236","priority":8},{"id":"E543","priority":7}]
Checked answer
{"ids": ["B236", "E543"]}
Seed

wrong_schema_or_value

{"ids": ["O828", "G559", "E998", "B236", "E543"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["B236","O828"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["B236","G559"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["B236","G559"]}
16 tokens
0869 · lookup · wording 1 · samed383d4a3daf42d67cdc11106d2a720150bb0d26efcbeb3fb1883f7a9444ebd94
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M387 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"O206","city":"Kyoto","units":12},{"id":"K434","city":"Perth","units":18},{"id":"M387","city":"Suva","units":17},{"id":"X835","city":"Perth","units":10},{"id":"K162","city":"Suva","units":11}]
Checked answer
{"city": "Suva", "units": 17}
Seed

Correct

{"city":"Suva","units":17}
12 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":17}
12 tokens
0870 · filter · wording 1 · same8613fd3cf69ea7ed4a41b6152c3e59a66a4a720f42873590f8c5f2001757558b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"K481","status":"closed"},{"id":"T618","status":"ready"},{"id":"V550","status":"closed"},{"id":"T163","status":"pending"},{"id":"F196","status":"pending"}]
Checked answer
{"ids": ["T618"]}
Seed

wrong_schema_or_value

{
  "ids": ["T163", "T618", "V550", "K481"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["T618"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T618"]}
11 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["T163"]}
11 tokens
0871 · total · wording 1 · samee1500e7e05251abd9945491d038fcb8622e589e60a1cdd029070873a403f7751
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"J744","units":17,"price":16},{"id":"Q713","units":5,"price":16},{"id":"M695","units":16,"price":17},{"id":"G918","units":7,"price":13},{"id":"R442","units":1,"price":13}]
Checked answer
{"total": 728}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0872 · sort · wording 1 · gain28d57e5f377d6adab6e27db4d644c9fcc4d2dc64810005586056ed0810f38257
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"N418","priority":7},{"id":"Z302","priority":2},{"id":"J827","priority":1},{"id":"E635","priority":6},{"id":"O788","priority":5}]
Checked answer
{"ids": ["N418", "E635"]}
Seed

invalid_json_object

[{"id":"N418","priority":7},{"id":"Z302","priority":2},{"id":"J827","priority":1},{"id":"E635","priority":6},{"id":"O788","priority":5}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["N418","E635"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["N418","E635"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["N418","E635"]}
16 tokens
0873 · lookup · wording 2 · gaind0663ce64aa54daa60a6931c76200f9f461e97bc3e6ebb06534a5453a32eab50
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to G928? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"Y697","city":"Bern","units":19},{"id":"A260","city":"Kyoto","units":9},{"id":"A266","city":"Perth","units":19},{"id":"I322","city":"Lima","units":13},{"id":"G928","city":"Oslo","units":15}]
Checked answer
{"city": "Oslo", "units": 15}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":15}
12 tokens
0874 · filter · wording 2 · same42fe6daae98f5527aaa978196b8d72100ea01888b82238c62ed6315f00b3566f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"K747","status":"pending"},{"id":"I385","status":"ready"},{"id":"U972","status":"ready"},{"id":"P296","status":"ready"},{"id":"V834","status":"pending"}]
Checked answer
{"ids": ["I385", "P296", "U972"]}
Seed

invalid_json_object

[{"id":"I385","status":"ready"},{"id":"U972","status":"ready"},{"id":"P296","status":"ready"},{"id":"K747","status":"pending"},{"id":"V834","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["I385","P296","U972"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["I385","P296","U972"]}
21 tokens
4 GPUs · local windows

invalid_json_object

["ids":["I385","P296","U972"]]
21 tokens
0875 · total · wording 2 · same3529794ba5294550be040aaba84c99604ea5c50f6d901d4334e3f640adc90ec8
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"P717","units":10,"price":3},{"id":"Q248","units":15,"price":8},{"id":"X572","units":4,"price":8},{"id":"K167","units":14,"price":10},{"id":"V976","units":17,"price":19}]
Checked answer
{"total": 645}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0876 · sort · wording 2 · gain891c8b21eddacfdd9873372f1ac77e472d79396ee2d30a68557623ece9e73840
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"O305","priority":9},{"id":"F603","priority":8},{"id":"M313","priority":6},{"id":"M834","priority":4},{"id":"F323","priority":1}]
Checked answer
{"ids": ["O305", "F603"]}
Seed

wrong_schema_or_value

{"ids": ["O305", "F603", "M313", "M834", "F323"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["O305","F603"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O305","F603"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["O305","F603"]}
16 tokens
0877 · lookup · wording 3 · samecd57fdca1185ea6b8ac2e859282c2129463bb4feb0e97f096c09434191f5d5d5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Q775. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"V790","city":"Perth","units":6},{"id":"K424","city":"Oslo","units":2},{"id":"Q775","city":"Accra","units":13},{"id":"E646","city":"Riga","units":5},{"id":"X165","city":"Kyoto","units":4}]
Checked answer
{"city": "Accra", "units": 13}
Seed

Correct

{"city": "Accra", "units": 13}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":13}
12 tokens
0878 · filter · wording 3 · gain2e0197c5a9242ed058c809b5da4ebc2314c870b8197c8b4855fe346a30ae1f68
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"E306","status":"pending"},{"id":"F436","status":"pending"},{"id":"D623","status":"closed"},{"id":"M405","status":"ready"},{"id":"B240","status":"closed"}]
Checked answer
{"ids": ["M405"]}
Seed

wrong_schema_or_value

{"ids": ["D623", "M405", "E306", "F436", "B240"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["M405"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["M405"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["M405"]}
11 tokens
0879 · total · wording 3 · samee70fee93bf126be9c1e26a28153099b21d2c8cd8556d6a59b1a5428e2dff095d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"P833","units":2,"price":4},{"id":"Q188","units":15,"price":13},{"id":"N282","units":13,"price":14},{"id":"R694","units":19,"price":10},{"id":"Y159","units":16,"price":12}]
Checked answer
{"total": 767}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0880 · sort · wording 3 · gainc64bc67351753b2b28314ae36ed0a5edf7df654ea04934d0245e16f65dd68bbf
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"S477","priority":4},{"id":"M281","priority":2},{"id":"Q971","priority":1},{"id":"H289","priority":3},{"id":"P129","priority":7}]
Checked answer
{"ids": ["P129", "S477"]}
Seed

invalid_json_object

[{"id":"S477","priority":4},{"id":"M281","priority":2},{"id":"Q971","priority":1},{"id":"H289","priority":3},{"id":"P129","priority":7}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["P129","H289"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["P129","S477"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["P129","S477"]}
16 tokens
0881 · lookup · wording 0 · gaina1cfc86a2644d2297dd65f20a35bc1293a9a0f7bcf14b308ae3a208d9d1504bc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"C744","city":"Accra","units":19},{"id":"R772","city":"Oslo","units":7},{"id":"O326","city":"Accra","units":4},{"id":"G325","city":"Perth","units":12},{"id":"H314","city":"Perth","units":19}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Accra",
  "units": 19
}
19 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0882 · filter · wording 0 · same2f6dbed5eb1ceeb7857d3ff80ad4d109b08510b2e122e7d83c2e515ff074d892
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Q895","status":"ready"},{"id":"O228","status":"ready"},{"id":"P156","status":"closed"},{"id":"Y388","status":"pending"},{"id":"J777","status":"ready"}]
Checked answer
{"ids": ["J777", "O228", "Q895"]}
Seed

wrong_schema_or_value

{"ids": ["O228", "Q895", "P156", "J777", "Y388"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J777","O228","Q895"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["J777","O228","Q895"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Q895","J777","O228"]}
21 tokens
0883 · total · wording 0 · same3a8867dcec8584c251925585b6f577c157b924614fc5c14164f88f8e7275fa50
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"C594","units":19,"price":8},{"id":"I246","units":8,"price":5},{"id":"M755","units":8,"price":18},{"id":"J590","units":13,"price":12},{"id":"X233","units":18,"price":14}]
Checked answer
{"total": 744}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":608}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0884 · sort · wording 0 · same69155078163a0183e3e791ee8503243c58ce1e49318e7d87355c02067de02e11
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"A738","priority":2},{"id":"N127","priority":7},{"id":"M686","priority":7},{"id":"R850","priority":7},{"id":"J523","priority":5}]
Checked answer
{"ids": ["M686", "N127"]}
Seed

wrong_schema_or_value

{"ids": ["A738", "M686", "R850", "J523", "N127"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["N127","M686"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N127","M686"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["N127","R850"]}
16 tokens
0885 · lookup · wording 1 · gain9a2da085d1d7e688c46f4401bdbbda2f3b1b81614a6b5ed51f38544d1f7ef9aa
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id U542 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"G433","city":"Lima","units":17},{"id":"F392","city":"Riga","units":16},{"id":"R443","city":"Suva","units":16},{"id":"U542","city":"Accra","units":7},{"id":"G784","city":"Bern","units":1}]
Checked answer
{"city": "Accra", "units": 7}
Seed

wrong_schema_or_value

{"city":"Lima","units":17}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":7}
11 tokens
0886 · filter · wording 1 · sameef9cf190af973f42ff41053c4876206b45f9e84ee7d61e9130a52efa9444224e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"R811","status":"pending"},{"id":"F846","status":"closed"},{"id":"O714","status":"ready"},{"id":"U427","status":"pending"},{"id":"F477","status":"ready"}]
Checked answer
{"ids": ["F477", "O714"]}
Seed

wrong_schema_or_value

{
  "ids": ["O714", "F477", "U427", "R811", "F846"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["F477","O714"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F477","O714"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O714","F477"]}
16 tokens
0887 · total · wording 1 · samea660909ec904db35ce734cfc3d26522a64d4f486065081b5319068b2806043d6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"J653","units":9,"price":19},{"id":"U460","units":5,"price":9},{"id":"J542","units":16,"price":13},{"id":"B972","units":11,"price":6},{"id":"Z170","units":14,"price":15}]
Checked answer
{"total": 700}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":548}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0888 · sort · wording 1 · gain692df95cf735da128999fa3420d3ba90930260efccc6a4c6b322ceba406c950d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"U375","priority":4},{"id":"J622","priority":5},{"id":"Q401","priority":2},{"id":"Q411","priority":2},{"id":"M382","priority":3}]
Checked answer
{"ids": ["J622", "U375"]}
Seed

invalid_json_object

[{"id":"U375","priority":4},{"id":"J622","priority":5},{"id":"Q401","priority":2},{"id":"Q411","priority":2},{"id":"M382","priority":3}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["J622","M382"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["J622","U375"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["J622","U375"]}
16 tokens
0889 · lookup · wording 2 · gain2c7addb9af4dada3d3319a0c4b5758087ccbf158c0925a6b9f75053059a6fea5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to N837? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"R818","city":"Suva","units":6},{"id":"I961","city":"Accra","units":14},{"id":"C129","city":"Kyoto","units":18},{"id":"B620","city":"Suva","units":13},{"id":"N837","city":"Riga","units":14}]
Checked answer
{"city": "Riga", "units": 14}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":14}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":14}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":14}
12 tokens
0890 · filter · wording 2 · same562ebfaf3c436690ae1a822df145e437745fa0d1ba190d22926039655a588667
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"Y555","status":"pending"},{"id":"W811","status":"ready"},{"id":"R428","status":"pending"},{"id":"L987","status":"pending"},{"id":"O244","status":"ready"}]
Checked answer
{"ids": ["O244", "W811"]}
Seed

invalid_json_object

[{"id":"W811","status":"ready"},{"id":"O244","status":"ready"},{"id":"R428","status":"pending"},{"id":"Y555","status":"pending"},{"id":"L987","status":"pending"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["O244","W811"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O244","W811"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["O244","W811"]]
16 tokens
0891 · total · wording 2 · same0259ca553380d8d9acf6775c993e87326f86238dfde2c51f9d15a936d40fdd1b
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R235","units":16,"price":13},{"id":"Q763","units":16,"price":12},{"id":"W979","units":16,"price":8},{"id":"Y565","units":0,"price":5},{"id":"Q289","units":19,"price":7}]
Checked answer
{"total": 661}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0892 · sort · wording 2 · gainfa3e27e98be17f507ac046a88630a02efbe5dd37d26b5142cb0349a36f27ce49
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"G739","priority":8},{"id":"D843","priority":7},{"id":"E667","priority":3},{"id":"A101","priority":9},{"id":"C990","priority":4}]
Checked answer
{"ids": ["A101", "G739"]}
Seed

wrong_schema_or_value

{"ids": ["G739", "D843"]}
17 tokens
1 GPU · AdamW

Correct

{"ids":["A101","G739"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A101","G739"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A101","G739"]}
16 tokens
0893 · lookup · wording 3 · same0bb84db4b9c0af70f6600efcc5b4738a66bb2317bec65b33f773f4a57504e168
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=K283. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E765","city":"Oslo","units":19},{"id":"K283","city":"Lima","units":7},{"id":"X218","city":"Oslo","units":9},{"id":"A813","city":"Bern","units":15},{"id":"Q172","city":"Lima","units":16}]
Checked answer
{"city": "Lima", "units": 7}
Seed

Correct

{"city":"Lima","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":7}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":7}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":7}
11 tokens
0894 · filter · wording 3 · gain44a5ab52a5305470f57146325973ee015fd773b3e97ab929f739823b3f814ee2
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"A421","status":"pending"},{"id":"M800","status":"pending"},{"id":"F539","status":"closed"},{"id":"I553","status":"pending"},{"id":"M174","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A421", "M800", "F539", "I553", "M174"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0895 · total · wording 3 · samed8752fd60d8e8678e1d602642a8f22d77c8457e02e2ba7ed3467a81b8a2fcc7f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"D412","units":2,"price":6},{"id":"Z528","units":2,"price":6},{"id":"I737","units":1,"price":15},{"id":"E676","units":1,"price":19},{"id":"V807","units":15,"price":18}]
Checked answer
{"total": 328}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0896 · sort · wording 3 · gain9a29655962ef9495e126ccae3b6de329e38709d367c01f945b91ef9264ef06ed
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"K383","priority":7},{"id":"D523","priority":5},{"id":"X892","priority":9},{"id":"Q543","priority":2},{"id":"X646","priority":6}]
Checked answer
{"ids": ["X892", "K383"]}
Seed

invalid_json_object

[{"id":"X892","priority":9},{"id":"Q543","priority":2},{"id":"K383","priority":7},{"id":"D523","priority":5},{"id":"X646","priority":6}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["X892","K383"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["X892","K383"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["X892","K383"]}
16 tokens
0897 · lookup · wording 0 · same05564b23581d2b0b301e650b90dc8ae44459d6508500c305af95385d4002bb37
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record B268. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"Q619","city":"Kyoto","units":16},{"id":"W157","city":"Bern","units":16},{"id":"B268","city":"Perth","units":3},{"id":"X102","city":"Kyoto","units":18},{"id":"Q310","city":"Accra","units":13}]
Checked answer
{"city": "Perth", "units": 3}
Seed

Correct

{
  "city": "Perth",
  "units": 3
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":3}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":3}
11 tokens
0898 · filter · wording 0 · gain736f8bb00ee62ee580736151fba65877ee2c23dcb9eeb9a2b2a56c6b67ba7aba
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"N717","status":"closed"},{"id":"P960","status":"closed"},{"id":"C910","status":"ready"},{"id":"N175","status":"pending"},{"id":"P802","status":"closed"}]
Checked answer
{"ids": ["C910"]}
Seed

wrong_schema_or_value

{"ids": ["C910", "N717", "N175", "P960", "P802"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C910"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["C910"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["C910"]}
11 tokens
0899 · total · wording 0 · same2ad66393bc5d436e6307e45ec5a2c588b4dac99eb41ffe8e46131c6775af9745
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"O141","units":3,"price":3},{"id":"F415","units":15,"price":13},{"id":"B763","units":11,"price":12},{"id":"O915","units":1,"price":8},{"id":"D288","units":0,"price":2}]
Checked answer
{"total": 344}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0900 · sort · wording 0 · gain6fd778b712f96fced3c6d7a8f84e8b269bb5457a4d6e54a7ce8085801bca121a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"E146","priority":6},{"id":"S734","priority":2},{"id":"Y953","priority":9},{"id":"Q794","priority":6},{"id":"X418","priority":1}]
Checked answer
{"ids": ["Y953", "E146"]}
Seed

wrong_schema_or_value

{"ids": ["S734", "E146", "Y953", "Q794", "X418"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["Y953","Q794"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Y953","E146"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Y953","E146"]}
16 tokens
0901 · lookup · wording 1 · gain640d45ea86fb6d0dad2fe834ad10ae7c3e13dcc4f03fb87573e79b6449f7f044
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"P210","city":"Riga","units":7},{"id":"O667","city":"Riga","units":16},{"id":"O453","city":"Bern","units":1},{"id":"U630","city":"Riga","units":9},{"id":"R968","city":"Riga","units":9}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0902 · filter · wording 1 · same530e264d63f54dc32824f042422a7cbd0edd225d38a3d174839250d02f6668e3
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"S967","status":"ready"},{"id":"P427","status":"ready"},{"id":"U535","status":"pending"},{"id":"E837","status":"pending"},{"id":"F612","status":"ready"}]
Checked answer
{"ids": ["F612", "P427", "S967"]}
Seed

wrong_schema_or_value

{
  "ids": ["S967", "P427", "U535", "E837", "F612"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["F612","P427","S967"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["F612","S967","P427"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["S967","F612","P427"]}
21 tokens
0903 · total · wording 1 · sameea4aa13e7a74bcd5660dc269dbacfdb98c6ef9906a352efd1c663dd1def85609
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"L474","units":19,"price":15},{"id":"S592","units":4,"price":18},{"id":"I504","units":4,"price":10},{"id":"X246","units":2,"price":15},{"id":"A218","units":11,"price":16}]
Checked answer
{"total": 603}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0904 · sort · wording 1 · gain4f243d265dc1805def3b994f08d2d4c5e4037a34a0fdc277d25469f1c9cfd0cb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"O327","priority":8},{"id":"D218","priority":5},{"id":"E992","priority":9},{"id":"I922","priority":5},{"id":"H256","priority":5}]
Checked answer
{"ids": ["E992", "O327"]}
Seed

invalid_json_object

[{"id":"O327","priority":8},{"id":"D218","priority":5},{"id":"E992","priority":9},{"id":"I922","priority":5},{"id":"H256","priority":5}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["E992","O327"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E992","O327"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E992","O327"]}
16 tokens
0905 · lookup · wording 2 · gainf17f579b04c9e4c03da5b83aa5e5d10249ed98a169d396926990e6f21543473d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to G162? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"G162","city":"Accra","units":13},{"id":"J763","city":"Accra","units":0},{"id":"C656","city":"Lima","units":17},{"id":"K133","city":"Suva","units":19},{"id":"P183","city":"Perth","units":4}]
Checked answer
{"city": "Accra", "units": 13}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":13}
12 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":13}
12 tokens
0906 · filter · wording 2 · same1efff789c965e0754704e670d200db8d122b698e7908c173a2ea9ca485aa28a0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"H145","status":"ready"},{"id":"M603","status":"pending"},{"id":"D918","status":"closed"},{"id":"M449","status":"pending"},{"id":"B241","status":"ready"}]
Checked answer
{"ids": ["B241", "H145"]}
Seed

invalid_json_object

[{"id":"B241","status":"ready"},{"id":"D918","status":"closed"},{"id":"H145","status":"ready"},{"id":"M449","status":"pending"},{"id":"M603","status":"pending"}]
63 tokens
1 GPU · AdamW

invalid_json_object

["B241","H145"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["B241","H145"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["H145","B241"]]
16 tokens
0907 · total · wording 2 · same24eb1d9da41f8a8e0bc98b6fa75bc505e85f4c70e642168275d5809c1b24a448
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"X309","units":12,"price":5},{"id":"O241","units":4,"price":4},{"id":"U603","units":2,"price":15},{"id":"Z571","units":5,"price":8},{"id":"E886","units":19,"price":5}]
Checked answer
{"total": 241}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0908 · sort · wording 2 · gaindccbb6258d1a055110afa3e4fa5f6958a4c637c325296ebfb011c260f8fca9d8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"S352","priority":7},{"id":"Q812","priority":5},{"id":"A239","priority":2},{"id":"I112","priority":5},{"id":"L552","priority":9}]
Checked answer
{"ids": ["L552", "S352"]}
Seed

wrong_schema_or_value

{"ids": ["S352", "Q812", "A239", "I112", "L552"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L552","S352"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["L552","S352"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["L552","S352"]}
16 tokens
0909 · lookup · wording 3 · gaind9aee52007411e8ff356d4789908d2feaa1d7b60cd5829e6194fbf92d0ee90fc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=S371. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"S385","city":"Lima","units":19},{"id":"X245","city":"Accra","units":6},{"id":"R329","city":"Lima","units":8},{"id":"H315","city":"Lima","units":17},{"id":"S371","city":"Accra","units":2}]
Checked answer
{"city": "Accra", "units": 2}
Seed

wrong_schema_or_value

{"city":"Lima","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":2}
11 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":2}
11 tokens
0910 · filter · wording 3 · gain03d0fc651fd89d8a9c3a74470216c44c898e4ea888ca5ea8073df2b430cbac11
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"W317","status":"pending"},{"id":"R962","status":"closed"},{"id":"Y795","status":"pending"},{"id":"B663","status":"ready"},{"id":"J412","status":"pending"}]
Checked answer
{"ids": ["B663"]}
Seed

wrong_schema_or_value

{"ids": ["B663", "J412", "Y795", "R962", "W317"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B663"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["B663"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["B663"]}
11 tokens
0911 · total · wording 3 · same3339536e924bf602f498705e29e53783223d07ca60c0c6c8c5fecc0ef8072a51
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"M228","units":19,"price":13},{"id":"F792","units":2,"price":3},{"id":"X862","units":7,"price":15},{"id":"U360","units":10,"price":1},{"id":"X826","units":14,"price":19}]
Checked answer
{"total": 634}
Seed

invalid_json_object

{
  "total": 13 + 3 + 15 + 10 + 19 = 52
}
30 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0912 · sort · wording 3 · gain07ba66773f64b03d17ff2195e5ab6a39aafe3f3492061563f559ae59ef378cab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"U439","priority":5},{"id":"T130","priority":2},{"id":"V966","priority":2},{"id":"U985","priority":2},{"id":"H364","priority":8}]
Checked answer
{"ids": ["H364", "U439"]}
Seed

invalid_json_object

[{"id":"U439"},{"id":"T130"},{"id":"V966"},{"id":"U985"},{"id":"H364"}]
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["H364","U985"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H364","U439"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H364","U439"]}
16 tokens
0913 · lookup · wording 0 · same9db75e7568d35e3d6eb966314ba3041f8d0e23e0c657be42081b7dde9cb75d3b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J553. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"D198","city":"Accra","units":11},{"id":"E548","city":"Suva","units":4},{"id":"I461","city":"Kyoto","units":6},{"id":"Q220","city":"Bern","units":16},{"id":"J553","city":"Accra","units":19}]
Checked answer
{"city": "Accra", "units": 19}
Seed

Correct

{
  "city": "Accra",
  "units": 19
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Accra","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Accra","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Accra","units":19}
12 tokens
0914 · filter · wording 0 · gaina75e80f1941df18dd02a51b2ba00103070849529f2b7960952fac49ad9a5eff7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"C443","status":"ready"},{"id":"W496","status":"pending"},{"id":"M654","status":"pending"},{"id":"V954","status":"pending"},{"id":"B528","status":"pending"}]
Checked answer
{"ids": ["C443"]}
Seed

wrong_schema_or_value

{"ids": ["C443", "M654", "V954", "B528"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["C443"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["C443"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["C443"]}
11 tokens
0915 · total · wording 0 · same880aa17ef632fb985608ae33a2c899e99ca7d1906293b624ec7ca0ed347ace29
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"R715","units":19,"price":16},{"id":"Q615","units":5,"price":6},{"id":"B233","units":2,"price":15},{"id":"I334","units":7,"price":18},{"id":"B343","units":15,"price":5}]
Checked answer
{"total": 565}
Seed

invalid_json_object

{"total": 19 * 16 + 5 * 6 + 2 * 15 + 7 * 18 + 15 * 5}
39 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0916 · sort · wording 0 · gain040e6e2631849651a0d9dc499ae411bc0c729bbeaa4a5b3b937c3205dd0099e7
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"C557","priority":9},{"id":"T297","priority":4},{"id":"M783","priority":6},{"id":"Q592","priority":6},{"id":"B712","priority":1}]
Checked answer
{"ids": ["C557", "M783"]}
Seed

wrong_schema_or_value

{"ids": ["M783", "C557", "T297", "Q592", "B712"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C557","M783"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C557","M783"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C557","M783"]}
16 tokens
0917 · lookup · wording 1 · same42fe244c733c8ace4068614a1b8baee52a0ac4915df1cd2691cd345ab7a4daf9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id L946 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"L946","city":"Lima","units":18},{"id":"M554","city":"Perth","units":6},{"id":"X689","city":"Oslo","units":8},{"id":"O656","city":"Accra","units":13},{"id":"J998","city":"Bern","units":9}]
Checked answer
{"city": "Lima", "units": 18}
Seed

Correct

{"city":"Lima","units":18}
12 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":18}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":18}
12 tokens
0918 · filter · wording 1 · samed6f8a9c3a22e2926f88f165ea0c60d351d993f1c3033233dcb9d56256b71371f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"W720","status":"closed"},{"id":"Z264","status":"ready"},{"id":"M229","status":"pending"},{"id":"V401","status":"ready"},{"id":"Q358","status":"closed"}]
Checked answer
{"ids": ["V401", "Z264"]}
Seed

wrong_schema_or_value

{
  "ids": ["M229", "Q358", "Z264", "W720"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["V401","Z264"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["V401","Z264"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["Z264","V401"]}
16 tokens
0919 · total · wording 1 · samebfd50de9b54def07a8c1596d9659b89d018dbf34fdb65b3e9119a058d48cd86d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"E856","units":8,"price":14},{"id":"M536","units":9,"price":7},{"id":"W157","units":16,"price":12},{"id":"H684","units":7,"price":18},{"id":"P235","units":9,"price":10}]
Checked answer
{"total": 583}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0920 · sort · wording 1 · gain88ad7b982a81095c648f81f304662ec296dc0f81c37899d6f27cb2be42a9e552
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"P133","priority":8},{"id":"A885","priority":8},{"id":"D317","priority":5},{"id":"O286","priority":2},{"id":"N503","priority":4}]
Checked answer
{"ids": ["A885", "P133"]}
Seed

invalid_json_object

[{"id":"P133","priority":8},{"id":"A885","priority":8},{"id":"D317","priority":5},{"id":"O286","priority":2},{"id":"N503","priority":4}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A885","P133"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A885","P133"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A885","P133"]}
16 tokens
0921 · lookup · wording 2 · samea4a13da3beba74884d04c7ef36ce4df3b384093f5b636ed225765bc779472862
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"D842","city":"Oslo","units":2},{"id":"X367","city":"Lima","units":1},{"id":"G279","city":"Kyoto","units":7},{"id":"N525","city":"Accra","units":16},{"id":"E518","city":"Accra","units":13}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0922 · filter · wording 2 · samebd532a25bd921f3b81c7c58e5ee6a1b8f9809c8d8a313d7a90a8ec3c8d29096c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"S563","status":"pending"},{"id":"I338","status":"closed"},{"id":"Y281","status":"ready"},{"id":"Y817","status":"closed"},{"id":"A835","status":"ready"}]
Checked answer
{"ids": ["A835", "Y281"]}
Seed

invalid_json_object

[{"id":"Y281","status":"ready"},{"id":"Y817","status":"closed"},{"id":"A835","status":"ready"},{"id":"S563","status":"pending"},{"id":"I338","status":"closed"}]
63 tokens
1 GPU · AdamW

Correct

{"ids":["A835","Y281"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A835","Y281"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["A835","Y281"]]
16 tokens
0923 · total · wording 2 · same96639a3f6f363d5f8ed49e3f7523c7ae72d96ba4d3a52bdddcb4098b68b7750a
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I407","units":7,"price":10},{"id":"V671","units":11,"price":8},{"id":"V517","units":3,"price":12},{"id":"A594","units":9,"price":16},{"id":"Z222","units":0,"price":19}]
Checked answer
{"total": 338}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":378}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0924 · sort · wording 2 · gaind939e1b00aec86059940e1bbacb04f4118b75ddf10094445f35bfe628664626b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"D291","priority":9},{"id":"D545","priority":3},{"id":"G821","priority":4},{"id":"P497","priority":1},{"id":"R215","priority":1}]
Checked answer
{"ids": ["D291", "G821"]}
Seed

wrong_schema_or_value

{"ids": ["D291", "P497", "R215", "G821", "D545"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D291","G821"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D291","G821"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D291","G821"]}
16 tokens
0925 · lookup · wording 3 · same752476f648917647fd030f30c64e5e47adc697bf61be9bc44f27cd8dc987a33d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=V577. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"V577","city":"Lima","units":1},{"id":"Z815","city":"Lima","units":5},{"id":"K604","city":"Perth","units":9},{"id":"Q167","city":"Suva","units":17},{"id":"Q235","city":"Kyoto","units":10}]
Checked answer
{"city": "Lima", "units": 1}
Seed

Correct

{"city":"Lima","units":1}
11 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":1}
11 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":1}
11 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":1}
11 tokens
0926 · filter · wording 3 · gain2631219b59caf54d529054a016e5ec3f20fe27f6134a7b67ca8243eec94ede34
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"K104","status":"pending"},{"id":"K358","status":"closed"},{"id":"X870","status":"ready"},{"id":"C739","status":"ready"},{"id":"Z757","status":"closed"}]
Checked answer
{"ids": ["C739", "X870"]}
Seed

wrong_schema_or_value

{"ids": ["X870", "C739", "K104", "K358", "Z757"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["C739","X870"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C739","X870"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["C739","X870"]}
16 tokens
0927 · total · wording 3 · sameeaa1edf101666ed618029b059d1e6b15bffa76046f6325af34d947b17373f8fb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"F126","units":0,"price":10},{"id":"N185","units":7,"price":15},{"id":"T928","units":15,"price":15},{"id":"G181","units":0,"price":3},{"id":"E920","units":3,"price":5}]
Checked answer
{"total": 345}
Seed

invalid_json_object

{
  "total": 10 + 105 + 135 + 45 + 15 = 285
}
34 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":485}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0928 · sort · wording 3 · gain0d208f9fc90456c08c86c9411607929e5e0086c238200bcf1adfc53e7ad7122f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"V706","priority":3},{"id":"T411","priority":4},{"id":"A134","priority":2},{"id":"W666","priority":2},{"id":"I888","priority":7}]
Checked answer
{"ids": ["I888", "T411"]}
Seed

invalid_json_object

[{"id":"V706","priority":3},{"id":"T411","priority":4},{"id":"A134","priority":2},{"id":"W666","priority":2},{"id":"I888","priority":7}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["I888","T411"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I888","T411"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I888","T411"]}
16 tokens
0929 · lookup · wording 0 · gain776b7f34d27541149b29580d4145be229b1261024f42f8894c6c99d8e3b25aab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record I637. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"L649","city":"Lima","units":19},{"id":"I637","city":"Oslo","units":15},{"id":"G681","city":"Lima","units":0},{"id":"V332","city":"Riga","units":0},{"id":"N615","city":"Suva","units":16}]
Checked answer
{"city": "Oslo", "units": 15}
Seed

wrong_schema_or_value

{
  "city": "Lima",
  "units": 19
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":15}
12 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":15}
12 tokens
0930 · filter · wording 0 · same5fc6eef8fd4e7bfda42c6e5bbef1083fd07645491c04c1d85a01cfde3d199004
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"B506","status":"ready"},{"id":"W393","status":"closed"},{"id":"E134","status":"ready"},{"id":"Y510","status":"pending"},{"id":"W792","status":"pending"}]
Checked answer
{"ids": ["B506", "E134"]}
Seed

wrong_schema_or_value

{"ids": ["B506", "E134", "Y510"]}
23 tokens
1 GPU · AdamW

Correct

{"ids":["B506","E134"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B506","E134"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E134","B506"]}
16 tokens
0931 · total · wording 0 · samee6ee7acbb223d9472e53a3afd6eb78d8429f067b861bcbcd9f594d9d326807e4
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"K196","units":8,"price":9},{"id":"T230","units":9,"price":5},{"id":"R203","units":4,"price":3},{"id":"X849","units":9,"price":11},{"id":"J181","units":7,"price":14}]
Checked answer
{"total": 326}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":348}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0932 · sort · wording 0 · gain2ed20e1ba9c2e4632ce2ed05433f42eb7b0ae2553ec27da63a525176a417abbc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Z426","priority":4},{"id":"A635","priority":8},{"id":"J294","priority":2},{"id":"J570","priority":5},{"id":"I713","priority":7}]
Checked answer
{"ids": ["A635", "I713"]}
Seed

wrong_schema_or_value

{"ids": ["J294", "J570", "J713", "A635", "Z426"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A635","I713"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A635","I713"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A635","I713"]}
16 tokens
0933 · lookup · wording 1 · same969d24e6670fdabd8ea43731347de002c1092b48e14f2a29099bc126676b0d20
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id O692 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"W875","city":"Oslo","units":4},{"id":"W216","city":"Perth","units":16},{"id":"R273","city":"Perth","units":19},{"id":"O692","city":"Suva","units":4},{"id":"W256","city":"Accra","units":2}]
Checked answer
{"city": "Suva", "units": 4}
Seed

Correct

{"city":"Suva","units":4}
11 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":4}
11 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":4}
11 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":4}
11 tokens
0934 · filter · wording 1 · same75574cef0eabb8138e4e3bec95144e980e783d4d590dee11110940a667dab654
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"D857","status":"ready"},{"id":"B927","status":"ready"},{"id":"V887","status":"closed"},{"id":"O187","status":"pending"},{"id":"M736","status":"ready"}]
Checked answer
{"ids": ["B927", "D857", "M736"]}
Seed

wrong_schema_or_value

{
  "ids": ["B927", "D857", "M736", "O187", "V887"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["B927","D857","M736"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["B927","D857","M736"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D857","M736","B927"]}
21 tokens
0935 · total · wording 1 · same152a92e2e0cf0cd3bed4629c10eea6daf4e5826eac14fa4499dc45e746868721
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"C120","units":3,"price":16},{"id":"O439","units":4,"price":4},{"id":"E722","units":18,"price":16},{"id":"A730","units":9,"price":17},{"id":"W971","units":6,"price":17}]
Checked answer
{"total": 607}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0936 · sort · wording 1 · gaind9bae17a12e78619aa3a2cf2405a81a3dd73f8581132911c7439211a8b008df5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V892","priority":3},{"id":"K120","priority":5},{"id":"V715","priority":1},{"id":"S136","priority":4},{"id":"U344","priority":2}]
Checked answer
{"ids": ["K120", "S136"]}
Seed

invalid_json_object

[{"id":"V892","priority":3},{"id":"U344","priority":2}]
26 tokens
1 GPU · AdamW

Correct

{"ids":["K120","S136"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K120","S136"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K120","S136"]}
16 tokens
0937 · lookup · wording 2 · gain674a6e5e10e635927da123224b68ef264855359e6afdf5d390a6b75d9a2288c9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to R989? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"T611","city":"Oslo","units":18},{"id":"G725","city":"Bern","units":14},{"id":"M189","city":"Perth","units":1},{"id":"R989","city":"Oslo","units":9},{"id":"R953","city":"Kyoto","units":11}]
Checked answer
{"city": "Oslo", "units": 9}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":9}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":9}
11 tokens
0938 · filter · wording 2 · same5663543912725965bda3f3a7adc544a3c273cfe4fd689dad04cd5f2bbd58b277
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"V816","status":"ready"},{"id":"U470","status":"pending"},{"id":"R607","status":"pending"},{"id":"V786","status":"pending"},{"id":"T902","status":"closed"}]
Checked answer
{"ids": ["V816"]}
Seed

invalid_json_object

[{"id":"V816"},{"id":"U470"},{"id":"R607"},{"id":"V786"},{"id":"T902"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["V816"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["V816"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["V816"]]
11 tokens
0939 · total · wording 2 · same6b60e5de737fd484504ac8c24142b6392ee3baf83202f60167ee358a2f3e7a4c
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"I877","units":5,"price":8},{"id":"Z199","units":11,"price":13},{"id":"I378","units":10,"price":4},{"id":"W653","units":17,"price":18},{"id":"C148","units":9,"price":13}]
Checked answer
{"total": 646}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0940 · sort · wording 2 · gaine987facadd6434494902eda5ca80725ec7cfb254f2bf35bfd76b2de1bf66d5d8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"W416","priority":4},{"id":"I540","priority":9},{"id":"N297","priority":8},{"id":"U218","priority":2},{"id":"S426","priority":3}]
Checked answer
{"ids": ["I540", "N297"]}
Seed

wrong_schema_or_value

{"ids": ["U218", "S426", "N297", "I540", "W416", "N297"]}
41 tokens
1 GPU · AdamW

Correct

{"ids":["I540","N297"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["I540","N297"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["I540","N297"]}
16 tokens
0941 · lookup · wording 3 · gain4d83eb168089646b83e2c7b9bd3f943692acc07a35ce9501f137e1453cafc957
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"E840","city":"Perth","units":7},{"id":"R632","city":"Accra","units":13},{"id":"R741","city":"Kyoto","units":13},{"id":"U348","city":"Oslo","units":5},{"id":"F417","city":"Bern","units":3}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Perth","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0942 · filter · wording 3 · gainb0d21c70ad90d922f33609d8168cd9b35b42ca522dde41688499bf2328366f72
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"D426","status":"pending"},{"id":"N584","status":"closed"},{"id":"S293","status":"ready"},{"id":"L913","status":"ready"},{"id":"I101","status":"pending"}]
Checked answer
{"ids": ["L913", "S293"]}
Seed

wrong_schema_or_value

{"ids": ["D426", "S293", "L913", "N584", "I101"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["L913","S293"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D426","L913","S293"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["L913","S293"]}
16 tokens
0943 · total · wording 3 · same534af701a7cda0f66b02864dcf366df8c21c47ceb772cd61e80e0a5eb289a989
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"E792","units":16,"price":19},{"id":"B750","units":2,"price":19},{"id":"G363","units":10,"price":6},{"id":"U591","units":9,"price":5},{"id":"G877","units":5,"price":15}]
Checked answer
{"total": 522}
Seed

invalid_json_object

{
  "total": 19 + 38 + 60 + 55 + 75 = 199
}
32 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":478}
8 tokens
0944 · sort · wording 3 · gaindde3f61fe09f99549af830b83387ab93cc3cb2ba16359730ff478a32a71543bd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"D374","priority":6},{"id":"Y562","priority":6},{"id":"S361","priority":7},{"id":"O752","priority":6},{"id":"H834","priority":6}]
Checked answer
{"ids": ["S361", "D374"]}
Seed

invalid_json_object

[{"id":"D374"},{"id":"Y562"},{"id":"S361"},{"id":"O752"},{"id":"H834"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":["S361","D374"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["S361","D374"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["S361","D374"]}
16 tokens
0945 · lookup · wording 0 · same6a826655ad4304cca5a512c282c25c45b326777aa9d187dc0cd7b009ad7faa0a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record C786. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"H730","city":"Accra","units":0},{"id":"C786","city":"Kyoto","units":15},{"id":"D939","city":"Bern","units":1},{"id":"E270","city":"Accra","units":19},{"id":"S268","city":"Suva","units":2}]
Checked answer
{"city": "Kyoto", "units": 15}
Seed

Correct

{
  "city": "Kyoto",
  "units": 15
}
20 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":15}
13 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":15}
13 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":15}
13 tokens
0946 · filter · wording 0 · samec26b3ff6ce62e2faaca8243d84ac283cc0401ea4b9d954c7f7074893ecad5a92
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"P267","status":"closed"},{"id":"V776","status":"ready"},{"id":"P674","status":"ready"},{"id":"F840","status":"closed"},{"id":"J102","status":"pending"}]
Checked answer
{"ids": ["P674", "V776"]}
Seed

wrong_schema_or_value

{"ids": ["P267", "V776", "P674", "F840", "J102"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["V776","P674"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["V776","P674"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J102","V776"]}
16 tokens
0947 · total · wording 0 · same1ccf2d8b386d783c70f23c11798d3007ec91571e498c765586def8cb2798b0f9
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"P771","units":18,"price":8},{"id":"A307","units":8,"price":5},{"id":"X422","units":14,"price":13},{"id":"R224","units":15,"price":9},{"id":"G857","units":0,"price":7}]
Checked answer
{"total": 501}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0948 · sort · wording 0 · gain0a87d20bc4a9ef93c6c62a6a4129b6e836f0a285cacb75f52bbd30d6a506a9cc
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"U743","priority":5},{"id":"Y246","priority":9},{"id":"E938","priority":9},{"id":"D631","priority":5},{"id":"F651","priority":5}]
Checked answer
{"ids": ["E938", "Y246"]}
Seed

wrong_schema_or_value

{"ids": ["U743", "F651", "D631", "E938", "Y246"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["E938","Y246"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["E938","Y246"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["E938","Y246"]}
16 tokens
0949 · lookup · wording 1 · gaincd447e37c27c7eed15174da006c8b5ce0f9fbfb88749a3f539b9bbac0ed448cd
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M536 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M613","city":"Lima","units":19},{"id":"T817","city":"Bern","units":13},{"id":"C764","city":"Lima","units":15},{"id":"C348","city":"Perth","units":2},{"id":"M536","city":"Riga","units":13}]
Checked answer
{"city": "Riga", "units": 13}
Seed

wrong_schema_or_value

{"city":"Lima","units":19}
12 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":13}
12 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":13}
12 tokens
0950 · filter · wording 1 · gain0ab791807413f212ffd099265095e7ee43a67575f68a143cb0cad29cfd8e6c90
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M950","status":"pending"},{"id":"T356","status":"ready"},{"id":"C339","status":"closed"},{"id":"R667","status":"closed"},{"id":"T947","status":"pending"}]
Checked answer
{"ids": ["T356"]}
Seed

wrong_schema_or_value

{
  "ids": ["M950", "T356", "C339", "R667", "T947"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["T356"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["T356"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["T356"]}
11 tokens
0951 · total · wording 1 · sameee5e5103242c5cf0957a5461d23a6660bffd7cad9e760e146e5f56d7531356e5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"V268","units":16,"price":11},{"id":"E874","units":19,"price":5},{"id":"P738","units":12,"price":14},{"id":"M789","units":17,"price":19},{"id":"O576","units":7,"price":10}]
Checked answer
{"total": 832}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0952 · sort · wording 1 · gain3d93a415a0bfff972c401c98d46e5f873723d6480872a9f8f44a7049c0ee4921
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"S111","priority":4},{"id":"O835","priority":4},{"id":"X555","priority":6},{"id":"T865","priority":9},{"id":"M904","priority":8}]
Checked answer
{"ids": ["T865", "M904"]}
Seed

invalid_json_object

[{"id":"S111","priority":4},{"id":"O835","priority":4},{"id":"X555","priority":6},{"id":"T865","priority":9},{"id":"M904","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["T865","M904"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["T865","M904"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["T865","M904"]}
16 tokens
0953 · lookup · wording 2 · gain3e79de9c69629644be442aafa203e7a9af80d1110e9b92c6e84c015213b863e6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to J852? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"J852","city":"Perth","units":17},{"id":"O510","city":"Kyoto","units":7},{"id":"X679","city":"Kyoto","units":13},{"id":"E393","city":"Oslo","units":5},{"id":"H334","city":"Riga","units":10}]
Checked answer
{"city": "Perth", "units": 17}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":17}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":17}
12 tokens
0954 · filter · wording 2 · same10ceef50fd752c1f8fafca1eb28dc5e62e33b14cdb5f54bd5a24b8fb8c1d9882
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"N496","status":"pending"},{"id":"Y338","status":"pending"},{"id":"U801","status":"closed"},{"id":"Q771","status":"closed"},{"id":"Z348","status":"closed"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"N496"},{"id":"Y338"},{"id":"U801"},{"id":"Q771"},{"id":"Z348"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

invalid_json_object

["ids":[]}
6 tokens
0955 · total · wording 2 · samec97095855e7ffe6f2464216f295f2d8c2dddd43560b63e0f37fae0daddd55989
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"F185","units":17,"price":9},{"id":"V223","units":19,"price":3},{"id":"B676","units":12,"price":12},{"id":"U769","units":5,"price":2},{"id":"N246","units":18,"price":5}]
Checked answer
{"total": 454}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0956 · sort · wording 2 · same44b3a4edb5d6892b6fb871df29fcee20e1c64a635f6e0b3e5e3c1c32a5cb320c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J922","priority":8},{"id":"G422","priority":8},{"id":"N196","priority":7},{"id":"E556","priority":7},{"id":"V110","priority":6}]
Checked answer
{"ids": ["G422", "J922"]}
Seed

wrong_schema_or_value

{"ids": ["J922", "G422"]}
17 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["J922","G422"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["J922","G422"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J922","G422"]}
16 tokens
0957 · lookup · wording 3 · samed52980e792d93ac44fb0e35f66a1efe5fd62abce418966373f07889795dbfd56
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=H733. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"H733","city":"Kyoto","units":2},{"id":"D472","city":"Riga","units":9},{"id":"N488","city":"Riga","units":8},{"id":"W396","city":"Bern","units":15},{"id":"P986","city":"Riga","units":19}]
Checked answer
{"city": "Kyoto", "units": 2}
Seed

Correct

{"city":"Kyoto","units":2}
12 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":2}
12 tokens
0958 · filter · wording 3 · gain259564e41af72e47b3d1e7a4b2781b79f7576983271c49d313e618854a7e3690
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"V133","status":"closed"},{"id":"H804","status":"ready"},{"id":"V339","status":"ready"},{"id":"C770","status":"pending"},{"id":"Z295","status":"pending"}]
Checked answer
{"ids": ["H804", "V339"]}
Seed

wrong_schema_or_value

{"ids": ["H804", "V133", "V339", "C770", "Z295"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["H804","V339"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["H804","V339"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["H804","V339"]}
16 tokens
0959 · total · wording 3 · same06abf58a20f4098dffa86c0f0108a0bbc98ddad9e0f7365543e223ac17130121
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"B805","units":5,"price":3},{"id":"A743","units":1,"price":14},{"id":"W818","units":14,"price":4},{"id":"A753","units":17,"price":15},{"id":"C192","units":17,"price":1}]
Checked answer
{"total": 357}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0960 · sort · wording 3 · samed49ca794890633d96cc5257968fb02e433e6a78bc5a3d78ea08c4061f1765d98
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"O136","priority":4},{"id":"P737","priority":9},{"id":"W444","priority":4},{"id":"G316","priority":7},{"id":"W160","priority":9}]
Checked answer
{"ids": ["P737", "W160"]}
Seed

invalid_json_object

[{"id":"O136"},{"id":"P737"},{"id":"W444"},{"id":"G316"},{"id":"W160"}]
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["W160","P737"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["W160","P737"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O136","W160"]}
16 tokens
0961 · lookup · wording 0 · gain1ea8ea79c527d4aa9af3a7e7583f30b86d732b9c6070942e41c15d1eb0cb949d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record Z000. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"P188","city":"Oslo","units":14},{"id":"P912","city":"Oslo","units":12},{"id":"U762","city":"Riga","units":17},{"id":"L175","city":"Oslo","units":11},{"id":"C219","city":"Perth","units":17}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{
  "city": "Oslo",
  "units": 14
}
19 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0962 · filter · wording 0 · gain46e279499a931b161a4996e92a88513674e1d73c994cf08f8ea2a2629d661375
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"Q926","status":"pending"},{"id":"M753","status":"closed"},{"id":"Y159","status":"ready"},{"id":"Z261","status":"ready"},{"id":"P958","status":"closed"}]
Checked answer
{"ids": ["Y159", "Z261"]}
Seed

wrong_schema_or_value

{"ids": ["Y159", "Z261", "P958", "M753", "Q926"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Y159","Z261"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["Y159","Z261"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Y159","Z261"]}
16 tokens
0963 · total · wording 0 · samec28cf881654617d4522f452cb4314e64d01b5a7ffa1f1b8a41e6251cbb561229
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"Y581","units":15,"price":3},{"id":"N285","units":16,"price":2},{"id":"T470","units":11,"price":14},{"id":"L658","units":2,"price":10},{"id":"X656","units":14,"price":5}]
Checked answer
{"total": 321}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0964 · sort · wording 0 · same91de7fe4cec0ad2096706d14b35212b8b54a44da1fdd6aaf7dc33e4e362f1892
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"G497","priority":4},{"id":"G798","priority":2},{"id":"P806","priority":8},{"id":"M790","priority":7},{"id":"U340","priority":9}]
Checked answer
{"ids": ["U340", "P806"]}
Seed

wrong_schema_or_value

{"ids": ["G798", "G497", "P806", "M790", "U340"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["M790","U340"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["M790","U340"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["U340","M790"]}
16 tokens
0965 · lookup · wording 1 · samec6153e2d1cf4977024e85532e81099045f5aa5e8c637ce53e40417412940f78a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id N415 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"N415","city":"Lima","units":11},{"id":"F817","city":"Bern","units":17},{"id":"E380","city":"Lima","units":16},{"id":"X385","city":"Kyoto","units":2},{"id":"U750","city":"Oslo","units":3}]
Checked answer
{"city": "Lima", "units": 11}
Seed

Correct

{"city":"Lima","units":11}
12 tokens
1 GPU · AdamW

Correct

{"city":"Lima","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Lima","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Lima","units":11}
12 tokens
0966 · filter · wording 1 · gain0e287e3aa27f6b1eb7ecb5d98eddca5fe28ee653ff0024071aca79eb5694a388
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"N368","status":"pending"},{"id":"O183","status":"closed"},{"id":"Z485","status":"closed"},{"id":"P866","status":"pending"},{"id":"A666","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{
  "ids": ["A666", "P866", "N368", "O183", "Z485"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0967 · total · wording 1 · samed9853aec54eb411bb0127e701876574af3930813e19477136b168e8e54857491
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"F813","units":19,"price":19},{"id":"X231","units":16,"price":2},{"id":"B574","units":15,"price":6},{"id":"J301","units":11,"price":13},{"id":"H837","units":6,"price":13}]
Checked answer
{"total": 704}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0968 · sort · wording 1 · same80a7408175af0b65ae7b146768e7b94baa19c7b919f1dc4ee4dc1fa20caf9c1b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"V510","priority":5},{"id":"O323","priority":9},{"id":"M236","priority":9},{"id":"A466","priority":9},{"id":"M714","priority":5}]
Checked answer
{"ids": ["A466", "M236"]}
Seed

invalid_json_object

[{"id":"V510","priority":5},{"id":"O323","priority":9},{"id":"M236","priority":9},{"id":"A466","priority":9},{"id":"M714","priority":5}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A466","M236"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A466","M236"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["M236","O323"]}
16 tokens
0969 · lookup · wording 2 · gain865e60c7b85d5602b2e6e65bcfac46118347b181183b0f72a33682cc5b02846a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to G758? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"W865","city":"Suva","units":10},{"id":"G758","city":"Oslo","units":0},{"id":"M632","city":"Lima","units":13},{"id":"B419","city":"Riga","units":9},{"id":"P937","city":"Bern","units":17}]
Checked answer
{"city": "Oslo", "units": 0}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":0}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":0}
11 tokens
0970 · filter · wording 2 · gaindc9807ba63212318113d3f0d128e9e997d352a6b866720a0fa1863ea82fb6d86
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"J743","status":"closed"},{"id":"N716","status":"pending"},{"id":"P354","status":"closed"},{"id":"Z155","status":"closed"},{"id":"S550","status":"pending"}]
Checked answer
{"ids": []}
Seed

invalid_json_object

[{"id":"J743"},{"id":"N716"},{"id":"P354"},{"id":"Z155"},{"id":"S550"}]
43 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0971 · total · wording 2 · same6d9ac457b2197a9d57b63785005755187311d56f06153a6fd5c30be9548a2995
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"A597","units":1,"price":15},{"id":"S642","units":6,"price":15},{"id":"V595","units":14,"price":6},{"id":"O277","units":16,"price":6},{"id":"Y442","units":6,"price":7}]
Checked answer
{"total": 327}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0972 · sort · wording 2 · gain75a46120a88dd1d7f8d4e92932f76f5b4519fa7114a9927a52ac8b391c94bef9
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"J658","priority":3},{"id":"R143","priority":9},{"id":"A339","priority":7},{"id":"U144","priority":3},{"id":"Q649","priority":6}]
Checked answer
{"ids": ["R143", "A339"]}
Seed

wrong_schema_or_value

{"ids": ["U144", "A339", "Q649", "J658", "R143", "J658"]}
41 tokens
1 GPU · AdamW

Correct

{"ids":["R143","A339"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["R143","A339"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["R143","A339"]}
16 tokens
0973 · lookup · wording 3 · same2e29494673f5e977edcd386e9e4e4a9b9997c01fc26b222e1045e74234c8c169
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z665. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"S823","city":"Oslo","units":10},{"id":"U249","city":"Kyoto","units":14},{"id":"W663","city":"Accra","units":2},{"id":"Z665","city":"Perth","units":0},{"id":"E928","city":"Suva","units":13}]
Checked answer
{"city": "Perth", "units": 0}
Seed

Correct

{"city":"Perth","units":0}
11 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":0}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":0}
10 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":0}
11 tokens
0974 · filter · wording 3 · gain4da67c31c77235ac5bb35f41a9bfc3486f6ab0a95809903aba522ae663512e53
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"S976","status":"pending"},{"id":"F493","status":"pending"},{"id":"A449","status":"pending"},{"id":"P274","status":"pending"},{"id":"F339","status":"pending"}]
Checked answer
{"ids": []}
Seed

wrong_schema_or_value

{"ids": ["A449", "F339", "F493", "S976", "P274"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":[]}
6 tokens
4 GPUs · DDP

Correct

{"ids":[]}
6 tokens
4 GPUs · local windows

Correct

{"ids":[]}
6 tokens
0975 · total · wording 3 · samedd78e2f7dea031a225039012a064428f8284fa7f37316aa9962e3cc6dbaada9d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"C287","units":7,"price":19},{"id":"C227","units":11,"price":19},{"id":"D965","units":9,"price":16},{"id":"H810","units":18,"price":2},{"id":"L646","units":0,"price":3}]
Checked answer
{"total": 522}
Seed

invalid_json_object

{
  "total": 19 * 7 + 19 * 11 + 16 * 9 + 2 * 18 + 3 * 0
}
42 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":468}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0976 · sort · wording 3 · samefc2a54b06ec539d925346be120cf2fb645a5a09c9a76bc2c5b13d717844695f8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"T895","priority":1},{"id":"O259","priority":9},{"id":"A773","priority":7},{"id":"N250","priority":8},{"id":"Q138","priority":3}]
Checked answer
{"ids": ["O259", "N250"]}
Seed

invalid_json_object

[{"id":"T895","priority":1},{"id":"O259","priority":9},{"id":"A773","priority":7},{"id":"N250","priority":8},{"id":"Q138","priority":3}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["O259","N250"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["N250","O259"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O259","A773"]}
16 tokens
0977 · lookup · wording 0 · samec270f9b1d5432ff973ebe918d597ce70280a425ac8a64ef0572af86bfae37d94
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record U259. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"M874","city":"Lima","units":0},{"id":"U259","city":"Suva","units":11},{"id":"G515","city":"Bern","units":2},{"id":"D976","city":"Perth","units":14},{"id":"G914","city":"Suva","units":0}]
Checked answer
{"city": "Suva", "units": 11}
Seed

Correct

{
  "city": "Suva",
  "units": 11
}
19 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":11}
12 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":11}
12 tokens
0978 · filter · wording 0 · same9177c387273503d0afb9a9cf55f13f012951bca54151be06e26f1a04e8fea48b
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"D444","status":"pending"},{"id":"P835","status":"closed"},{"id":"H800","status":"ready"},{"id":"B797","status":"ready"},{"id":"W114","status":"closed"}]
Checked answer
{"ids": ["B797", "H800"]}
Seed

wrong_schema_or_value

{"ids": ["H800", "B797", "D444", "P835", "W114"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["B797","H800"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B797","H800"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["H800","B797"]}
16 tokens
0979 · total · wording 0 · same42b19502511b8e165afd368ad1f50e9e48ad6db209eaae9ad84f5f96f5be053a
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"U116","units":2,"price":2},{"id":"D177","units":0,"price":5},{"id":"D789","units":5,"price":6},{"id":"H212","units":8,"price":6},{"id":"R561","units":11,"price":5}]
Checked answer
{"total": 137}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":28}
7 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":278}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0980 · sort · wording 0 · same7bf5f59b450f1049a9d842efb8b79e7b69c7c350236445d31d6f84954ddbc35c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"Y839","priority":2},{"id":"R227","priority":5},{"id":"V691","priority":6},{"id":"D446","priority":8},{"id":"D343","priority":6}]
Checked answer
{"ids": ["D446", "D343"]}
Seed

wrong_schema_or_value

{"ids": ["R227", "Y839", "V691", "D446", "D343"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["D446","R227"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["D446","R227"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["D446","V691"]}
16 tokens
0981 · lookup · wording 1 · gain877a4f406fd6607c55ae655e4792d42b2e32b487b84ff2c2cd021ccd8d494487
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id Z000 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"F635","city":"Riga","units":6},{"id":"O736","city":"Accra","units":11},{"id":"X224","city":"Suva","units":13},{"id":"I416","city":"Kyoto","units":1},{"id":"O333","city":"Perth","units":0}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Riga","units":6}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
0982 · filter · wording 1 · samefd4f0c07965bf5e3e586f418d21b82b1b73b4e907cfc94850c7818eccbf2f62d
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"O825","status":"ready"},{"id":"S429","status":"closed"},{"id":"Q645","status":"pending"},{"id":"G454","status":"closed"},{"id":"M807","status":"ready"}]
Checked answer
{"ids": ["M807", "O825"]}
Seed

wrong_schema_or_value

{
  "ids": ["O825", "S429", "G454", "M807"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["M807","O825"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M807","O825"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O825","M807"]}
16 tokens
0983 · total · wording 1 · same54e191c5ae9bb31ee4a1e686599efe3f80e1a5881caf737d8d4ddb826f0319ac
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"D842","units":0,"price":12},{"id":"B162","units":17,"price":3},{"id":"U292","units":19,"price":14},{"id":"F764","units":8,"price":17},{"id":"L374","units":1,"price":19}]
Checked answer
{"total": 472}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0984 · sort · wording 1 · gaind4237cd6d3407bef5c5aa963667e28de707c56085c10b7b409188401a6be214e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"U453","priority":8},{"id":"B987","priority":3},{"id":"I461","priority":1},{"id":"W565","priority":7},{"id":"M744","priority":8}]
Checked answer
{"ids": ["M744", "U453"]}
Seed

invalid_json_object

[{"id":"U453","priority":8},{"id":"B987","priority":3},{"id":"I461","priority":1},{"id":"W565","priority":7},{"id":"M744","priority":8}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["M744","U453"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["M744","U453"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["M744","U453"]}
16 tokens
0985 · lookup · wording 2 · gain562e918ae254dbfd744a5401643ee5950265e7a0df3c543e5a6f67041f11f610
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to C365? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"C365","city":"Kyoto","units":5},{"id":"N506","city":"Riga","units":14},{"id":"P944","city":"Accra","units":0},{"id":"A133","city":"Oslo","units":16},{"id":"Z597","city":"Riga","units":0}]
Checked answer
{"city": "Kyoto", "units": 5}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

wrong_schema_or_value

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":5}
12 tokens
0986 · filter · wording 2 · same31909e7e3d6246dd65c9205ed53076120fc278ba75b785b35d24d095c45f4a6c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"X747","status":"pending"},{"id":"C940","status":"ready"},{"id":"S955","status":"pending"},{"id":"Y430","status":"ready"},{"id":"F898","status":"closed"}]
Checked answer
{"ids": ["C940", "Y430"]}
Seed

invalid_json_object

[{"id":"C940","status":"ready"},{"id":"Y430","status":"ready"},{"id":"F898","status":"closed"}]
39 tokens
1 GPU · AdamW

Correct

{"ids":["C940","Y430"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["C940","Y430"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["C940","Y430"]]
16 tokens
0987 · total · wording 2 · same72940af7766c8c0313bb515658baec49490295d5b0f8288d86dc4c25b4089b45
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"A451","units":12,"price":9},{"id":"V112","units":9,"price":5},{"id":"T586","units":12,"price":11},{"id":"Z486","units":12,"price":4},{"id":"X336","units":11,"price":8}]
Checked answer
{"total": 421}
Seed

wrong_schema_or_value

{"total": 120}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0988 · sort · wording 2 · gain0ea4717768cd205a86b51782200482764b62e424e5180c3fad7d4b493846b623
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"N190","priority":1},{"id":"W374","priority":4},{"id":"D678","priority":8},{"id":"W362","priority":5},{"id":"R311","priority":8}]
Checked answer
{"ids": ["D678", "R311"]}
Seed

wrong_schema_or_value

{"ids": ["N190", "W374", "D678", "R311", "W362"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["D678","R311"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["D678","R311"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["D678","R311"]}
16 tokens
0989 · lookup · wording 3 · gainf9f3ab6e6b8be11a58109aecd50706ed761dea3bc4325cf729d7b7f3ccb2ce1f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=G560. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P844","city":"Oslo","units":7},{"id":"G560","city":"Perth","units":19},{"id":"O371","city":"Suva","units":18},{"id":"H971","city":"Bern","units":17},{"id":"T852","city":"Perth","units":3}]
Checked answer
{"city": "Perth", "units": 19}
Seed

wrong_schema_or_value

{"city":"Oslo","units":7}
11 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":19}
12 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":19}
12 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":19}
12 tokens
0990 · filter · wording 3 · gain089bfefe8ba83b39ffff7f9ef1bab4ad0a4de9de0ba3e992232edd24ffc4709f
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"X678","status":"pending"},{"id":"O813","status":"ready"},{"id":"G474","status":"pending"},{"id":"Y674","status":"pending"},{"id":"F153","status":"ready"}]
Checked answer
{"ids": ["F153", "O813"]}
Seed

wrong_schema_or_value

{"ids": ["O813", "G474", "F153", "X678", "Y674"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F153","O813"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F153","O813"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F153","O813"]}
16 tokens
0991 · total · wording 3 · same84e3785ec8b3841bd9d892e47d2e8a543a22ae30ddec1ef0e6318b7110dae77e
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"K292","units":9,"price":18},{"id":"B109","units":11,"price":13},{"id":"S749","units":6,"price":18},{"id":"U331","units":13,"price":5},{"id":"K125","units":6,"price":17}]
Checked answer
{"total": 580}
Seed

wrong_schema_or_value

{
  "total": 122
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
0992 · sort · wording 3 · gainf8ddc4fe848821c4687bf4747e4647807a5ba7ba9a1f6581c20d53e54fe5c5fb
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"O119","priority":6},{"id":"U812","priority":5},{"id":"G876","priority":2},{"id":"Z420","priority":9},{"id":"U283","priority":3}]
Checked answer
{"ids": ["Z420", "O119"]}
Seed

invalid_json_object

[{"id":"U283","priority":3},{"id":"U812","priority":5},{"id":"G876","priority":2},{"id":"O119","priority":6},{"id":"Z420","priority":9}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["O119","Z420"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["O119","Z420"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["Z420","O119"]}
16 tokens
0993 · lookup · wording 0 · same40161abd92114749fd9f638a0f7fa5d3451f1cf25acf45539b45916bb455add8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record J940. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"J940","city":"Oslo","units":8},{"id":"L478","city":"Bern","units":8},{"id":"M455","city":"Riga","units":1},{"id":"A711","city":"Lima","units":2},{"id":"V731","city":"Suva","units":18}]
Checked answer
{"city": "Oslo", "units": 8}
Seed

Correct

{
  "city": "Oslo",
  "units": 8
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Oslo","units":8}
11 tokens
4 GPUs · DDP

Correct

{"city":"Oslo","units":8}
11 tokens
4 GPUs · local windows

Correct

{"city":"Oslo","units":8}
11 tokens
0994 · filter · wording 0 · gain9d2ed41c25f08f91c01a1f9d956e2e707fc7d49b908682eb2b6b0abf94aee992
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"K775","status":"closed"},{"id":"W397","status":"pending"},{"id":"Y665","status":"ready"},{"id":"G129","status":"closed"},{"id":"M295","status":"closed"}]
Checked answer
{"ids": ["Y665"]}
Seed

wrong_schema_or_value

{"ids": ["Y665", "G129", "M295", "K775", "W397"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["Y665"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["Y665"]}
11 tokens
4 GPUs · local windows

Correct

{"ids":["Y665"]}
11 tokens
0995 · total · wording 0 · samee867d7c70db051c27e12110e17fb11cad1de5555a36c8aec478e1b5e9b61a79b
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"H449","units":8,"price":16},{"id":"S516","units":15,"price":15},{"id":"L684","units":4,"price":3},{"id":"I455","units":5,"price":7},{"id":"I495","units":6,"price":12}]
Checked answer
{"total": 472}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
0996 · sort · wording 0 · gainaa0ddc720d5dcf015ba0933b3c97698762bb9de05204eeb8f2043b5ae18c8385
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"J300","priority":1},{"id":"N564","priority":3},{"id":"I823","priority":1},{"id":"W323","priority":2},{"id":"F616","priority":8}]
Checked answer
{"ids": ["F616", "N564"]}
Seed

wrong_schema_or_value

{"ids": ["J300", "I823", "W323", "F616"]}
29 tokens
1 GPU · AdamW

Correct

{"ids":["F616","N564"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F616","N564"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F616","N564"]}
16 tokens
0997 · lookup · wording 1 · gain5605f8004b493e15c8603200f694686e04c6ee36dc2d857af61a39a799c9ec96
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id S579 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"D662","city":"Accra","units":2},{"id":"S579","city":"Perth","units":5},{"id":"D107","city":"Accra","units":16},{"id":"A529","city":"Suva","units":12},{"id":"E524","city":"Accra","units":12}]
Checked answer
{"city": "Perth", "units": 5}
Seed

wrong_schema_or_value

{"city":"Accra","units":16}
12 tokens
1 GPU · AdamW

Correct

{"city":"Perth","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Perth","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Perth","units":5}
11 tokens
0998 · filter · wording 1 · same59314d12d0b7c8ad93a563a90c9565100a2bc207a4e3238888e0bc461b2cd631
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"M662","status":"closed"},{"id":"N643","status":"ready"},{"id":"P162","status":"closed"},{"id":"B803","status":"ready"},{"id":"C823","status":"ready"}]
Checked answer
{"ids": ["B803", "C823", "N643"]}
Seed

wrong_schema_or_value

{
  "ids": ["B803", "C823", "M662", "N643", "P162"]
}
38 tokens
1 GPU · AdamW

Correct

{"ids":["B803","C823","N643"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["B803","C823","N643"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["C823","N643","B803"]}
21 tokens
0999 · total · wording 1 · samefe3009e8f83b897398d21d0b40285944b26114bb690f13e295e94f09f81c28a5
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"B893","units":16,"price":19},{"id":"A270","units":6,"price":9},{"id":"K871","units":7,"price":5},{"id":"R813","units":7,"price":14},{"id":"L765","units":13,"price":10}]
Checked answer
{"total": 621}
Seed

wrong_schema_or_value

{"total": 103}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
1000 · sort · wording 1 · gain7ad7a0ed3cc3ba29cce8e824fb96e17f6b61728147e23c48fe8be4e8671bb4ee
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"A706","priority":4},{"id":"M487","priority":7},{"id":"D221","priority":5},{"id":"T626","priority":6},{"id":"K405","priority":9}]
Checked answer
{"ids": ["K405", "M487"]}
Seed

invalid_json_object

[{"id":"A706","priority":4},{"id":"M487","priority":7},{"id":"D221","priority":5},{"id":"T626","priority":6},{"id":"K405","priority":9}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["K405","M487"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["K405","M487"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["K405","M487"]}
16 tokens
1001 · lookup · wording 2 · samecc56ab959389e077eb1bad0216dd20eb0971ab85ac7736d2e907408b94174a96
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to Z000? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"K962","city":"Kyoto","units":4},{"id":"B968","city":"Riga","units":14},{"id":"H984","city":"Lima","units":7},{"id":"P348","city":"Suva","units":2},{"id":"E503","city":"Kyoto","units":13}]
Checked answer
{"city": null, "units": null}
Seed

Correct

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
1002 · filter · wording 2 · same6b9fccfeef3dbb90ce742a95871c9ed96119c703ed5776de6686ab7f7ff3b10a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"U839","status":"closed"},{"id":"N287","status":"pending"},{"id":"P935","status":"pending"},{"id":"I270","status":"ready"},{"id":"X881","status":"closed"}]
Checked answer
{"ids": ["I270"]}
Seed

invalid_json_object

[{"id":"I270","status":"ready"}]
15 tokens
1 GPU · AdamW

Correct

{"ids":["I270"]}
11 tokens
4 GPUs · DDP

Correct

{"ids":["I270"]}
11 tokens
4 GPUs · local windows

invalid_json_object

["ids":["I270"]]
11 tokens
1003 · total · wording 2 · same178a7b084f82a1e97b7794723edcbcd726adaba97bffa7c26449c9b9b0e11d52
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"R854","units":19,"price":13},{"id":"V165","units":13,"price":5},{"id":"T877","units":17,"price":17},{"id":"Y186","units":10,"price":13},{"id":"Z115","units":0,"price":15}]
Checked answer
{"total": 731}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":648}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
1004 · sort · wording 2 · same337067e2de167ec01d600eec7e6feb4cf7e6e282f789ec9f76e0869ac5f4958a
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"L127","priority":5},{"id":"E102","priority":5},{"id":"D910","priority":5},{"id":"K526","priority":7},{"id":"C537","priority":2}]
Checked answer
{"ids": ["K526", "D910"]}
Seed

wrong_schema_or_value

{"ids": ["L127", "E102", "D910", "K526", "C537"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["K526","L127"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["K526","L127"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["K526","L127"]}
16 tokens
1005 · lookup · wording 3 · gain5394971b981e487af22477f182f4718836582855e3734872ca26448e8cfdda46
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=N920. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"F242","city":"Bern","units":4},{"id":"S919","city":"Suva","units":0},{"id":"Q505","city":"Bern","units":0},{"id":"Y161","city":"Oslo","units":5},{"id":"N920","city":"Kyoto","units":2}]
Checked answer
{"city": "Kyoto", "units": 2}
Seed

wrong_schema_or_value

{"city":"Bern","units":4}
10 tokens
1 GPU · AdamW

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · DDP

Correct

{"city":"Kyoto","units":2}
12 tokens
4 GPUs · local windows

Correct

{"city":"Kyoto","units":2}
12 tokens
1006 · filter · wording 3 · gain2c1d6f7c97cde238411d3f139f40ccd97bc62015f3675dab66f57347f8fa4101
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"V433","status":"ready"},{"id":"E925","status":"pending"},{"id":"F834","status":"ready"},{"id":"P223","status":"closed"},{"id":"H791","status":"pending"}]
Checked answer
{"ids": ["F834", "V433"]}
Seed

wrong_schema_or_value

{"ids": ["F834", "V433", "E925", "H791", "P223"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["F834","V433"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["F834","V433"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["F834","V433"]}
16 tokens
1007 · total · wording 3 · same2c7cfce445910037a12531fadaa375faaf936f1c9b82e183046914b30fec4bab
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"V886","units":4,"price":5},{"id":"T148","units":15,"price":10},{"id":"M872","units":9,"price":18},{"id":"U593","units":6,"price":1},{"id":"N842","units":12,"price":11}]
Checked answer
{"total": 470}
Seed

wrong_schema_or_value

{
  "total": 100
}
12 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":448}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
1008 · sort · wording 3 · gain49e5f61a84166df4988d94be9aa87a2ee4a4afe16048f8d24ce4f2f22fb45b42
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"K142","priority":5},{"id":"O510","priority":3},{"id":"C633","priority":3},{"id":"B251","priority":9},{"id":"Y747","priority":4}]
Checked answer
{"ids": ["B251", "K142"]}
Seed

invalid_json_object

[{"id":"K142","priority":5},{"id":"O510","priority":3},{"id":"C633","priority":3},{"id":"B251","priority":9},{"id":"Y747","priority":4}]
62 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["B251","C633"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["B251","K142"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["B251","K142"]}
16 tokens
1009 · lookup · wording 0 · same5ecea86127e78f8b02c3fec40e685263e0cd47cbffe1673567c0b7eebed0b0f0
Use only the records below. Return a single JSON object, without Markdown or commentary.
Find record G233. Return its city and units as JSON with exactly keys "city" and "units". Use null for both if the record is absent.
Records:
[{"id":"V284","city":"Bern","units":17},{"id":"G233","city":"Riga","units":5},{"id":"E102","city":"Perth","units":16},{"id":"I924","city":"Kyoto","units":0},{"id":"M745","city":"Kyoto","units":13}]
Checked answer
{"city": "Riga", "units": 5}
Seed

Correct

{
  "city": "Riga",
  "units": 5
}
18 tokens
1 GPU · AdamW

Correct

{"city":"Riga","units":5}
11 tokens
4 GPUs · DDP

Correct

{"city":"Riga","units":5}
11 tokens
4 GPUs · local windows

Correct

{"city":"Riga","units":5}
11 tokens
1010 · filter · wording 0 · same824ec9c4bc40964edd1c3fbad10c4ba3d08ac854a0f9d6dc70438321a6fe2164
Use only the records below. Return a single JSON object, without Markdown or commentary.
Select records whose status is exactly "ready". Return {"ids": [...]} with their ids in ascending alphabetical order.
Records:
[{"id":"G265","status":"ready"},{"id":"N938","status":"closed"},{"id":"R493","status":"closed"},{"id":"J797","status":"ready"},{"id":"A376","status":"ready"}]
Checked answer
{"ids": ["A376", "G265", "J797"]}
Seed

wrong_schema_or_value

{"ids": ["A376", "G265", "J797", "N938", "R493"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["A376","G265","J797"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["A376","G265","J797"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["G265","J797","A376"]}
21 tokens
1011 · total · wording 0 · same4ed1b0c667dda8793b88d708444f8b54da2d74fbb90a849826c3414febbe0235
Use only the records below. Return a single JSON object, without Markdown or commentary.
For each record multiply units by price, then add the products across all records. Return only {"total": integer}.
Records:
[{"id":"B455","units":7,"price":7},{"id":"G923","units":1,"price":12},{"id":"P300","units":6,"price":18},{"id":"F954","units":0,"price":2},{"id":"N647","units":17,"price":17}]
Checked answer
{"total": 458}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
1012 · sort · wording 0 · samee53b52d9f1156fb48b0f1fce1b908e9c554e36bc7e329754eeed76ea0970bdf6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Choose the two records with highest priority. Larger priority numbers rank first; break ties by ascending id. Return their ids in ranking order as {"ids": [...]}.
Records:
[{"id":"S637","priority":1},{"id":"R478","priority":7},{"id":"O956","priority":9},{"id":"X113","priority":6},{"id":"V313","priority":1}]
Checked answer
{"ids": ["O956", "R478"]}
Seed

wrong_schema_or_value

{"ids": ["S637", "R478", "X113", "V313", "O956"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["O956","R478"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["O956","R478"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["O956","S637"]}
16 tokens
1013 · lookup · wording 1 · gain10afc370716f8348956c8657ec878fbdbed6586ebbdf95d9392012bdaedfdea6
Use only the records below. Return a single JSON object, without Markdown or commentary.
Look up id M248 in the supplied records. Reply with {"city": city, "units": units} only; absent ids require two null values.
Records:
[{"id":"M248","city":"Suva","units":0},{"id":"B525","city":"Perth","units":17},{"id":"B383","city":"Perth","units":15},{"id":"H801","city":"Perth","units":7},{"id":"I941","city":"Kyoto","units":8}]
Checked answer
{"city": "Suva", "units": 0}
Seed

wrong_schema_or_value

{"city":"Suva","units":null}
11 tokens
1 GPU · AdamW

Correct

{"city":"Suva","units":0}
11 tokens
4 GPUs · DDP

Correct

{"city":"Suva","units":0}
11 tokens
4 GPUs · local windows

Correct

{"city":"Suva","units":0}
11 tokens
1014 · filter · wording 1 · sameacff76dd0c2f135f7f4fb9327ccc53db528178f27201c0fed683f00425fcd520
Use only the records below. Return a single JSON object, without Markdown or commentary.
Reply with one JSON key "ids": the alphabetically sorted ids of all ready records. Exclude pending and closed records.
Records:
[{"id":"E579","status":"ready"},{"id":"Z379","status":"pending"},{"id":"Q268","status":"pending"},{"id":"C958","status":"ready"},{"id":"O903","status":"ready"}]
Checked answer
{"ids": ["C958", "E579", "O903"]}
Seed

wrong_schema_or_value

{
  "ids": ["C958", "E579", "O903", "Q268"]
}
32 tokens
1 GPU · AdamW

Correct

{"ids":["C958","E579","O903"]}
21 tokens
4 GPUs · DDP

Correct

{"ids":["C958","E579","O903"]}
21 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["E579","C958","O903"]}
21 tokens
1015 · total · wording 1 · same884203b93c3633bd0b911d4f41886a6e0fd689e5113d4db54da6a02409779a3f
Use only the records below. Return a single JSON object, without Markdown or commentary.
Calculate the total invoice value: sum(units * price) for every entry, irrespective of status. Answer as JSON with exactly one integer key "total".
Records:
[{"id":"Q159","units":1,"price":13},{"id":"V941","units":0,"price":19},{"id":"Z446","units":2,"price":15},{"id":"B481","units":15,"price":19},{"id":"V899","units":14,"price":10}]
Checked answer
{"total": 468}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":477}
8 tokens
1016 · sort · wording 1 · gain0bfd5e09ae4c51c9a9167abd0c97238e25f867ab87d30b0ea7b9c1b3f9339b70
Use only the records below. Return a single JSON object, without Markdown or commentary.
Rank rows by descending priority, then ascending identifier for equal priorities. Output the first two ids under the JSON key "ids".
Records:
[{"id":"M974","priority":5},{"id":"P256","priority":3},{"id":"O821","priority":1},{"id":"A825","priority":8},{"id":"L906","priority":1}]
Checked answer
{"ids": ["A825", "M974"]}
Seed

invalid_json_object

[{"id":"M974","priority":5},{"id":"P256","priority":3},{"id":"O821","priority":1},{"id":"A825","priority":8},{"id":"L906","priority":1}]
62 tokens
1 GPU · AdamW

Correct

{"ids":["A825","M974"]}
16 tokens
4 GPUs · DDP

Correct

{"ids":["A825","M974"]}
16 tokens
4 GPUs · local windows

Correct

{"ids":["A825","M974"]}
16 tokens
1017 · lookup · wording 2 · gaina917a7c1c1d9ac13ff53e762611657a633ca2c2aff5989a0a6daa9c7f7e893d8
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which city and unit count belong to F403? Produce only a JSON object with keys "city" and "units"; both must be null if no id matches.
Records:
[{"id":"E292","city":"Bern","units":7},{"id":"F403","city":"Bern","units":10},{"id":"S327","city":"Lima","units":13},{"id":"O231","city":"Suva","units":18},{"id":"X567","city":"Accra","units":0}]
Checked answer
{"city": "Bern", "units": 10}
Seed

wrong_schema_or_value

{
  "city": null,
  "units": null
}
15 tokens
1 GPU · AdamW

Correct

{"city":"Bern","units":10}
11 tokens
4 GPUs · DDP

Correct

{"city":"Bern","units":10}
11 tokens
4 GPUs · local windows

Correct

{"city":"Bern","units":10}
11 tokens
1018 · filter · wording 2 · samec20102f0e2ac5ea111b850a951a78f67f7154621e9f2f37fde2add5f6c990711
Use only the records below. Return a single JSON object, without Markdown or commentary.
Keep only entries marked ready, then alphabetize their identifiers. Output a JSON object containing only the array "ids".
Records:
[{"id":"T119","status":"pending"},{"id":"J148","status":"pending"},{"id":"P274","status":"closed"},{"id":"R661","status":"ready"},{"id":"Y922","status":"ready"}]
Checked answer
{"ids": ["R661", "Y922"]}
Seed

invalid_json_object

[{"id":"R661","status":"ready"},{"id":"Y922","status":"ready"}]
27 tokens
1 GPU · AdamW

invalid_json_object

["R661","Y922"]
12 tokens
4 GPUs · DDP

Correct

{"ids":["R661","Y922"]}
16 tokens
4 GPUs · local windows

invalid_json_object

["ids":["R661","Y922"]]
16 tokens
1019 · total · wording 2 · samea5fa6d6b8183790f8648e934053ed6e83a9e3364210351a660dffd60f7658422
Use only the records below. Return a single JSON object, without Markdown or commentary.
What is the combined value of all entries? Each contributes its unit count times its price. Output {"total": integer} without explanation.
Records:
[{"id":"B346","units":9,"price":15},{"id":"B624","units":16,"price":4},{"id":"C190","units":0,"price":10},{"id":"B207","units":16,"price":8},{"id":"R700","units":5,"price":10}]
Checked answer
{"total": 377}
Seed

wrong_schema_or_value

{"total": 100}
9 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":478}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
1020 · sort · wording 2 · same39b806f9a380d4a2932e3d9302ce1ba27fd821887e7f19064bdb2c52f3b1bd9c
Use only the records below. Return a single JSON object, without Markdown or commentary.
Which two identifiers lead when priority is sorted highest first and tied ids alphabetically? Return {"ids": [...]} in that order.
Records:
[{"id":"Y631","priority":6},{"id":"J690","priority":8},{"id":"R477","priority":4},{"id":"C949","priority":5},{"id":"F205","priority":6}]
Checked answer
{"ids": ["J690", "F205"]}
Seed

wrong_schema_or_value

{"ids": ["J690", "Y631", "R477", "C949", "F205"]}
35 tokens
1 GPU · AdamW

Correct

{"ids":["J690","F205"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["J690","Y631"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["J690","C949"]}
16 tokens
1021 · lookup · wording 3 · gain0e1885e281b79c6e2a841bea7b73677b9ec4486b1aa965dbd10f000b2c697833
Use only the records below. Return a single JSON object, without Markdown or commentary.
Extract city and units for id=Z000. Required output schema: {"city": string or null, "units": integer or null}. If missing, return null in both fields.
Records:
[{"id":"P767","city":"Suva","units":9},{"id":"N880","city":"Lima","units":11},{"id":"M637","city":"Perth","units":11},{"id":"O572","city":"Kyoto","units":5},{"id":"J452","city":"Bern","units":8}]
Checked answer
{"city": null, "units": null}
Seed

wrong_schema_or_value

{"city":"Suva","units":9}
11 tokens
1 GPU · AdamW

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · DDP

Correct

{"city":null,"units":null}
10 tokens
4 GPUs · local windows

Correct

{"city":null,"units":null}
10 tokens
1022 · filter · wording 3 · gain7f04cb000f4dfcf8f5b06cf3d7d53d9a19d79e693bddde68f23e0fc0ff0a52d3
Use only the records below. Return a single JSON object, without Markdown or commentary.
List every ready id, sorted lexicographically ascending. The answer must have the shape {"ids": [...]}.
Records:
[{"id":"H179","status":"closed"},{"id":"K608","status":"ready"},{"id":"P657","status":"closed"},{"id":"I233","status":"ready"},{"id":"L359","status":"ready"}]
Checked answer
{"ids": ["I233", "K608", "L359"]}
Seed

wrong_schema_or_value

{"ids": ["H179", "K608", "P657", "I233", "L359"]}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["I233","L359","K608"]}
21 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["I233","L359","K608"]}
21 tokens
4 GPUs · local windows

Correct

{"ids":["I233","K608","L359"]}
21 tokens
1023 · total · wording 3 · samef60354dfdedb0a391561e851cc3612242f829a4e4a7cc2d3f42f7f8b5fa1bd10
Use only the records below. Return a single JSON object, without Markdown or commentary.
Compute the sum of quantity-times-price using units as quantity for all listed rows. Use only the JSON field "total", with an integer value.
Records:
[{"id":"L303","units":18,"price":19},{"id":"L953","units":12,"price":9},{"id":"W563","units":14,"price":14},{"id":"L729","units":7,"price":2},{"id":"B116","units":18,"price":8}]
Checked answer
{"total": 804}
Seed

invalid_json_object

{
  "total": 19 + 156 + 78 + 114 + 174 = 406
}
35 tokens
1 GPU · AdamW

wrong_schema_or_value

{"total":578}
8 tokens
4 GPUs · DDP

wrong_schema_or_value

{"total":488}
8 tokens
4 GPUs · local windows

wrong_schema_or_value

{"total":407}
8 tokens
1024 · sort · wording 3 · same8ca14de0922151fad2636908d0175fe1e5b0eafd4bfd24c05250ada4ae3fec32
Use only the records below. Return a single JSON object, without Markdown or commentary.
Take the top two records using priority descending and id ascending as the tie-break. Answer with only a JSON object containing their ordered "ids".
Records:
[{"id":"X340","priority":1},{"id":"W160","priority":1},{"id":"T700","priority":7},{"id":"C575","priority":3},{"id":"B815","priority":3}]
Checked answer
{"ids": ["T700", "B815"]}
Seed

invalid_json_object

[{"id":"X340"},{"id":"W160"},{"id":"T700"},{"id":"C575"},{"id":"B815"}]
43 tokens
1 GPU · AdamW

wrong_schema_or_value

{"ids":["T700","C575"]}
16 tokens
4 GPUs · DDP

wrong_schema_or_value

{"ids":["T700","C575"]}
16 tokens
4 GPUs · local windows

wrong_schema_or_value

{"ids":["T700","C575"]}
16 tokens

LZ · Apache-2.0 · All final candidates and evaluations retained; no native promotion.