{"id":"reasoning-01","category":"reasoning","prompt":"A shop sells pencils at 3 for $0.75. How many cents do 14 pencils cost? Reply with only the final number, no units or working.","grader":{"type":"number","answer":350},"reference":"350"}
{"id":"reasoning-02","category":"reasoning","prompt":"A train leaves at 09:40 and arrives at 13:15 the same day. How many minutes does the trip take? Reply with only the final number, no units or working.","grader":{"type":"number","answer":215},"reference":"215"}
{"id":"reasoning-03","category":"reasoning","prompt":"Mara is three times as old as her son. In 12 years she will be twice as old as him. How old is Mara now, in years? Reply with only the final number, no units or working.","grader":{"type":"number","answer":36},"reference":"36"}
{"id":"reasoning-04","category":"reasoning","prompt":"Pipe A fills an empty tank in 6 hours and pipe B fills it in 4 hours. With both pipes open, how many minutes does it take to fill the empty tank? Reply with only the final number, no units or working.","grader":{"type":"number","answer":144},"reference":"144"}
{"id":"reasoning-05","category":"reasoning","prompt":"How many whole numbers from 1 to 100 inclusive are divisible by 3 or by 5 (or both)? Reply with only the final number, no units or working.","grader":{"type":"number","answer":47},"reference":"47"}
{"id":"reasoning-06","category":"reasoning","prompt":"Five friends meet and each pair shakes hands exactly once. How many handshakes happen? Reply with only the final number, no units or working.","grader":{"type":"number","answer":10},"reference":"10"}
{"id":"reasoning-07","category":"reasoning","prompt":"A recipe for 4 people uses 300 g of rice. How many grams of rice are needed for 10 people at the same ratio? Reply with only the final number, no units or working.","grader":{"type":"number","answer":750},"reference":"750"}
{"id":"reasoning-08","category":"reasoning","prompt":"Ana is taller than Ben. Ben is taller than Caro. Dev is shorter than Caro. Who is the second shortest of the four? Reply with only the name.","grader":{"type":"exact","answer":"Caro","case_insensitive":true},"reference":"Caro"}
{"id":"reasoning-09","category":"reasoning","prompt":"A jacket costs $80. It is discounted by 25%, then 10% sales tax is added to the discounted price. What is the final price in dollars? Reply with only the final number, no units or working.","grader":{"type":"number","answer":66,"tolerance":0.001},"reference":"66"}
{"id":"reasoning-10","category":"reasoning","prompt":"Today is a Wednesday. What day of the week will it be 100 days from today? Reply with only the day name.","grader":{"type":"exact","answer":"Friday","case_insensitive":true},"reference":"Friday"}
{"id":"extraction-01","category":"extraction","prompt":"Extract fields from this text as a JSON object with keys invoice_id (string), customer (string), total (number) and currency (ISO 4217 code).\n\nText: \"Invoice INV-20931 was issued on 3 March 2026 to Halvorsen Bakery for a total of €1,284.50, due in 30 days.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"invoice_id":{"type":"string","const":"INV-20931"},"customer":{"type":"string","const":"Halvorsen Bakery"},"total":{"type":"number","const":1284.5},"currency":{"type":"string","const":"EUR"}},"required":["invoice_id","customer","total","currency"],"additionalProperties":false}},"reference":"{\"invoice_id\":\"INV-20931\",\"customer\":\"Halvorsen Bakery\",\"total\":1284.5,\"currency\":\"EUR\"}"}
{"id":"extraction-02","category":"extraction","prompt":"Extract the contact details from this message as a JSON object with keys name, email and extension (all strings).\n\nMessage: \"Hi, it's Priya Natarajan from the Leeds office. Reach me at priya.n@example.org or on extension 4417.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"name":{"type":"string","const":"Priya Natarajan"},"email":{"type":"string","const":"priya.n@example.org"},"extension":{"type":"string","const":"4417"}},"required":["name","email","extension"],"additionalProperties":false}},"reference":"{\"name\":\"Priya Natarajan\",\"email\":\"priya.n@example.org\",\"extension\":\"4417\"}"}
{"id":"extraction-03","category":"extraction","prompt":"Extract the meeting details as a JSON object with keys weekday (string), time_24h (string, HH:MM), room (string, the room code only) and attendees (array of first names, in the order mentioned).\n\nText: \"Let's move the design review to Thursday at 3:30pm in Room B2; Tomas and Ines will join.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"weekday":{"type":"string","const":"Thursday"},"time_24h":{"type":"string","const":"15:30"},"room":{"type":"string","const":"B2"},"attendees":{"type":"array","const":["Tomas","Ines"]}},"required":["weekday","time_24h","room","attendees"],"additionalProperties":false}},"reference":"{\"weekday\":\"Thursday\",\"time_24h\":\"15:30\",\"room\":\"B2\",\"attendees\":[\"Tomas\",\"Ines\"]}"}
{"id":"extraction-04","category":"extraction","prompt":"Extract the shipment as a JSON object with keys order (string), origin (string), ship_date (string, YYYY-MM-DD), weight_kg (number) and carrier (string).\n\nText: \"Order 55-X ships from Rotterdam on 2 July 2026, weight 18.4 kg, carrier: Nordlink Freight.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"order":{"type":"string","const":"55-X"},"origin":{"type":"string","const":"Rotterdam"},"ship_date":{"type":"string","const":"2026-07-02"},"weight_kg":{"type":"number","const":18.4},"carrier":{"type":"string","const":"Nordlink Freight"}},"required":["order","origin","ship_date","weight_kg","carrier"],"additionalProperties":false}},"reference":"{\"order\":\"55-X\",\"origin\":\"Rotterdam\",\"ship_date\":\"2026-07-02\",\"weight_kg\":18.4,\"carrier\":\"Nordlink Freight\"}"}
{"id":"extraction-05","category":"extraction","prompt":"Extract the rating from this review as a JSON object with keys rating (number) and out_of (number).\n\nReview: \"Battery lasts two days and the speaker is loud, but the screen cracked within a week. I'd give it 2 out of 5.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"rating":{"type":"number","const":2},"out_of":{"type":"number","const":5}},"required":["rating","out_of"],"additionalProperties":false}},"reference":"{\"rating\":2,\"out_of\":5}"}
{"id":"extraction-06","category":"extraction","prompt":"List every year mentioned in this text, in ascending order, as a JSON object with a single key years (array of integers).\n\nText: \"The bridge opened in 1932, was closed in 1989 after a storm damaged it in 1987, and reopened in 2004.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"years":{"type":"array","const":[1932,1987,1989,2004]}},"required":["years"],"additionalProperties":false}},"reference":"{\"years\":[1932,1987,1989,2004]}"}
{"id":"extraction-07","category":"extraction","prompt":"Extract the phone number from this text as a JSON object with a single key phone_digits: a string of its digits only, no spaces or symbols.\n\nText: \"For bookings, call the front desk on (020) 7946 0018 between 9 and 5.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"phone_digits":{"type":"string","const":"02079460018"}},"required":["phone_digits"],"additionalProperties":false}},"reference":"{\"phone_digits\":\"02079460018\"}"}
{"id":"extraction-08","category":"extraction","prompt":"Parse this log line into a JSON object with keys level (string), service (string), message (string) and user_id (integer).\n\nLog: 2026-11-02T08:14:55Z level=ERROR service=billing msg=\"card declined\" user=8812\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"level":{"type":"string","const":"ERROR"},"service":{"type":"string","const":"billing"},"message":{"type":"string","const":"card declined"},"user_id":{"type":"integer","const":8812}},"required":["level","service","message","user_id"],"additionalProperties":false}},"reference":"{\"level\":\"ERROR\",\"service\":\"billing\",\"message\":\"card declined\",\"user_id\":8812}"}
{"id":"extraction-09","category":"extraction","prompt":"List the distinct people named in this text, in order of first mention, as a JSON object with a single key people (array of names).\n\nText: \"Olga emailed Sam; Sam forwarded it to Olga's manager, Rui, and later to Kim.\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"people":{"type":"array","const":["Olga","Sam","Rui","Kim"]}},"required":["people"],"additionalProperties":false}},"reference":"{\"people\":[\"Olga\",\"Sam\",\"Rui\",\"Kim\"]}"}
{"id":"extraction-10","category":"extraction","prompt":"Split this address into a JSON object with keys street, postal_code, city and country (all strings).\n\nAddress: \"Ship to: 41 Calle Mayor, 28013 Madrid, Spain\"\n\nReply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"street":{"type":"string","const":"41 Calle Mayor"},"postal_code":{"type":"string","const":"28013"},"city":{"type":"string","const":"Madrid"},"country":{"type":"string","const":"Spain"}},"required":["street","postal_code","city","country"],"additionalProperties":false}},"reference":"{\"street\":\"41 Calle Mayor\",\"postal_code\":\"28013\",\"city\":\"Madrid\",\"country\":\"Spain\"}"}
{"id":"instruction-01","category":"instruction","prompt":"Write exactly three bullet points about why sleep matters. Start each bullet with \"- \". Write nothing else: no heading, intro or closing line.","grader":{"type":"constraints","checks":[{"kind":"bullet_count","equals":3},{"kind":"line_count","equals":3}]},"reference":"- Sleep restores the body.\n- It consolidates memory.\n- It steadies mood and focus."}
{"id":"instruction-02","category":"instruction","prompt":"Write one sentence of at least six words about the ocean, using only lowercase letters and no commas.","grader":{"type":"constraints","checks":[{"kind":"all_lowercase"},{"kind":"no_char","char":","},{"kind":"word_count","min":6},{"kind":"line_count","equals":1}]},"reference":"the ocean covers most of our planet."}
{"id":"instruction-03","category":"instruction","prompt":"Describe a bicycle in 20 to 30 words. Do not use the words \"wheel\" or \"wheels\".","grader":{"type":"constraints","checks":[{"kind":"word_count","min":20,"max":30},{"kind":"excludes_words","words":["wheel","wheels"]}]},"reference":"A bicycle is a light, pedal-powered machine with a metal frame, handlebars for steering, a chain that drives it forward, and brakes for stopping safely."}
{"id":"instruction-04","category":"instruction","prompt":"Reply with the word OK in capital letters and nothing else.","grader":{"type":"exact","answer":"OK"},"reference":"OK"}
{"id":"instruction-05","category":"instruction","prompt":"List the eight planets of the solar system from closest to the Sun outward, separated by commas, on a single line, with nothing else.","grader":{"type":"regex","pattern":"^Mercury\\s*,\\s*Venus\\s*,\\s*Earth\\s*,\\s*Mars\\s*,\\s*Jupiter\\s*,\\s*Saturn\\s*,\\s*Uranus\\s*,\\s*Neptune\\.?$","flags":"i"},"reference":"Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune"}
{"id":"instruction-06","category":"instruction","prompt":"Write a short poem about rain in exactly three lines. The last line must end with the word \"roof\". No title.","grader":{"type":"constraints","checks":[{"kind":"line_count","equals":3},{"kind":"ends_with","text":"roof"}]},"reference":"Grey clouds gather slow\nsoft drops tap a steady beat\nsinging on the roof"}
{"id":"instruction-07","category":"instruction","prompt":"In exactly five words, answer: what is the capital of France?","grader":{"type":"constraints","checks":[{"kind":"word_count","min":5,"max":5},{"kind":"includes_words","words":["Paris"]}]},"reference":"The capital is Paris, France."}
{"id":"instruction-08","category":"instruction","prompt":"Write a short note that starts with \"Dear team,\" and ends with \"Best, Kim\". In between, write two sentences announcing that the office closes early on Friday.","grader":{"type":"constraints","checks":[{"kind":"starts_with","text":"Dear team,"},{"kind":"ends_with","text":"Best, Kim"},{"kind":"includes_words","words":["Friday"]}]},"reference":"Dear team,\nThe office will close early on Friday. Please plan your day accordingly.\nBest, Kim"}
{"id":"instruction-09","category":"instruction","prompt":"Spell the word \"reliable\" backwards in uppercase letters, with no spaces and nothing else.","grader":{"type":"exact","answer":"ELBAILER"},"reference":"ELBAILER"}
{"id":"instruction-10","category":"instruction","prompt":"Give a numbered list of four fruits that are usually red, one per line, formatted \"1. ...\" to \"4. ...\". Write nothing else.","grader":{"type":"constraints","checks":[{"kind":"numbered_count","equals":4},{"kind":"line_count","equals":4}]},"reference":"1. Strawberry\n2. Cherry\n3. Raspberry\n4. Pomegranate"}
{"id":"json-01","category":"json","prompt":"Return a JSON object for a fictional person with exactly these keys: name (string), age (integer) and tags (array of exactly 2 strings). Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","required":["name","age","tags"],"additionalProperties":false,"properties":{"name":{"type":"string","minLength":1},"age":{"type":"integer","minimum":0},"tags":{"type":"array","minItems":2,"maxItems":2,"items":{"type":"string"}}}}},"reference":"{\"name\":\"Rosa Lind\",\"age\":41,\"tags\":[\"cyclist\",\"chemist\"]}"}
{"id":"json-02","category":"json","prompt":"Return a JSON array of the first five prime numbers. Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"array","const":[2,3,5,7,11]}},"reference":"[2,3,5,7,11]"}
{"id":"json-03","category":"json","prompt":"Convert this reading to a JSON object with keys temperature_c (number), humidity_pct (number) and station (string): \"Temperature 21.5 C, humidity 40%, station ID WX-7.\" Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"temperature_c":{"type":"number","const":21.5},"humidity_pct":{"type":"number","const":40},"station":{"type":"string","const":"WX-7"}},"required":["temperature_c","humidity_pct","station"],"additionalProperties":false}},"reference":"{\"temperature_c\":21.5,\"humidity_pct\":40,\"station\":\"WX-7\"}"}
{"id":"json-04","category":"json","prompt":"Return a JSON object with a single key ok whose value is the boolean true. Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","const":{"ok":true}}},"reference":"{\"ok\":true}"}
{"id":"json-05","category":"json","prompt":"Return a JSON object that maps each of the letters a, b and c to its position in the English alphabet (1-based) as an integer. Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","const":{"a":1,"b":2,"c":3}}},"reference":"{\"a\":1,\"b\":2,\"c\":3}"}
{"id":"json-06","category":"json","prompt":"Return a JSON object describing a fictional book with exactly these keys: title (non-empty string), year (integer from 1800 to 2026) and authors (non-empty array of strings). Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","required":["title","year","authors"],"additionalProperties":false,"properties":{"title":{"type":"string","minLength":1},"year":{"type":"integer","minimum":1800,"maximum":2026},"authors":{"type":"array","minItems":1,"items":{"type":"string","minLength":1}}}}},"reference":"{\"title\":\"The Salt Road\",\"year\":1994,\"authors\":[\"Ines Varga\"]}"}
{"id":"json-07","category":"json","prompt":"Sort these words alphabetically and return them as a JSON array of strings: pear, apple, fig, banana. Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"array","const":["apple","banana","fig","pear"]}},"reference":"[\"apple\",\"banana\",\"fig\",\"pear\"]"}
{"id":"json-08","category":"json","prompt":"Return a JSON object with a single key items: an array of exactly 3 objects, each with exactly the keys sku (a string like \"ABC-123\": three capital letters, a hyphen, three digits) and qty (integer of at least 1). Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","required":["items"],"additionalProperties":false,"properties":{"items":{"type":"array","minItems":3,"maxItems":3,"items":{"type":"object","required":["sku","qty"],"additionalProperties":false,"properties":{"sku":{"type":"string","pattern":"^[A-Z]{3}-\\d{3}$"},"qty":{"type":"integer","minimum":1}}}}}}},"reference":"{\"items\":[{\"sku\":\"ABC-123\",\"qty\":2},{\"sku\":\"XYZ-900\",\"qty\":1},{\"sku\":\"KJN-001\",\"qty\":5}]}"}
{"id":"json-09","category":"json","prompt":"Count the vowels (a, e, i, o, u) in the word \"education\" and return a JSON object with keys word (the word) and vowels (the count as an integer). Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"word":{"type":"string","const":"education"},"vowels":{"type":"integer","const":5}},"required":["word","vowels"],"additionalProperties":false}},"reference":"{\"word\":\"education\",\"vowels\":5}"}
{"id":"json-10","category":"json","prompt":"A support ticket's latest note says: \"Customer replied; we're now waiting on engineering.\" Return a JSON object with a single key status set to exactly one of \"open\", \"closed\" or \"pending\", whichever fits best. Reply with only the JSON, no code fence or commentary.","grader":{"type":"json_schema","schema":{"type":"object","properties":{"status":{"type":"string","const":"pending"}},"required":["status"],"additionalProperties":false}},"reference":"{\"status\":\"pending\"}"}
