Working with Model Outputs
14 min read
A language model answers with text. Your app needs records: a task with a due date, a quiz question with four options, a calendar entry with a start and an end. The gap between those two things is where most agents are actually built - and it is the part that decides whether an app is reliable or only looks reliable in a demo.
The good news is that the model is not the problem. It will happily produce exactly the shape you ask for most of the time. The problem is 'most of the time': the field that arrives as the word 'urgent!' instead of a priority, the date that arrives as 'next Friday', the quiz question with three options when you asked for four. Each of those is a tiny failure that becomes a broken screen, a wrong sort order, or a crash on a page nobody tested.
So this is a chapter about treating a reply as a draft. You ask for a shape, you check the draft against rules you wrote, you fix what failed, and - crucially - you keep the failures visible instead of quietly deleting the record that caused them.
A reply is a draft, not a record
When you ask a model for data, what comes back is a suggestion written in the same language it uses for everything else. It has no idea that your database expects a number, that your board sorts by priority, or that a date of 'next Friday' cannot be compared with any other date. It is doing its best to be helpful, and being helpful is not the same as being correct.
This is why 'it returned valid JSON' is not a finish line. JSON is just a container. `{"priority": "urgent!"}` is perfectly valid JSON and a perfectly useless priority, because nothing in your app knows what 'urgent!' means. The shape can be right while every value inside it is wrong.
The fix is not a longer prompt. Prompts are good at describing what you want and bad at guaranteeing it - there is no sentence you can write that makes a model *always* return one of three words. What guarantees it is code that checks, and a plan for what to do when the check fails.
Ask for a shape, not a paragraph
The first half of the job is making the shape explicit. A model cannot match a format you only had in your head, so the prompt names the fields, the allowed values and the types - and then the code assumes none of it.
A request with no shape
Read this note and tell me what homework I have.
Note: maths homework, science lab report and read chapter 4javascriptRead this note and reply with JSON only, in exactly this shape:
{ "tasks": [ { "title": "...", "subject": "math|science|english|history|art|pe|other", "dueDate": "YYYY-MM-DD", "priority": "low|medium|high" } ] }
Note: maths homework, science lab report and read chapter 4javascriptThe weak version asks a question, so a friendly paragraph is a correct answer - and now your app has to parse English. The improved version names the container, every field, and the only values each field may take. You still do not trust the answer; you have simply made it possible to check.
A shape with no rules attached
Return JSON: { "subject": "...", "dueDate": "..." }javascriptReturn JSON: { "subject": "one of: math, science, english, history, art, pe, other", "dueDate": "an ISO date, YYYY-MM-DD" }javascriptBoth ask for the same two fields. The second one says what a valid value looks like, which is the difference between a model that guesses and a model that has something to aim at. Writing the rule in the prompt and the same rule in your validator is not duplication - the prompt reduces failures, the validator catches the ones that happen anyway.
Validate, then repair
A structured-output pipeline has four steps, and the third and fourth are the ones people skip. You ask for the shape. You parse it - carefully, because a model that was asked for JSON can still hand back a sentence with JSON inside it. You check every record against a schema. Then, for each field that failed, you fix it.
A schema is just a list of fields and the rule each one must satisfy, expressed in code: a subject has to be one of seven words, a due date has to match `YYYY-MM-DD`, a priority has to be one of three. Writing it down in one place is what makes the failures usable. Each failure names the field it came from, the reason a human can read, and the rule that was broken - and that triple is exactly what you need to repair the field.
Repair is the step that turns a pipeline into a *useful* one. Throwing away a task because its priority arrived as 'urgent!' loses real information: the model understood the task, it just used the wrong word. So instead of discarding it, you hand the field back to the model with the failure attached - 'this field broke this rule' - and ask it to fix that field. Then you validate again, because a repair can break a different rule.
const draft = await llm({ system: EXTRACT_PROMPT, user: note, json: true });
const parsed = safeParse(draft.content); // never throws
for (const raw of parsed.tasks) {
let task = raw;
let check = validate(task); // rules live in schema.js
if (!check.ok) {
const fix = await llm({
system: REPAIR_PROMPT,
user: JSON.stringify({ record: task, issues: check.issues }),
json: true,
});
task = { ...task, ...safeParse(fix.content).record };
check = validate(task); // and check again
}
results.push({ ...task, issues: check.issues }); // issues travel with the record
}javascriptThe whole pipeline. Notice that the repair is one more model call, and that the loop validates again afterwards instead of assuming the repair worked.
Building the task board, step by step
This is the shape of the first exercise in the section. The goal is a board of tasks with a subject, a due date and a priority - from a note like 'maths homework, science lab report and read chapter 4'.
1. Write the rules before the prompt
Start with the schema, because the schema is what you are asking the model to hit. Four fields, four rules. Writing this first also tells you what to put in the prompt: the allowed subjects, the date format, the three priorities. In the exercise this file is yours - the starter has a title rule and three TODOs.
export function validate(record) {
const issues = [];
if (SUBJECTS.indexOf(record.subject) === -1) {
issues.push({
field: "subject",
message: "unknown subject: " + String(record.subject),
rule: "one of: " + SUBJECTS.join(", "),
});
}
// ...one rule per field, each pushing { field, message, rule }
return { ok: issues.length === 0, issues };
}javascript2. Run it and read the failures
Run the starter with the schema rules in place and the repair step missing. The board fills up with issues, and each one tells you something specific: 'urgent!' is not a priority, 'next Friday' is not a date, 'Maths!' is not a subject. This is the moment the exercise is designed around - the app is telling you exactly what it cannot accept, instead of quietly showing you a board that looks fine.
3. Repair the record, then validate again
Now the repair. Hand the model the record and the issues the schema found, and ask for the record back with those fields fixed. Two details matter: you repair the *record*, not one field at a time, because the fields are related ('3pm' and an empty end time are one problem, not two); and you validate again afterwards, because nothing guarantees the repair obeyed the rule.
const fix = await llm({
system: REPAIR_PROMPT,
user: JSON.stringify({ record: current, issues: check.issues }),
json: true,
});
const fixed = safeParse(fix.content);
if (fixed && fixed.record) {
current = Object.assign(current, fixed.record);
repairs += 1;
}
check = validate(current);javascript4. Keep the issues on the record
Whatever is still broken after repair is information the student should see, not something to hide. The exercise returns every record with its own `issues` array, and the board draws a task that still breaks a rule as a task that still breaks a rule - outlined, with the field and the reason underneath. An app that silently drops a bad record is worse than one that shows you the problem, because you never find out that your prompt needs work.
Where the failures should go
There are three places a validation failure can end up, and they are not equally good. It can be swallowed - the record is dropped, the board looks clean, and nobody learns anything. It can be shouted - the whole run fails because one field of one record was wrong, which is a bad trade when nine of ten records were fine. Or it can be shown, attached to the record it came from, so the student sees a task that says 'priority: unknown priority: urgent!' next to the three tasks that were fine.
The third option is what the exercises in this section do, and it is worth copying for a reason that has nothing to do with politeness: a visible failure is a prompt you can improve. When the same field fails on every run, your prompt is not describing it well enough. When one field fails occasionally, your validator is doing its job and you can leave it alone. Neither of those things is visible if the failure disappears into a dropped record.
It also changes what the student trusts. An app that shows 'not saved' next to a calendar entry that could not be normalized is honest; an app that saves a half-broken entry is the one that embarrasses you later.
Mistakes worth catching early
| What goes wrong | What it looks like | What to do instead |
|---|---|---|
| Trusting JSON.parse | The run dies with 'Unexpected token' because the model wrapped the JSON in a sentence. | Parse inside a try/catch and treat a failed parse as a result to report, not a crash. |
| Validating nothing | The board shows `priority: urgent!` and every task looks fine, because nothing checked. | Write the rules down in one schema file, and check every record against it. |
| Dropping the bad records | Ten tasks went in, eight came out, and nobody can say which two were lost. | Repair the failing fields, and keep the record with its remaining issues attached. |
| Repairing one field at a time | Six model calls for one record, and a '3pm' start still paired with a missing end time. | Hand over the whole record with its issues, and ask for the whole record back. |
| Assuming the repair worked | A repaired value that breaks a different rule and reaches the app anyway. | Validate again after every repair, and count the repairs so you can see they happened. |
| A schema nobody else can see | The rules live in a prompt, so the code cannot tell a good record from a bad one. | Keep the rules in code, and let the prompt describe the same rules in words. |
Debugging a structured output
- Print the raw reply before you parse it. Half of all 'the model is broken' problems are a reply that was never JSON in the first place, and the fix is in the prompt.
- Check the shape before the values. If the field is missing entirely, that is a prompt problem; if the field is there with the wrong value, that is a validation-and-repair problem.
- Read your own failure messages as if you were the model. 'not an ISO date: next Friday' is a fixable instruction; 'invalid' is not.
- Test the awkward input on purpose: the note with no due date, the quiz with two options, the event with no end time. A pipeline is only as good as its behaviour on the input you did not design for.
- Keep the count of repairs. A run that repairs six fields is telling you the prompt could be clearer, and it is the cheapest signal you have.