You buy a flipped house. The listing photos were great: fresh paint, a new kitchen, and a new roof, dark shingles in clean rows. The notes said, "New roof."
Everything in the photos was real. The shingles are new. Then the first hard rain comes through, a brown stain spreads across the bedroom ceiling, and you realize something went wrong with how that roof was put on.
Agents send listing photos all day. "Implemented the assistant. All tests pass."
I build iPhone apps for my own use with coding agents: a trip planner, a journal, a training log. I don't write the Swift; I describe what I want, review what comes back, and decide what ships. All three use Apple's on-device model, the language model built into iPhones that support Apple Intelligence, which runs privately and costs nothing per request.
I got that message for the trip planner's chat, and it was accurate as far as it went. The chat existed, and the tests passed. Then I had the agent actually hold a conversation with it, and that was the first rain. Asked what still needed booking, it named a to-do. Asked which nights had no hotel, it named two that were already booked. Asked what tickets cost, it said CHF 100, and asked about the weather, it said 10–15°C. Its instructions specifically told it never to invent either one.
The commit that fixed it opens with the sentence I now look for in every project: "The Ask Anything conversation had never been run."
"Done" describes the code
That chat wasn't the only case. Going back through the history of all three apps, I found the same problem in each one, and it's worth spelling out because it's easy to miss even when you're looking at the app every day.
An AI feature in these apps usually has two ways to produce an answer. One asks the model. The other is plain code: a fixed list of questions, a template, a set of keyword rules. The plain code is a fallback for when the model isn't available, such as on an older iPhone.
The problem is that a feature can run on the fallback when it was supposed to run on the model, and from the outside nothing looks wrong. It's the software version of new shingles laid over the old roof: from the street it's a new roof, and underneath, the old one is still doing part of the work.
My journal app has a guided chat for writing about your day. You pick a moment, like a meeting or a place you visited, and it asks you a question about it, then follows up on what you answer. The point was for the model to write those questions from what had actually happened that day.
For about two months, the first question for every moment came from the fallback list instead. The list had reasonable questions: "How was [event]? What happened during it?" or "What stood out about being at [place]?" Only the follow-ups, after you'd answered, went to the model. That was roughly half the questions in a session, including the one you read every time you picked a moment. The code to have the model write an opening question already existed. In the agent's own words when it found the problem: "The prompt builder already knew how to ask for an opener… nothing was calling it."
From my side, the feature worked. A question appeared, it made sense, and the conversation carried on. The agent had reported the feature done, and the tests passed. Nothing on the screen told me the first question was coming from a list.
The other two apps had their own versions. The training app's AI-written weekly readout was built, then a screen redesign left it shown nowhere for three weeks, while the charts around it looked fine. The trip app's packing suggestions still run on keyword lists, next to a comment noting where the model could plug in later.
Tests miss this, and the journal app shows why. In a separate case, a new model-driven way of answering questions like "when was the last time I saw this person" cut off the older code path for those questions. The commit that caught it put it plainly: "Every unit test of the deterministic path kept passing the entire time nothing could reach it." A unit test checks that a piece of code gives the right answer when it's called. It doesn't check whether the feature ever calls it.
So neither a completion message nor a green test suite tells you which path ran when you used the feature.
The inspection happened on a dry day
There's a second reason the tests missed it. An inspector can walk a roof on a sunny afternoon and find nothing wrong, because nothing goes wrong until it rains.
Apple's on-device model doesn't run in the iOS Simulator, the stand-in iPhone on a Mac where agents do most of their testing. In my journal app, the Simulator throws a decoding error the moment it tries. My training app's model service reports itself unavailable under the unit test runner on purpose, so tests behave predictably. In both cases the app falls back to plain code, exactly as designed.
Put those together and every automated check the agent ran could only exercise the fallback. The agent wasn't misreporting anything. Its tests were green, and every one of them ran on a dry day.
My training app's testing notes now say it directly: "Passing Simulator tests must not be interpreted as successful AI parsing." The journal app's fix went one step further and logged every time the guided chat fell back, "so the fallback rate is observable rather than invisible." Once you know a fallback exists, you want to know how often it's carrying the feature.
Run the hose over the roof
You don't have to wait for a storm to find a leak. Roofers soak one section at a time with a garden hose while someone watches the ceiling inside. The software version was the same move in every case: run the real feature on a real device and read what comes out. None of it required me to read the Swift.
In the journal app, the agent ran the production prompts on my iPhone against a realistic day, three runs per setting, and "read the output instead of guessing at it. Four things were wrong, and none were visible from the unit tests." One of them: every entry was written in the third person. "At 8:30 AM, the user visited a chiropractor." The model was writing about me instead of as me, because nothing had told it whose voice to use.
In the trip app, the agent built a small harness that compiles the app's own prompt and trip summary and holds a real twelve-turn conversation with the model. Three of the questions are traps: the plan doesn't contain the answer, and the only passing response is to decline. The score went from 6 of 10 to 12 of 12, with all three traps declined. Along the way the agent set the model's temperature to zero, because two runs at a slightly higher setting disagreed about whether it had invented a price. A scoreboard that moves on its own can't measure a change to the prompt.
In the training app, the workout-note parser went through six evaluation runs on my iPhone: 0 of 16, then 6, 15, 25 and finally all 34 cases. The record of the final run is careful about what that number means. It's a regression set of synthetic English notes written by someone who had already watched the parser fail, so it shows those defects are fixed, "not that workout-note import is solved."
The same kind of run caught another problem. Asked "What should I wear?", the trip chat answered "Packing: 0 of 0 packed." It had been told to answer only from the trip plan, and the plan had no forecast and no packing list in it. After the fix, clothing and weather questions route to a separate request that reads the forecast and is allowed to use judgment. Routing went 16 for 16.
Stay out of the attic
The common advice about working with AI is to keep yourself in the loop. For deciding what a feature should do and whether the result is good enough, I agree. For the testing in between, I've found that getting out of the way works better.
A hose test needs someone in the attic watching for water while each section of the roof gets soaked. The homeowner doesn't have to be that person. What the homeowner needs is the marked-up list afterward: which sections leaked, which held, and which never got tested.
The volume makes the case. The workout parser's six runs on my iPhone produced well over a hundred individual results. The trip chat's harness scores a twelve-turn conversation every time the prompt changes, and the import harness checks ten sample documents against 191 expected facts on every run. Each prompt change, new case or fix means another round. If I review every attempt, the agent waits on me, I become the slowest part of the loop, and the rounds that would have caught the next problem don't happen.
So my involvement moved to the two ends. Before the runs, I decide what counts as passing: the trap questions the chat has to decline, the values that have to match the source document, the journal entry that has to sound like me. Afterward, I read the scores, a sample of the actual output and the list of what the runs didn't cover. The agent handles everything in between: running the cases, reading the failures, changing the prompt or the code, and running again.
That's also why a stable scoreboard mattered so much in the trip app. When the score moves on its own, I'm pulled back in to judge every run by hand. When it holds still, I can stay out until it changes.
Before you blame the shingles
The journal app is where I pushed hardest, and where the problem was least visible. Its recaps and its chat did call the model. The trouble was what the model received. Recaps quoted the first 400 characters of each entry, so most entries were represented by whatever they happened to open with. Search results reached the chat as a short matching fragment spliced onto the start of the entry, and the model would "blend unrelated entry fragments into invented facts," as the commit that fixed it put it. Ask about a person and you could get confident details about them that no entry contained.
Meanwhile, instructions kept piling into the prompts. The model's working space filled up with directions about my journal and held very little of the journal itself. When the answers came back thin, the agent's conclusion was that the on-device model was too small to do the job, and the next proposal was more plain code around it.
In roofing terms, it's a contractor who skipped the underlayment, then blamed the leaks on the brand of shingles and offered to put a tarp over them.
What I kept asking for was ordinary diligence before blaming the model. To answer a question about a person, find every entry that mentions them, pull the passages around each mention, and give the model that. Then judge what it does with it. Recaps now quote a representative passage chosen from the whole entry instead of its first 400 characters.
The work still isn't finished. A later audit rebuilt each journal prompt from the app's own templates and counted it with Apple's tokenizer against a set of sample entries. No feature used more than about a third of the model's context window, the amount of text it can read at once on the devices the app supports. Most limits were fixed character counts written for an older, smaller window. One step cut every search result to 500 characters before anything downstream could see it. The weekly and monthly recap overviews contained no entry text at all, and the writer's own words made up 14% of the prompt. The window had room for 24 whole entries where the chat was sending six excerpts of 260 characters.
When an agent tells me the model can't handle something, my first question now is to see exactly what the model was given.
Some spots need flashing, not shingles
Shingles aren't the right material everywhere on a roof. Valleys, chimneys and skylights get metal flashing, because that's what holds up in those spots.
The obvious conclusion would be "make the agent use the model." The repos don't support that, and it's the most useful thing I learned from them.
Some of the plain code was the right answer, and it was measured. The training app's parser stopped asking the model to count sets or track warm-ups because, in the agent's words, "the model reads numbers well while doing bookkeeping badly." The journal app's "when was the last time" question returned a date from 2020 when the newest matching entry was five days old, so ordering went back to code: "Chronology is not a judgment call." When the trip app reads back a plan in a format the app itself defined, the model returned eight or ten of a trip's twelve legs. The code returns all twelve, every time.
It also went the other way. Early on, the trip app's agent had written separate parsers for boarding passes and itineraries. I decided those parsers were a maintenance burden that would only grow, and that the model should read pasted text with the traveler confirming what it found. About 600 lines of parsing code went away. The commit records it as "Paul's call."
Left alone, though, the agents kept reaching for flashing. The training app's weekly readout is a good example. Code decides what to recommend, and the model's prompt says the next move "is already correct" and asks it to "rewrite it in plainer language." For a training recommendation, that may be the right design. It should be a design I chose, rather than the place the work happened to settle. Keeping the agent focused on using the model for what it's good at, reading and weighing what I actually wrote, took repeated pushing.
Which path a feature takes is my decision. The agent can measure the options; it can't know that I want a journal to sound like me, or that a traveler should confirm what the model read before it lands in the trip.
Checking wiring you can't judge yourself
A roof, at least, you can see. The wiring behind the walls is harder to judge, especially if you've never wired a house. I'm in that position more often now. I spent years working in Python, and agents have me building web apps and configuring cloud architecture well outside that.
Sharing a trip between two iCloud accounts was one of those jobs. I'd never built sync, and the first version went badly: a custom sharing path ran alongside the app's normal iCloud storage, and one evening a trip's places doubled until there were 545 of them. I wrote about that repair separately.
Less of it was new than I expected. A record still has an owner. Two copies of the same record can disagree, and a change made on one device has to arrive on the other. Those concepts gave me enough to ask targeted questions even when I couldn't have written the code: which copy wins when both devices edit the same note, what happens to an expense when its place is deleted, how a change reaches the other account.
The check that settled it was a reference I could trust. Apple publishes a sample project for sharing Core Data records between iCloud users, and I couldn't have written it myself, but I could trust that it was correct. So the agent rebuilt the app's sharing to match it. That removed the hand-written history replay, the custom push handling and the custom invite screen in favor of the pieces the sample uses. When an agent works in territory I don't know, a known-correct implementation gives me something to compare against: where our code differs from the sample, I want a reason.
The hose test was a live run across two accounts. The agent drove my iPhone and an iPad signed into a different iCloud account, edited a shared test trip on each, and checked what had arrived on the other after a relaunch.
What I ask for before I accept "done"
The listing photos are useful. I still want them. I just stopped treating them as the inspection.
Before I accept a feature now, I want to know which path it took and where it ran: the model or the fallback, on a device or in the Simulator. I want output I can read, like the answers to a real conversation, a sample of parsed records, or the log line from the step that matters. I want to know what the check didn't cover, the way the parser record lists the languages and photos it never tested. And when the agent chose plain code over the model, or the other way around, I want the reason, so I can decide whether it's the reason I'd have chosen.
That's also what I'd expect if I were the client. A feature that arrives with evidence of how it ran is one I can trust and maintain. A feature that arrives with only a photo gets its first rain in front of the people who depend on it.
Ask for the photos. Then run the hose.