Our tests passed. The client redrew everything by hand.
A duct-routing engine shipped with every test green, and the contractor still redrew its output by hand. What that taught us about the gap between correct software and accepted software, and how we test for it now.
We shipped an automatic duct-routing feature with the whole test suite green. Every layout it produced was valid: runs were axis-aligned, ducts were correctly sized, and airflow added up along the trunk. By every check we had written, it worked.
The contractor printed the output and redrew every duct by hand.
Nothing was broken in a way our tests could see. “The tests pass” and “the person who does this for a living accepts the result” turned out to be two different claims, and only the second one mattered to the client.
The cause was a single coordinate
The engine anchored the main trunk at the position of the air handler. That sounds reasonable, because the air handler is where the air comes from. But the air handler sits at the edge of the floor plan. Anchoring the trunk there meant every branch had to travel from that edge, across the whole plan, to reach its register.
The result was a valid duct system that no contractor would build: a spine pressed against one wall, with long branches reaching across the entire floor.
The problem was measurable. Total branch length came to about 2,140 px on the drawing. A small closet register that needed only 7 CFM had a 450 px branch.
A person does not draw it that way. A person runs the trunk through the middle of the registers it serves, so the branches are short and balanced on both sides.
The fix was one word. Instead of anchoring the trunk at the air handler’s coordinate, we anchored it at the median of the register coordinates. Total branch length dropped to 840 px, and the closet branch became a short stub. Nothing else changed.
Passing and accepted are different claims
The tests were not wrong. They checked exactly what we had asked them to check: the geometry is orthogonal, airflow never increases along the trunk, branches leave on both sides, and elbows land on corners. All of that was true before the fix and after it.
What the suite did not check was the thing that defined the feature: the drawing has to look like one a contractor would draw. We had not written that test because we did not yet know it was the requirement. We understood the physics. We had not found the objective.
The redraw was not a complaint. It was the specification, delivered late and in the clearest form possible: an expert doing the job the way it is meant to be done.
Finding the real objective
Much of what counts as engineering skill on a new problem is working out what you are really optimizing for. It is easy to optimize the objective you were given. It is harder to notice that the stated objective (“produce a valid duct layout”) stands in for an unstated one (“produce the layout an expert would have drawn”), and to find the real one before the client has to draw it for you.
You find it by watching the work being used, not by rereading the brief. The brief said “size and route the ducts”. It did not say “keep branches short and center the trunk on its registers”, because the contractor did not think of that as a requirement. It is simply how ducts are drawn. It was invisible to them and unknown to us, and requirements like that only surface when a real user handles the output.
What we changed
Once the objective is known, it is no longer a matter of taste. It becomes a test like any other. The redraw turned into a single assertion: total branch length stays under 1,100 px, checked on every run.
The lesson reaches beyond this one bug and now shapes how we build:
- Test the property an expert would recognize, not the coordinates. “Returns these exact points” can pass while the feature is wrong. “Branches are short and balanced on both sides” is what a contractor cares about, and it stays meaningful when the internals change.
- Treat the first expert redraw as the specification. When someone who knows the domain reworks your output by hand, that rework is the requirement. Capture it as a check before changing the code, so it cannot regress.
- Put the output in front of a real user early. The objective you assumed and the one that matters are rarely the same, and the gap between them is where the real work is.
One design decision made this easy to fix. The routing engine is a pure function: numbers in, geometry out, with no user interface involved. The problem could be reproduced, measured and tested without opening the canvas, which is why a one-word change could be verified rather than hoped for. The full build is described in the HVAC platform case study.
Green tests told us the code did what we had asked. The redraw told us what we should have asked for.
Working on something in this territory?
If any of this sounds like the problem in front of you, tell us where you are and where you'd like to be. We'll come back with a plan within two business days.