For a client project that needed a dependable way to produce and maintain personalized courses, the problem looked familiar at first.
The output was inconsistent.
One lesson could be strong. The next one could miss a basic requirement. A reviewer could approve an artifact, and another review could later find an important gap. More prompts, more rules, and more correction passes increased the work, but they did not keep the quality stable.
The client had already tried the obvious solutions before I was hired.
There were many Markdown files describing the standards. Responsibilities were divided by task. Specialized agents produced different parts of the course. Other agents reviewed them. Judge agents checked whether the results should be accepted.
This was not a careless process.
It was a serious attempt to control AI-assisted production.
But after many revisions, it still could not produce one complete block at the expected minimum quality.
I stopped treating the symptoms as findings
I already knew that the standards were spread across multiple Markdown files. I already knew that documenting more rules had not guaranteed that those rules would be applied consistently. I already knew that the process was repetitive and expensive.
Repeating those facts would not explain anything.
I needed to find the mechanisms behind them.
I reconstructed the course-creation process, the review-and-adjustment process, and the expected post-lesson improvement process. I inspected the standards, review records, scripts, tests, artifact states, and the existing harness base.
I also separated three types of work:
That separation made the real problems visible.
The rules documented quality, but they could not guarantee it
The first problem was not missing documentation.
The problem was expecting documentation to behave like a control system.
A Markdown file can describe what a good lesson requires. It cannot prove that every required explanation exists. It cannot stop an artifact from advancing with missing practice. It cannot invalidate old evidence after a change. It cannot guarantee that the next agent will apply the same interpretation as the previous one.
Those rules become effective only when the production system converts them into contracts, gates, evaluations, and state transitions.
Every agent had to find the correct Markdown files, decide which rules applied, resolve precedence, remember the relevant constraints, and interpret whether the output complied with them.
The written standard could remain unchanged while the effective standard changed from one prompt to another.
Context selection changed. Attention changed. Interpretation changed. Recent instructions received more weight than older ones.
Adding more detail to the files could even increase the amount of context each agent had to reconstruct without increasing enforcement.
Markdown can define the standard. It cannot guarantee the result.
Reviewers could not compensate for unenforced rules
The reviewers had different responsibilities, but they still judged prose standards probabilistically.
They did not evaluate against a versioned benchmark containing known good outputs, known failures, and expected findings. Their agreement was not measured. Missed defects and false findings were not tracked as evaluation results.
A judge could catch a problem. It could also miss another one or interpret the same Markdown differently.
The process had review roles, but it did not have calibrated review quality or an enforced connection between a rule, its evidence, and the decision to continue.
Deterministic checks were still controlled by LLMs
The existing system already had useful scripts and tests.
It could validate schemas, identifiers, paths, hashes, code execution, media metadata, PDF mechanics, progress behavior, and other exact conditions.
But an LLM often remained responsible for deciding which checks to run, what scope to use, how to interpret the result, whether a failure should block progress, and what status to record afterward.
The check itself was deterministic.
The gate around it was not.
This created avoidable cost and a simple failure mode: a valid test cannot protect the process if the agent forgets to run it or advances the artifact without complete evidence.
The process checked artifacts, but the product failed as a block
A lesson could look good. A script could pass. A PDF could render correctly. A media file could be technically valid.
None of those results proved that the complete block was ready.
The required success unit was the full block: coherent sequence, complete teaching, practice, assessments, source accuracy, media, PDFs, platform integration, and preserved learner state.
The evidence was collected below that level.
In one reviewed tranche, the platform tests passed and 44 content steps existed, but 33 of 35 numbered concepts still lacked the required paired practice.
That is the difference between artifact validity and product readiness.
Readiness was a label, not an enforced state
The process used useful labels: authored, generated, technically checked, visually reviewed, listened to, viewed, approved, and ready.
But those labels were distributed across Markdown files, manifests, reports, tests, and learner-facing metadata.
There was no single state machine enforcing valid transitions and required evidence.
When a standard changed, an artifact changed, or a hash changed, the system could not always invalidate the correct evidence automatically.
Agents had to reconstruct readiness from records. Different runs could reach different conclusions about the same artifact.
Corrections did not know their real impact
A correction can affect a lesson, its exercises, a PDF, narration, media, platform integration, and the evidence attached to each artifact.
The current process did not calculate that impact deterministically.
That created two bad options.
The process could reuse evidence that was no longer valid, or it could regenerate and review more work than the correction actually affected.
One option risks quality. The other increases cost.
The history could not explain why quality changed
The project history showed repeated correction, review, regeneration, and validation work.
But the process did not preserve complete execution lineage for each artifact.
It could not reliably connect the standard version, prompt, model, tool calls, retries, evaluation results, cost, and correction that produced a quality change.
I could not reconstruct the exact historical LLM cost or prove which configuration caused each improvement.
Without that lineage, the system could observe that quality changed but could not learn why.
The learner cannot be the final quality gate
The product also needs a process to improve a lesson after I complete it and discuss it with a teacher agent.
That feedback is valuable, but it must improve an already acceptable lesson.
If I have to discover missing definitions, absent practice, broken explanations, or incomplete resources in every lesson, the production system has failed.
The same applies to full-block review.
A useful harness cannot ask one person to inspect and approve dozens of lessons before using the course. It must enforce the agreed quality contract across the entire block, aggregate the evidence, and escalate only a genuinely new exception.
This diagnosis changes what I build next
The harness I built is the base.
It already provides strict contracts, execution policies, routing, telemetry, an API boundary, and a working smoke graph.
The audit showed what the course-production layer is still missing: operational standards, calibrated evaluations, direct deterministic gates, full-block readiness, evidence invalidation, dependency-aware reruns, and complete execution lineage.
I have not designed that layer yet.
That is deliberate.
Designing the graph before understanding the failures would only turn assumptions into architecture.
The result of this phase is a bounded engineering diagnosis. It converts a collection of written rules, prompts, agents, and reviews into seven specific system problems that can be designed, implemented, and evaluated.
The next step is not to write another Markdown file and expect it to control the process.
It is to decide which written standards must become contracts, deterministic gates, evaluations, state transitions, and observable workflow behavior inside the harness.