For a client project that needs a dependable way to create and maintain personalized courses, creating the content once is not the hardest part.
The hard part is keeping the same quality over time.
A course changes. Lessons are reviewed. Exercises are added. Media is produced. The learner finds gaps that were not obvious during development. A discussion with a teacher can reveal that something technically correct was not explained well enough.
Every change can affect something else. Without a controlled process, quality starts to drift.
Before I was hired, the client already had a serious AI-assisted production process.
There were many Markdown files describing what had to be done. Responsibilities were separated. Agents had specific tasks. Other agents reviewed the results. Judge agents were used to catch problems and maintain standards.
This was not one big prompt trying to do everything.
On paper, it was a good approach.
But it did not work well enough.
More instructions did not create more control
The problem was not the absence of instructions. The instructions existed, and many of them were detailed.
The problem was that instructions are not enforcement.
An agent could read a standard and still miss part of it. Two agents could interpret the same rule differently. One judge could approve something that another judge rejected. A problem fixed in one lesson could return in the next one.
The process also became repetitive.
Agents kept reading the same files, rebuilding the same context, checking the same requirements, and reviewing decisions that had already been made. When one part failed, too much work had to be generated and reviewed again.
More agents meant more calls. More judges meant more reviews. More instructions meant more context.
The cost increased, but the quality did not increase at the same rate. More importantly, the process could not keep the quality and standards stable.
I was hired to understand why.
The problem was the division of work
The mistake was not using agents.
The mistake was using agents for almost everything.
Some work really needs interpretation. Designing a learning sequence, explaining a difficult idea, creating a useful example, identifying a misconception, or understanding learner feedback are good tasks for an LLM.
Other tasks have exact answers.
A file exists or it does not. A required section is present or missing. A test passes or fails. A reference is valid or broken. An approved artifact changed or remained untouched. A process has the required evidence to continue or it does not.
An LLM should not be paid to guess the answer to something that code can verify exactly.
Using another LLM as a judge does not solve this. It only adds another probabilistic decision to a deterministic problem.
That was the main issue I found: the process had specialized agents, but it did not have a strong deterministic system around them.
The solution is a harness
I am building a reviewable AI harness for course production.
The harness controls how the work is performed. It brings deterministic tools, LLMs, evaluations, guardrails, evidence, and human decisions into the same system.
It can decide when code should perform the work and when an LLM is actually necessary. It can stop an operation that breaks a rule. It can record what happened, control retries and costs, preserve accepted work, and continue from a checkpoint instead of starting everything again.
The harness also provides guardrails, evaluations, persistent state, checkpoints, observability, cost controls and evidence for important decisions.
The goal is not to remove LLMs.
The goal is to stop using them for work that should never have depended on them.
The graph is part of the product
The harness is not the graph.
The harness provides the common controls and infrastructure. The graph coordinates the workflow for a specific product.
For the course product, I need three main graphs:
- Create a new course.
- Review and adjust an existing course, block, or lesson.
- Improve a lesson after I complete it and discuss it with a teacher agent.
These processes have state, dependencies, decisions, failures, retries, validations, and approval points. That is where a graph is useful.
But not every graph node is an agent.
A node can run a deterministic function, execute a test, validate an artifact, call an LLM, evaluate a result, or wait for a human decision.
And not everything needs to be a graph. If a simple function can perform a task reliably, the harness should run that function. There is no reason to make a deterministic operation more complicated just to call it agentic.
What I mean by reviewable
Reviewable does not mean stopping after every node and asking a human to approve it.
It means I can understand what happened.
I can see what was requested, which standards were applied, what changed, which tools and agents were used, what passed, what failed, and why the process made a decision.
If something fails, I should not need to restart a large group of agents and hope for a better result.
The harness should identify what was affected, return to the correct checkpoint, and repeat only the necessary work. Everything that is still valid should remain valid.
This is not about adding more agents
The client already had documentation, specialized agents, and judge agents.
Adding more of the same would only make the process larger and more expensive.
The real change is moving from a collection of agents to an engineered production system.
The system must be deterministic where the answer is exact, agentic where interpretation is valuable, observable throughout, and human-directed where judgment matters.
That is what the harness provides.
The graphs organize the course-production workflows inside it. Together, they should make the process more consistent, less repetitive, less expensive, and easier to improve after real use.