Three Months of Pitfalls: Our Agent Harness Practice and Reflections
Harness should be driven by business needs.
Harness should be business-driven.
👦🏻 Author: Wang Dami (Youth)
🥷 Editors: Jason Hu, William Jin
🧑🎨 Layout: NCon
This article comes from Dami (author), Jason (editor), and William (editor) at the Nexad team. Nexad is an AI-native advertising platform based in the Bay Area and Shanghai, having raised $6M from funds including a16z, Prosus, and Point72.
In early 2026, OpenAI and Anthropic published in-depth articles on Harness Engineering. OpenAI demonstrated an experiment where three engineers used Codex to produce a million lines of code. Anthropic showed how a two-stage Initializer-Coder architecture lets agents work continuously across multiple context windows. Both articles were excellent. But they addressed Coding Agents — writing correct code, passing tests, merging PRs.
What we've done over the past three months is somewhat different. Nexad is an AI-native advertising platform. Our agents don't just write code — they operate across multiple ad platforms, generate creative assets, manage campaign budgets, and coordinate data pipelines. When we tried to directly transplant the industry's Harness best practices, we found many assumptions didn't hold in the Marketing Agent context.
This article shares some judgments we formed through this process, along with the specific practices that support them. Some validate industry consensus, some are conclusions we reached independently, and some remain unresolved.
Enjoy.
Why our Harness bar is higher than usual
Before discussing how we harness Coding Agents, it's worth explaining why we need to go beyond standard approaches. The answer lies in what we're building.
Reinforcement learning offers a useful set of intuitions: Environment, State, Action, Reward, Policy. The agents we build — our Marketing Agents — operate in an environment whose constraints fundamentally differ from those of Coding Agents.
Irreversibility. The failure modes of Coding Agents are benign: code doesn't compile, tests don't pass, PR gets rejected. You git revert and try again. Our Marketing Agents submit campaigns to Google Ads, spend client budgets, and publish creative that may violate platform policies. A suspended ad account isn't fixed by changing a few lines of code. So every bug introduced by our Coding Agents into the Marketing Agent codebase carries amplified risk — it's not just bad code, it's code that can cost real money.
Tension between creativity and constraints. Our Marketing Agents must balance creative quality (compelling copy, effective targeting) against hard constraints (brand guidelines, platform policies, budget caps). This means our codebase is saturated with subtle business logic — where "technically correct" and "actually correct" frequently diverge. A Coding Agent that doesn't understand these nuances may write code that passes all tests but breaks the product.
Delayed feedback. The quality of our Marketing Agents' output is measured by CTR, ROAS, and conversion rates — signals that take hours or even days to materialize. This means bugs in ad-serving logic may only surface after real budget has been spent. We can't rely on fast feedback loops to catch Coding Agent errors; Harness must intercept them preemptively.
These characteristics — irreversible downstream impact, subtle business logic, delayed feedback — explain why we treat Harness Engineering as infrastructure, not process. Every Harness decision traces back to a business requirement and deep product context — not engineering preference or brainstormed inspiration from a conference room.
Constraints are only real when machine-enforceable
Frankly, this is where we've stepped on the most rakes.
Early on, we wrote rules in documentation: CLAUDE.md said "please use kwargs for logging," "please follow import hierarchy," "please use soft deletes," even nicknames in CLAUDE.md (when it forgets, that's when CLAUDE.md fails 🤣). The agent read them, mostly complied, but frequently forgot under pressure — especially after Claude's upgrade to 1M context.
Our team spent 30–40% of time on manual quality assurance — manual review, manual test runs, manual spec compliance checks. Clearly unscalable. Too SaaS.
We studied OpenAI's Harness blog closely (thanks to them) and began encoding rules as automated checks. We built 8 custom lint scripts: check_structured_logging.py verifies f-string logging usage; check_soft_delete.py catches direct session.delete() calls; check_api_doc_sync.py ensures API changes sync with documentation, and so on. These run as non-blocking in CI, with grandfathering allowed for legacy code as separate debt to repay.
From this we distilled a constraint enforcement hierarchy:
Enforcement strength decreases from bottom to top. Now, for every new rule, we first ask: Can this be made into a Linter? If yes, write the linter first, then document. This aligns almost exactly with OpenAI's conclusion in their article — "when documentation falls short, promote the rule into code" — which we independently validated in practice.
This is especially critical for Marketing Agents. check_soft_delete.py looks like a code style check, but it encodes a business requirement: hard-deleting ad account records breaks audit trails needed for billing reconciliation. This mapping between business rules and code conventions is precisely what makes Marketing Agent Harness distinctive.
The root cause of self-evaluation failure is shared context, not same model
In their Long-Running Agents article, Anthropic proved an important conclusion: letting agents evaluate their own output fails systematically. Agents mark features complete when functionality doesn't actually work. Their solution separates Generator and Evaluator.
From practice we've formed a more precise judgment: What fails isn't same-model evaluation, but same-context evaluation.
In our Review stage, after the Main Agent completes Spec Compliance checks (requiring full context of the solution), it spawns a SubAgent for code quality review. This SubAgent uses the same model (Claude) but receives completely different context: only the git diff, project rule files (docs/rules/*.md), and a dedicated role definition (.claude/agents/code-reviewer.md) set as a "skeptical senior reviewer." It knows nothing of the Main Agent's reasoning process, what tradeoffs it made, what it skipped.
The results are surprisingly good. The SubAgent consistently catches cross-layer import violations, logging format errors, missing test coverage — issues the Main Agent had "rationalized away" within the same context.
This finding matters for startups.
Cross-model evaluation (e.g., using OpenAI to review Claude's output) adds cost, latency, and integration complexity. Context isolation gives you roughly 90% of the benefit at a fraction of the cost. We do use cross-model review (sanity checks with Codex during Plan phase), but for daily code quality, same-model SubAgent with isolated context is more appropriate.
Harness strictness should be a variable, not a constant
This is what we consider our most worth-sharing insight.
Both OpenAI's and Anthropic's articles implicitly assume all code changes receive identical quality control. The entire pipeline treats every change equally. This is roughly reasonable in pure Coding Agent scenarios — code is either right or wrong, with no intermediate state.
But in real product development, a production API facing unknown users and an internal admin panel require fundamentally different security review depth. A dev environment script and a core transaction flow need different test coverage standards. We found agents waste roughly 30% of time on unnecessarily high-standard checks against low-risk code.
Our solution is a Delivery Tier graded governance system, T0 to T3, with core logic being user controllability determines Harness strictness:
More details on this in our next blog :)
| Tier | Scenario | User Profile | Control Intensity |
|---|---|---|---|
| T0 | Production core paths | Default stranger/malicious users | Maximum across all 6 dimensions |
| T1 | Customer preview | Close partner relationships | Security + observability focus |
| T2 | Internal systems | Fully controllable internal users | Basic protection |
| T3 | Dev / EVL | Product & engineering staff | Minimal constraints |
Tiers are auto-detected from code paths. Our monorepo contains 14 packages; apps/web/ auto-maps to T0, apps/admin-web/ to T2, scripts/ to T3. The agent declares Tier during Plan phase, with human confirmation or override. Each subsequent stage — Review, Test, Ship — auto-loads the corresponding checklist.
The final output is a 6-dimension compliance matrix (security, reliability, observability, performance, UX, compliance) in the PR body.
Anthropic proposed "harness complexity should match model capability." We believe this can be extended further: Harness strictness must match both model capability and delivery risk. Model capability is one dimension; business exposure is another. Delivery Tier is the concrete implementation of the latter.
Progressive disclosure isn't optional optimization, it's architectural necessity
Our CLAUDE.md grew from 50 lines to 200+. Performance first improved, then suddenly degraded — the agent began selectively ignoring rules.
OpenAI has a precise summary: "Too much guidance becomes non-guidance. When everything is 'important,' nothing is." We independently validated this in practice.
The solution is progressive disclosure. CLAUDE.md stays at roughly 180 lines, serving as an entry directory pointing to detailed rules in docs/. Each skill loads only the rule files it needs when triggered. /nex-reviewer reads import-rules.md (49 hierarchy rules) and logging-rules.md only when triggered; /nex-tester reads test-pattern-guide.md only when triggered. Agents receive these rules only when specific rules are needed.
This is essentially the same strategy Cursor uses for MCP tool descriptions — lazy loading rather than preloading. It also shares the same origin as Claude Code's SKILL.md mechanism. The industry is converging on the same conclusion through different paths: agent context is a scarce resource that must be managed as carefully as memory.
Data
Enough judgments, let's look at data. The following comes from our git logs, covering January 2026 to present. Specific numbers aren't disclosed, but the trends speak for themselves.
First, weekly commit trends. In W08 we concentrated on building Harness infrastructure — first batch of linters went live February 12, first hook deployed February 26, and 4 hooks plus 7 core skills deployed in a single day on March 2. The effect is visible in W09 data:
From W09 to W10, output index jumped from 0.7x to 2.1x — nearly 3x. It has since stabilized around 2x. The acceleration mechanism isn't mysterious: Harness absorbed quality assurance work previously done manually. Before going live, we spent extensive time on manual review, testing, and spec validation. After going live, this work was automated by the Skill Pipeline and hooks. Harness didn't make agents faster — it turned humans from checkers into decision-makers.
Second, commit type distribution. A notable signal: feat and fix are nearly 1:1. This isn't coincidental — agents rapidly produce features while also rapidly producing issues requiring fixes. This precisely validates why Harness is necessary: without automated constraints and validation, fix count grows linearly with feat count.
Another data point: across nearly 2,000 commits, 27% involved documentation changes, 15% involved test file changes. This isn't accidental — our skill definitions mandate testing and documentation synchronization as required steps. Eight custom linters run continuously at pre-push stage. Agent output passes through at least 2–3 automated checks before reaching main.
Harness infrastructure build timeline:
Problems we still haven't solved
Technical debt accelerates under AI. Agents faithfully replicate existing patterns in the repository, including bad ones.
Once an incorrect logging format is introduced, agents continue copying it across all new code. Human engineers typically realize "this pattern is wrong, I shouldn't follow it," but agents' pattern-matching mechanism inclines them to perpetuate existing patterns. We've intercepted known bad patterns through linters, but haven't yet built what OpenAI calls a "garbage collection" mechanism — an agent that periodically scans for code drift and automatically opens fix PRs. This is next on our roadmap.
Design rationale for the Plan-Todo-Progress trio. Some have asked why we need three-piece documentation (docs/plan/active/, docs/todo/active/, docs/progress/active/), whether it's over-engineering. This system wasn't invented in a vacuum; it emerged from a specific failure mode: without external artifacts, agents attempt to complete complex features in one shot, exhaust context during implementation, and leave the next session to guess what happened. This aligns exactly with the failure mode Anthropic documented in their Long-Running Agents paper — they solve it with claude-progress.txt and feature lists; we use the three-piece system.
A key design decision here: each Todo must include a Verify command and Expect output, precise to shell command and expected string. This granularity isn't arbitrary — it enables the TaskCompleted hook to mechanically verify completion status rather than trust agent self-evaluation.
The last mile of creative quality. Our Harness can check policy compliance, brand guideline adherence, and technical correctness. But whether ad creative will perform well — whether copy resonates, whether visuals make people stop scrolling — remains human judgment. The gap between "policy-compliant" and "high-performing creative" is what we currently cannot automate.
Conclusion
Junyang wrote in his article on Agentic Thinking: "The future is a shift from training models to training agents, and from training agents to training systems." Our experience partially validates this judgment.
Over the past three months, we may have spent more time designing Skill Pipelines, debugging hook scripts, and refining the Delivery Tier matrix than writing any single feature. But it's precisely this scaffolding that lets agents produce code daily that reliably serves real users.
Harness Engineering has just begun. As model capabilities improve, constraints that are necessary today may become redundant tomorrow — Anthropic has observed this trend with each Claude generation.
But one core judgment remains unchanged: Marketing Agent Harness design should be driven by business requirements, not engineering preferences.
When a single wrong decision by your agent could get a client's ad account suspended, "enforce invariants, not micromanage implementations" isn't merely an engineering principle — it's a survival strategy.
The form of the harness changes; the role of the harness does not.
March 2026 nex.ad[2]
We may be one of the most AI-native startup teams globally. If you're interested in the relevant technology and business model, feel free to DM me :)
References
[1] CLAUDE.md: http://CLAUDE.md
[2] nex.ad: https://nex.ad