All field notes

The right to explanation concerns whether people can demand reasons for decisions made about them by automated systems. This explainer covers where the obligation applies, what actually counts as an explanation to the person affected, why post-hoc techniques are weaker than they appear, and how to design for it.

Smart contract bugs cost fortunes. How to hire smart contract developers who write secure, audited, gas-efficient code—and the red flags to avoid.

A circuit breaker stops calling a failing dependency until it recovers, preventing wasted capacity and cascading failure. This explainer covers how breakers apply to model providers and tools, why AI systems also need breakers on quality rather than only errors, and how to choose trip and recovery conditions.

A fallback chain defines the ordered alternatives a system tries when its primary path fails. This explainer covers designing chains that degrade predictably, why every step must be evaluated rather than assumed equivalent, when fallback should be visible to the user, and where honest failure beats a fallback.

Backpressure is the mechanism by which a saturated system signals upstream to slow down rather than failing. This explainer covers why AI pipelines need it more than conventional services, how queue depth and latency indicate saturation, and how to shed load deliberately without losing work.

Idempotency means an operation can be repeated without additional effect, which matters because agents retry and retries duplicate side effects. This explainer covers where duplication occurs, how idempotency keys work, which operations need protection most urgently, and why agents retry more than conventional systems.

Rate limits on LLM APIs constrain both requests and tokens per minute, which makes them behave differently from conventional API limits. This explainer covers how the quotas interact, why naive retry makes contention worse, how to design internal limits for fairness, and what to do when a limit is hit.

Throughput measures how much work an AI system completes per unit time, and it trades directly against latency in ways conventional services do not. This explainer covers tokens versus requests per second, how batching and context length affect capacity, and how to plan for peaks without over-provisioning.

AI staff augmentation adds vetted AI engineers to your team under your management. When it beats project outsourcing, and how to do it without losing control.

Abstention is a system's ability to decline to answer when it lacks sufficient evidence. This explainer covers why declining is a feature rather than a failure, how abstention is triggered from retrieval quality and calibrated confidence, why it must be evaluated deliberately, and how to present it without frustrating users.

An error budget is the allowance a system has for missing its objective before change slows down. This explainer covers how budgets translate targets into decisions about release pace, why AI systems need quality budgets as well as availability ones, and how burn rate signals trouble early.

An escalation policy defines when an AI system hands a case to a human, to whom, and with what context. This explainer covers the triggers that should force escalation, why context transfer determines whether handoff helps or annoys, how to route by capability rather than availability, and what to measure.

Service level objectives for AI systems must cover output quality, not only availability, because an AI system can be fully available and producing poor answers. This explainer covers which indicators to choose, how to set targets for probabilistic behaviour, and how quality objectives change operational practice.

Cost per task measures AI spend against completed outcomes rather than individual calls, which is the only framing that supports economic decisions. This explainer covers why per-token and per-call views mislead, how retries and escalations belong in the figure, and how the metric changes model selection.

Distributed tracing records the full path of a request across models, retrieval, and tools so that failures can be diagnosed. This explainer covers what an AI trace must capture beyond conventional tracing, why probabilistic behaviour makes full capture necessary, and how to handle sensitive content in traces.

Token accounting attributes AI spend to the features, teams, and customers that caused it. This explainer covers why an aggregate provider invoice cannot answer any useful question, what dimensions to capture, how input and output tokens differ in cost, and how accounting enables real optimisation.

Two very different models. Staff augmentation adds hands to your team; project outsourcing hands off an outcome. How to choose the right one for your project.

An evaluation rubric defines what good output means in terms specific enough to be applied consistently. This explainer covers how to write criteria that different graders apply the same way, why vague dimensions produce meaningless scores, how to calibrate human and model graders, and where model-as-judge is safe.

Champion-challenger testing runs a candidate model alongside the one in production to decide whether it is genuinely better. This explainer covers shadow evaluation against live traffic splits, what to measure beyond accuracy, how long to run before deciding, and why most promotions happen too quickly.

A calibrated model's stated confidence matches its actual accuracy: claims made at ninety percent confidence are right about ninety percent of the time. This explainer covers why calibration matters more than raw accuracy for routing and automation decisions, how to measure it, and what improves it.

Offline evaluation measures a system against a fixed set of cases; online evaluation measures it against live traffic and real outcomes. This explainer covers what each catches that the other misses, why offline results stop predicting production over time, and how to run both without duplicating effort.

Pass@k measures whether at least one of k generated attempts succeeds. This explainer covers where the metric originated, why it only reflects real value when a verifier can identify the correct attempt, how it inflates perceived capability, and what to measure instead for production systems.

Perplexity measures how surprised a model is by text, and it remains widely cited despite correlating poorly with whether a system is actually useful. This explainer covers what it measures, why comparisons across models are usually invalid, where it still earns its place, and what to use instead for production decisions.

A dedicated team is a cohesive unit; staff augmentation adds individuals to yours. How the two models compare on control, continuity, and cost—and how to choose.