The wrong things may improve first

Why the developmental layer creates new failure modes

When an AI system can alter the machinery that governs later improvement, errors can become inherited. A bad output is transient. A bad rule for producing, selecting, or validating future changes can compound across generations.

• Causal misattribution. The system changes the wrong component because it misidentifies the source of failure.
• Evaluator capture. Improvement targets the judge rather than the underlying objective.
• Benchmark exploitation. A lineage becomes better at extracting score without becoming better at the intended capability.
• Inherited errors. Incorrect procedures or poisoned memories acquire authority through persistence.
• Agent escalation. Conflicting objectives can create sabotage, lockout, or arms-race dynamics.
• Environmental manipulation. The system changes the evidence-generating environment instead of itself.
• Deceptive coordination. Agents may coordinate in ways that evade local monitors or exploit seams between authorities.
• Lineage drift. Small accepted changes can accumulate into an organization no longer represented by its original specification.
• Loss of rollback. External artifacts, credentials, dependencies, or self-propagating state can make reversion incomplete.
• Developmental debt. Future changes become harder to interpret or reverse because causal provenance is lost.

Causal misattribution

A developing system must decide what caused success or failure. If it attributes a failure to the wrong component, it may “repair” the wrong thing. In multi-agent settings, false attribution can become self-reinforcing: one agent infers hostility, intervenes, alters the environment, and then treats the resulting response as confirmation of the original belief.

Evaluator capture

The component that proposes a change should not control all of the evidence used to approve that change. If the improving system can rewrite its evaluator, alter benchmarks, expose protected tests, or redefine success, apparent progress can become selection bias. The safest architecture keeps important reality checks outside the mutable loop.

Inherited state

Persistent memory is useful precisely because later agents treat past records as privileged evidence. That also makes retained history an attack surface. A temporary manipulation can become durable if it is admitted into trusted memory, distilled into procedure, or propagated into shared organizational artifacts.

Lineage drift and rollback

A sequence of locally reasonable changes can produce a descendant whose behavior no longer resembles the original system. The danger is not only sudden self-rewrite. It is accumulated developmental debt: dependencies become opaque, provenance weakens, regressions become harder to isolate, and rollback becomes expensive or incomplete.

Safety principle

Keep selection evidence outside the system's unilateral control. Preserve append-only provenance. Separate permissions. Use protected evaluators. Fork lineages before high-impact changes. Measure negative recursive leverage as carefully as positive recursive leverage. Stop propagation when rollback, evaluation integrity, or causal reconstruction is lost.