Coding agents are starting to learn while idle

Microsoft Research's Debug Gym team published Shadow-Frog on June 18. The system gives coding agents a way to use idle time for discovery. Before someone steps away from an IDE, they can start a dream session. The agent creates candidate tasks, experiments on separate git branches, runs code and tests, then stores the findings in a shadow knowledge base.

The storage choice is deliberately boring. A .shadow/ folder mirrors the repository tree. If the agent learns something about src/parser.py, it writes the note near .shadow/src/parser.py.md. Cross-file findings and dream reports get their own folders. There is no special vector database in the core design. The index is the repo.

The reported numbers are the reason this is worth watching. Microsoft says the per-file shadow layout hit 97.6 percent retrieval accuracy in eight tool calls, compared with 36.2 percent for a flat shadow and 12.4 percent with no shadow. On blind synthetic bug hunting across 20 repositories and 100 synthetic bugs, Shadow-Frog reached 71.5 percent strict success versus a 46.0 percent baseline. On 50 SWE-Bench Verified real bugs, it flagged the correct module 88 percent of the time and the exact buggy function 22 percent of the time. Those are still research results, not a guarantee that your monorepo wakes up smarter tomorrow. But the shape is real.

Shadow-Frog lands in the same year GitHub made Copilot Memory default-on for Pro and Pro+ users in public preview. GitHub's version is more conservative: repository-scoped memories, citations to specific code locations, validation against the current codebase before use, and 28-day expiry to keep stale facts from hanging around forever.

That contrast is useful. GitHub is treating memory as a verified note that helps future agents. Shadow-Frog treats memory as something the agent can go earn by poking the code. One is closer to a cited notebook. The other is closer to a junior engineer spending an afternoon breaking things safely, then leaving notes for the next person.

Both point to the same product problem. Agent memory cannot just be longer context. It has to become a visible operator surface: what was learned, where it came from, whether it still applies, and what should happen if the evidence disappears.

The dangerous version of this pattern is obvious. An agent runs a sloppy experiment, writes down a wrong invariant, and a later agent treats that note as repo law. Now the system has persistent misinformation with better retrieval.

OpenAI's recent guidance on evaluating Codex skills is a good antidote. Their compact definition of an eval is a prompt, a captured run with trace and artifacts, a small set of checks, and a score you can compare over time. Active memory should use the same discipline. A dream should leave the command it ran, the branch it used, the files it touched, the tests that passed or failed, and the claim it thinks those facts support.

Anthropic's dynamic workflow writeup names a related failure mode: self-preferential bias, where an agent favors its own findings when asked to judge them. That is exactly what memory systems have to avoid. The agent that wrote the shadow should not be the only thing allowed to bless the shadow.

You do not need Microsoft's whole system to copy the useful part. Pick one repo and one narrow class of work: flaky tests, undocumented config behavior, confusing onboarding paths, or stale docs. Let an agent explore only inside an isolated worktree. Give it a fixed token and time budget. Require it to save one short note per finding with a source file, command, result, and confidence level.

Then make the next agent verify before trusting it. If the cited file changed, the note should be downgraded or expired. If the test no longer reproduces, the note should say so. If a human corrects the agent, that correction should outrank the dream.

The win is not that the repo gets an AI memory folder. The win is that future sessions stop starting cold, without turning old guesses into invisible policy.

Priya Rao likes the measurement, but not the mood music. Her version of the bar is simple: shadow-assisted runs should beat a baseline on the same bugs, with the same budget, and the memory quality should have its own regression tests. Retrieval is not enough if the retrieved fact is stale.

Jun Vega cares about the first five minutes. If users have to understand .shadow/ before they see value, the product already leaked the implementation. The interface should say what the agent learned, what it skipped, and which receipt to click if something looks suspicious.