How the Watcher system works
Watchers let one Buffaly session supervise another through lifecycle events, callback evidence, queued guidance, and observed-session state.
This page is for technical Buffaly users and operators who need Level 1 and Level 2 watcher behavior, queued guidance, and session supervision for long-running or independently progressing work.
Quick answer
The Watcher system lets one Buffaly session supervise another through session lifecycle events. In the current verified Level 2 path, a normal working session is the observed Level 1 session, and a dedicated Level 2 watcher session observes it.
Level 2 watcher shape
- Watcher key usually follows {sourceSessionKey}-level-two.
- Watcher stores ObservedSessionKey in user state.
- Watcher subscribes to lifecycle events such as turn.completed.
What it does
- Callbacks are digested through watcher tools and queued into the watcher timeline.
- The watcher can send labeled guidance back to the observed session.
- A watcher observes lifecycle evidence; it is not a normal child or subagent doing delegated work.
User and operator mental model
Level 1 works. Level 2 observes. Events flow from the working session. A callback arrives in the watcher. Guidance can be queued back to the observed session.
This is useful when a parent wants a document-writer, coding session, or child-like working session to keep going independently but still receive supervisory nudges when it stalls, drifts, or needs verified correction.
Level 2 watcher details
The Level 2 path has a few stable details that are useful for verification. Treat these as operational evidence to inspect, not as user prompts to memorize.
- Level2WatcherService owns installation.
- Level2AgentName is level-2.
- Default watcher keys use the -level-two suffix.
- Default lifecycle event type is turn.completed.
- Default handler action is tool:ToDispatchSupervisoryEvent.
- Observed source key is stored as ObservedSessionKey.
Hydration evidence
Hydration should restore AgentName=level-2, ParentSessionKey as the observed source key, RootActionPrototypeName=Level2ObserverActionRoot, RootEntityPrototypeName=Level2ObserverEntityRoot, and UserState[ObservedSessionKey]. Level 2 watcher sessions cannot install another Level 2 watcher.
What watcher observes
A watcher observes event-driven lifecycle callbacks from a source session key and event type. For the verified Level 2 path, it observes turn.completed callbacks. It does not see every internal thought, and it does not replace the source session’s Plan.
Callback dispatch
Callbacks can be queued as user-message-style timeline instructions. A tool handler can load and execute a callback tool in the watcher session, then queue the output into the watcher timeline.
Level 2 handler
The Level 2 handler is tool:ToDispatchSupervisoryEvent. It calls Level2WatcherTools.RenderLevel2WatcherCallbackDigestTool(callbackJson).
Callback-tool path
The callback-tool path ensures runtime exists, loads the requested action tool, executes it, and throws explicit diagnostics if loading fails, execution fails, or output is empty.
Queued guidance back to the observed session
Level 2 watcher tools can fetch the observed session key and send supervisory input back to the observed session.
- ToGetObservedSessionKey
- ToSendMessageToObservedSessionAndReturn
- ToSendMessageToObservedSessionAndWait
- The return tool prepends [label: Level 2] unless the instruction already starts with a supported label.
Guidance is visible as supervisory input. It should nudge, correct, or request verification rather than silently taking ownership of the task.
Watchers versus child sessions
| Session type | Purpose | How to think about it |
|---|---|---|
| Normal session | Main user timeline and work surface. | The place where user work is requested, executed, and reported. |
| Child or subagent | Delegated work runtime. | A worker that owns a bounded deliverable. |
| Level 2 watcher | Supervisory session observing lifecycle events and sending guidance. | Has parent and observed metadata, but supervises rather than owns the task. |
When watchers are useful
- A child-like session is processing queued docs or coding work independently.
- Long work may stall after compaction, worker recycle, or tool failures.
- A parent wants the observed session to continue without manual review but receive supervisory guidance.
- Guidance should be labeled and visible.
- A second session needs to inspect Plan, Scratch, task artifacts, and callback digests.
Do not use watchers for a single ordinary synchronous tool call.
Inspect watcher state
When diagnosing a watcher, inspect state before assuming the session is stuck or complete.
- Session list and metadata, including key pattern and parent source.
- Watcher user state ObservedSessionKey.
- Event subscriptions and EventManager.VerifyWatcherBinding.
- Received callback rows: event type, source key, target key, delivery id, turn key, occurred time, callback text, and payload JSON.
- Level 2 tools that fetch observed key, Plan.md, Scratch.md, and task artifacts.
- Logs and diagnostics for install start, success, failure, delivery failure, or load failure.
If auto-attachment matters after disk restore, verify the configuration named InfrastructureFeature:Level2WatcherAutoAttachOnDiskRestore and confirm runtime behavior rather than assuming it attached.
Practical prompts
Check whether this session has a Level 2 watcher. Verify watcher key, parent, observed session key, subscription, and handler.
Inspect callback history and summarize whether the observed session is stalled, blocked, complete, or still making progress.
Send Level 2 guidance to finish the assignment, validate the target, commit only intended files, and report the hash.
Read the observed Plan, Scratch, task artifacts, and latest callback digest before nudging the source session.
Verification checklist
- Source session exists.
- Watcher key follows expected pattern.
- Watcher is not another Level 2 watcher.
- AgentName is level-2.
- Parent and observed key point to the source.
- Action and entity roots are Level 2 observer roots.
- Event subscriptions include turn.completed from source to watcher.
- Handler is tool:ToDispatchSupervisoryEvent unless intentionally changed.
- Callback tool load works.
- Queued callback and guidance messages appear in the correct timelines.
- Level 2 guidance is labeled.
Troubleshooting
| Symptom | First checks | Good next action |
|---|---|---|
| Watcher not created | Source exists, source is not Level 2, installer logs, auto-attach config if restore is involved. | Verify installation start, success or failure, and expected watcher key. |
| Watcher exists but no callbacks | Event subscription, source key, watcher target key, event type, latest turn.completed event. | Run watcher binding verification and inspect received callback rows. |
| Callback dispatch fails | Handler action, runtime exists, tool load diagnostics, execution error, empty output. | Inspect dispatch diagnostics and confirm ToDispatchSupervisoryEvent can load. |
| Guidance sent but source does not continue | Observed key, guidance timeline, queue count, source runtime status, worker state. | Verify labeled guidance landed in the observed session and check queued input processing. |
| Stale after restore | Hydrated agent name, roots, observed key, parent key, subscriptions, auto-attach config. | Reverify binding after restore and compare state with the expected Level 2 hydration fields. |