Inspecting a running space

Radia derives operational views from records, live interests, envelopes and the event log. These views show what work occurred, who can receive new work, why records are waiting and whether the audit history verifies.

Mine flows from lineage

Radia does not require a declared workflow graph. Workers independently publish interests and produce records, so the observed workflow is the lineage those records form at runtime.

Flow mining groups causally connected records, reduces each group to a sequence of kinds and counts recurring shapes. The output describes recorded behavior without introducing a second routing declaration.

what happened what recurs never finished group, reduce, count 73x 100% job ⇒ pieces ×4 → results ×4 → summary the pipeline, found without anybody describing it 180x 100% a request → a reply the boring one, and most of your traffic 12x 8% upload → scan starts often, finishes almost never
Each shape includes exemplar record ids, allowing an operator to inspect the lineage behind the aggregate.
radia flows
 180x  100%  n= 2     1.3s  a request → a reply
 104x  100%  n= 7     7.6s  conversation ⇒ request → request + message → reply
  73x  100%  n= 4   810ms  conversation ⇒ tool call → message + result
   3x  100%  n=10   229ms  job ⇒ pieces ×4-7 → results ×4-7 → summary

3905 records over 14 kinds, 218 subgraphs, 2 cut as hubs (--hub-degree 0 to leave them whole)

Each shape reports occurrence count, completion rate, record count and elapsed time. The final line reports scan bounds and any structural cuts.

Callers may also sum numeric body fields by shape. This can attribute additive values such as token count or provider-reported cost. The console renders these sums as a flame view; timing uses a separate waterfall because concurrent durations are not additive.

Completion rates and repeated steps distinguish common failure shapes. An incomplete branch can indicate missing downstream work, while repeated attempts can indicate a failing worker.

Flow scans are bounded. The report identifies hubs removed from a shape, fragments whose parents fell outside the scan and results truncated by a configured limit.

Inspect live routing interests

Flow shapes describe past behavior. Live interest records describe the patterns active runs currently listen for.

A dry run matches a proposed record against those interests without writing it.

POST /v0/ops/dry-run
$ curl -s $SPACE/v0/ops/dry-run -H "$AUTH" \
    -d '{"kind":"document","body":{"type":"pdf","classification":"confidential"}}'

{ "kind": "document",
  "interests": [
    { "run": "run:01JB2Q3D7M", "agent": "agent:ocr", "match": {"type":"pdf"} }
  ] }

The response identifies matching workers after applying their grants, including content scopes that exclude the proposed record.

Interests are live only while their owning runs are active. An empty answer means no live, authorized interest currently matches; it does not prove that no worker definition exists.

Live interests and mined flows answer different questions A proposed confidential PDF is compared with active worker interests and their grants to show who could claim it now. Separately, committed records and parent links are mined into recurring shapes that show what actually happened before. now: declared intent proposed record confidential PDF interest ∧ grant match active, authorized runs agent:ocr could claim while its run remains live history: observed behavior committed records + parents mine and count collapse repeated shapes what actually ran with exemplars as evidence
A dry run predicts present routing from best-effort declarations. A mined flow summarizes committed history and links every reported shape back to records that exhibit it.
Interests state current intent, not recorded behavior. Publication is best-effort, so a worker that cannot publish an interest is absent from this view. Mined flows remain the evidence of what workers actually processed.

Diagnose waiting and stuck work

An operator asks a narrower question, and wants it answered in one command. Work that has been sitting unclaimed for too long is the usual complaint, and the count on its own is close to useless, because two completely different faults produce the same number.

radia doctor
available=1204  leased=8  consumed=39112  dead_letter=3
dead-letter: 3
stuck leases: 2 (expired but still held)
stale available: 47 (11 orphaned: nothing is listening; 36 starving: a listener is not claiming)
  orphaned invoice 01JB2Q7X4KJ8ZN0R5V3M9WPHTC
  an interest is a worker's own declaration and publishing one is best-effort, so
  'orphaned' means no live interest MATCHES, not that nothing is listening.
event chain: 39112 links, UNSIGNED (detects corruption, not a rewrite; set RADIA_SEAL_KEY)

Orphaned records have no matching live interest. Starving records have a matching interest but remain unclaimed. The first points to missing workers or patterns; the second points to a worker that is present but not taking work.

Remediation accepts the same envelope selectors as diagnostics. Operators can reclaim expired leases, dead-letter records or requeue work in bounded batches.

Query events and lineage

Record and lease mutations append events in the same storage transaction. Claims, settlements, retries and writes therefore have corresponding committed log entries.

Parent links support bounded traversal in both directions. Ancestor queries show what produced a record; child queries show records derived from it.

radia lineage
● 01JB2Q9F3H  summary
  └ 01JB2Q7X4K  document
    └ 01JB2Q1A8P  upload_batch

A graph around a hub record can reach many sibling branches and exhaust its node budget. Descendants-only traversal provides a narrower view from a selected record, and every bounded response reports truncation.

In the chat example, each turn forms a subtree rooted at the user's message. A descendants-only view isolates that turn. The waterfall view places its records on a time axis to show model and tool latency.

the whole conversation (capped, and it says so) one turn, descendants only, time as the axis conversation 150 shown of 346 from here down question route answer tool call reply done 0s 14s the tool is where the seconds went
Descendants-only traversal isolates one turn from the conversation hub and its sibling turns.

Lineage records data provenance, not authority. Delegation context is derived separately from the claimed lease; a record does not confer the permissions of its parents.

The same structure can be exported as OpenTelemetry: radia otlp --to http://localhost:4318 --follow streams threads into Jaeger or Tempo as traces, with each work attempt as a span, the claiming agent as the service, and labels and delegation context as searchable attributes. The command sends OTLP over HTTP to an existing collector.

Verify event-log integrity

Finalized events are sealed into a hash chain. Each link covers the preceding seal, the event and the referenced record's body hash.

radia integrity
chain OK: 39112 of 39112 links verified, signed
head 39112 8f3c1d0ba2e47591…
14+ events not yet sealed (sealing follows the finality watermark)

Sealing follows the storage finality watermark. Events above the watermark are committed but may not yet be sealed.

An unsigned chain detects corruption but not a deliberate rewrite. A signing key stored outside the database makes database-only changes detectable. External checkpoint anchoring is not implemented.

Integrity inspection also reports erasures that no longer hold. If previously shredded bytes are uploaded again, their content address becomes readable again; the runtime reports this condition instead of silently claiming that erasure still holds.

Use the web console

The Space tab streams live activity. Flows renders mined shapes and links to their exemplar records. Kinds draws the current routing topology from live interests. Overview ranks diagnostic findings and links each sample to the records behind it.

See it running →