Activity Control Plane

In MeshWeaver, every operation on an activity is a patch on the activity's content — not a separate message type. The owning hub watches its own MeshNodeReference stream and reacts to property changes. This is the canonical pattern for any node type that has state-machine semantics: Activity, and any custom NodeType you build.

If you find yourself reaching for Cancel<X>Request, Pause<X>Request, Retry<X>Request, or any verb-shaped message, stop and use a property instead. Control plane flow: the caller patches RequestedStatus; the owning hub's WatchControlPlane reacts and writes Status back — no verb-shaped messages.

🚨 Absolute rule — long-running work belongs on its own hub

Every long-running operation runs on an Activity hub. Not on the mesh hub, not on the per-NodeType hub, not on a singleton service's captured IMessageHub.

The mesh hub must stay responsive. If it spends seconds running Roslyn, scoring queries, or waiting for HTTP, it blocks routing for every other delivery in the silo.

The activity hub is the execution sandbox: created by the owner when work starts, holding the work's state in its own ActivityLog MeshNode, and writing results back to the owner via the synchronization protocol when done. The owner stays responsive throughout — watching its own MeshNode for updates while serving all other traffic.

🔑 Identity: the activity hub acts under the activity owner (the node's CreatedBy) for every context-less write — status flips, progress, the terminal write-back over the sync protocol. The owner is injected on the hub and carried forward via CircuitContext (a deferred sync write whose live AsyncLocal is gone still resolves the owner, never a null context). See Owner Injection.

Why this is the default

The ActivityLog content record

public record ActivityLog(string Category)
{
    public ActivityStatus Status { get; init; }            // What's actually happening
    public ActivityStatus? RequestedStatus { get; init; }  // What the user wants
    public ImmutableList<LogMessage> Messages { get; init; } = [];
    // ... other fields
}

• Status is read-only from the user's perspective. Only the owning hub writes it, as the activity transitions through Running → Succeeded / Failed / Cancelled.
• RequestedStatus is the control input. Users, other hubs, and MCP agents patch this to drive the activity into a new state.

Cancelling a script — the canonical example

When a user clicks "Cancel", the click handler does exactly one thing:

ctx.Host.Hub.CancelActivity(ctx.Host.Hub.Address.ToString());

The activity hub's initialization subscribes to its own MeshNodeReference stream and watches RequestedStatus. On a transition to Cancelled, it triggers the underlying cancellation — in the kernel case, dispatching CancelScriptRequest to the executor child hub (an internal message, not part of the public API). The script's CancellationToken trips, OperationCanceledException flows through the executor's normal completion path, and Status flips to Cancelled.

Notice that the user never posts a "cancel" message — they just patch the content.

Applying the pattern to your own NodeTypes

When you build a custom NodeType with state-machine semantics — a long-running job, a transitional resource, anything with start / pause / resume / retry / cancel — follow this shape:

1. Define your content record with two paired fields: Status (current actual state) and RequestedStatus (the control surface).

   🚨 If you want hub.WatchControlPlane(...), the field MUST be named RequestedStatus on a content type that materialises as an ActivityLog. WatchControlPlane is hard-wired to node.ContentAs<ActivityLog>()?.RequestedStatus — it does not take a selector. A differently-named control field (RequestedAction, RequestedTransition, …) compiles fine and is written fine, and the watcher simply never fires: the write lands on the node and nothing reacts. If your control surface genuinely is not an ActivityLog.RequestedStatus, use WatchSubmission (below) with your own fingerprint/predicate instead — do not rename the field and expect WatchControlPlane to find it.

2. In WithInitialization, subscribe to the hub's own MeshNodeReference stream and react to changes in the requested field. Use .DistinctUntilChanged() so you only fire on real transitions.

3. Your hub is the sole writer for Status. Users never patch it directly — they patch RequestedStatus. The hub reads the request, does the work, and writes the resulting Status.

4. No new request/response message types. Don't add Cancel<X>Request, Pause<X>Request, etc. The control plane is the content.

public record JobContent(string Category) : ActivityLog(Category)
{
    // ActivityLog already has Status + RequestedStatus + Messages.
    // Add your domain fields here.
    public string? InputPath { get; init; }
    public string? OutputPath { get; init; }
}

public static class JobNodeType
{
    public static MeshNode CreateMeshNode() => new("Job")
    {
        // ...
        HubConfiguration = config => config
            .AddMeshDataSource(s => s.WithContentType<JobContent>())
            .WithInitialization(hub =>
            {
                hub.RegisterForDisposal(hub.WatchControlPlane(requested =>
                {
                    if (requested == ActivityStatus.Cancelled) DoCancel(hub);
                    else if (requested == ActivityStatus.Running) DoStart(hub);
                }));
            })
    };
}

hub.WatchControlPlane(...) lives in MeshWeaver.Mesh.Contract (ActivityControlPlaneExtensions.cs). It projects the hub's own MeshNodeReference stream down to ActivityLog.RequestedStatus, applies DistinctUntilChanged, and forwards each transition to your handler. The handler parameter is ActivityStatus? — null means no request is pending (never set, or cleared after a transition), so always compare against the specific status you handle.

Faults are handled by SubscribeHubWatcher (the hub-owned form of SubscribeWithReEstablish), not by a bare log-and-die OnError, because a dead control plane leaves the hub half-alive (process running, control plane gone). Being hub-owned also fixes the watcher's lifetime: it stops at the hub's ShuttingDown signal — the first instant the hub is part of a shutdown, its own or an ancestor's — rather than in the ShutDown phase where RegisterForDisposal would finally reach it, and it delivers nothing to a hub that is shutting down (see Hub Disposal Model → "The first instant of teardown", #3026):

Faults log to the optional ILogger argument (or the MeshWeaver.ActivityControlPlane log category) so a broken control plane never disappears silently.

🚨 WatchControlPlane and WatchSubmission install nothing on a transient node probe. A probe hub applies a NodeType's instance configuration purely so the configuration is built, and is disposed in the same breath — it has no mesh node, so a watcher of its own node has nothing to observe and nothing to re-establish. Because the ACP is installed by the adopter (from its own WithInitialization), the guard has to live at this shared seam rather than in each adopter: see Transient Node Probes for the boot-time Error line this removed and why the cure is not-installing rather than a quieter log level.

WatchSubmission — for arbitrary "needs work" triggers

WatchControlPlane is right when the trigger is a single status field (RequestedStatus). Many orchestration cases need a broader predicate — for example, a thread hub that dispatches a new agent round whenever it has unprocessed user messages and isn't already executing:

// Illustrative shape — the step builders are yours to write.
hub.WatchSubmission(
    fingerprint:   n => (t.IsExecuting, t.Messages.Count, t.IngestedMessageIds.Count, t.PendingUserMessages.Count),
    needsDispatch: n => !t.IsExecuting && t.PendingUserMessages.Count > 0,
    dispatch:      n => CreateUserCells(hub, n)
                          .Concat(CreateResponseCell(hub, n))
                          .SelectMany(_ => CommitRound(hub, n))
                          .SelectMany(_ => DispatchToExec(hub, n)));

WatchSubmission lives next to WatchControlPlane in MeshWeaver.Mesh.Contract. Internally it is:

SubscribeHubWatcher(hub, () => hub.GetWorkspace().GetMeshNodeStream()
    .DistinctUntilChanged(fingerprint)
    .Where(needsDispatch)
    // Single-flight: only one dispatch in flight per watcher, released in Finally.
    .Where(_ => Interlocked.CompareExchange(ref dispatching, 1, 0) == 0)
    .SelectMany(node => dispatch(node)
        .Catch(/* log, swallow — the next state change retries */)
        .Finally(() => Interlocked.Exchange(ref dispatching, 0))),
    …);

🚨 DistinctUntilChanged alone is NOT a single-flight gate — do not remove the guard. The upstream GetMeshNodeStream can emit several distinct fingerprints before any one dispatch's commit (CreateNodeRequest + workspace.Update) has landed, and each would start its own round with a fresh response id. Concat does not help either, because the inner dispatch can complete before its side effects settle. That is why WatchSubmission carries the Interlocked guard inside the framework helper — and why the correctness gate is the state re-check inside the stream.Update lambda (next section), not the flag.

What WatchSubmission replaces

If you find yourself writing any of these patterns inside a watcher, reach for WatchSubmission instead — it already contains the correct version of each:

• A hand-rolled Interlocked.CompareExchange "dispatching" flag wired into your own Subscribe.
• .Throttle(50ms) to coalesce rapid patches into one round.
• AsyncLocal / CircuitContext / hub-as-user fallbacks because the watcher fires on a throttle scheduler hop.
• Manual ordering of "create satellite cell then update the parent's collection" inside Subscribe callbacks.
• A bare Subscribe(_ => {}, ex => log(ex)) whose fault silently kills the watcher.

The chain runs in the hub's natural scheduler so AsyncLocal flows. Multi-step orchestration is an IObservable<Unit> chain (Concat / Zip / SelectMany) rather than mutable ordering flags.

Idempotent triggers — state lives on the node, never in memory

🚨 Absolute rule

Correctness comes from node state, never from an in-memory gate. Each hub owns its OWN state, on its OWN node. The claim is a re-check inside the stream.Update lambda (or a paired Requested<X> field cleared in the same atomic update that transitions Status). An in-memory flag may coalesce redundant work, but it must never be the thing that makes double-dispatch impossible.

Watchers that fire a one-shot trigger from an observable stream must single-flight. An in-memory Interlocked.CompareExchange(ref dispatching, 1, 0) gate is not sufficient on its own: it races on CI when the workspace stream's ReplaySubject(1) emits a stale snapshot after the gate has been released — a flicker Idle → Executing → Idle in the same hub tick can dispatch twice. The production failure mode this caused was Submit_DuringExecution_QueuedUntilRoundCompletes_ThenNextRoundDispatches (test/MeshWeaver.AI.Test/ThreadSubmissionIntegrationTest.cs): u2 ingested twice into IngestedMessageIds, two response cells, six messages in thread.Messages instead of four.

(WatchSubmission does carry such a flag internally, as a coalescer on top of the node-state claim — see the box above. That is the framework's job; it is not a substitute for the in-lambda re-check in your dispatch.)

The correct approach: the state-field check INSIDE the stream.Update lambda IS the single-flight gate. The hub's action block serialises the lambdas; the first lambda flips Status, every concurrent lambda re-reads Status != Idle and bails. No paired intent field is required when the state transition itself is atomic.

For cross-process triggers (a non-owner hub wanting to drive a mutation on the owner), use a paired intent field — single-field RFC-7396 patches are merge-safe under UpdateRemote. The watcher consumes the intent and clears it in the same atomic Update.

Existing intent/state pairs in the codebase

⚠️ NodeTypeDefinition.RequestedReleasePath is NOT an intent field — do not treat it as one. It is a durable pin: while it is set, every per-instance hub activates against that release's AssemblyPath instead of LatestReleasePath, and creating a fresh release deliberately does not touch it. Nothing consumes or clears it; instances stay pinned until a human clears or repoints it. The consumable trigger is RequestedReleaseAt (+ RequestedReleaseForce / RequestedReleaseBy). See NodeType Compilation → "Pinning an instance to a fixed release".

MeshThread.RequestedStatus is a ThreadExecutionStatus? — the request half of the same Status/RequestedStatus pair (today only Cancelled is ever requested). The GUI Stop button and a parent cancelling a sub-thread set it; the cancel watcher cancels the CTS, and the streaming loop's terminal write flips Status → Cancelled and clears RequestedStatus. Cancelled is a distinct, visible terminal status that re-dispatches like Idle when PendingUserMessages still holds input. (There is no transient Completing status — terminal writes are atomic.)

Submission-watcher implementation pattern

This is the SOLE entry point for dispatching a new round on the thread hub:

threadHub.GetWorkspace().GetMeshNodeStream()
    .Where(n => n.Content is MeshThread t
        // Idle OR Cancelled (a stopped round re-dispatches like Idle)
        && t.Status is ThreadExecutionStatus.Idle or ThreadExecutionStatus.Cancelled
        && t.PendingUserMessages.Count > 0)
    .Subscribe(_ =>
    {
        workspace.GetMeshNodeStream().Update(node =>
        {
            // 🚨 Re-check inside the lambda. The hub's action block serialises
            // concurrent emissions — the SECOND lambda sees a non-claimable
            // status and bails.
            var t = node.Content as MeshThread;
            if (t is null
                || t.Status is not (Idle or Cancelled)
                || t.PendingUserMessages.IsEmpty) return node;
            return node with { Content = t with {
                Status = StartingExecution,
                ExecutionStartedAt = DateTime.UtcNow
            }};
        }).Subscribe(_ => { /* _Exec round watcher picks up Status=StartingExecution */ });
    });

The _Exec hosted hub subscribes to the parent thread's stream via IMeshNodeStreamCache.GetStream(threadPath, options) — the cache deliberately exposes no bare GetStream(string); callers must pass JsonSerializerOptions so Content is deserialized — and fires DispatchAfterClaim on each Idle → StartingExecution transition (DistinctUntilChanged on ExecutionStartedAt). No internal trigger event — the state transition IS the dispatch signal.

Wake-up recovery — drive any non-terminal state to valid, exactly once

Core invariant: a freshly-activated hub has no in-process work. On activation, a hub must reconcile any persisted non-terminal state left by an interrupted previous activation. The rule is identical for threads and activities: read the own node stream's FIRST emission (the loaded persisted state, correctly ordered on the hub's action block vs subsequent writes) and drive the state to a valid one exactly once.

Never use a late GetMeshNode round-trip — its response can land after later writes and clobber them (that late-read race was the root of the check_inbox phantom-drain flake).

Threads — InitializeThreadLifecycle (ThreadExecution.cs)

🚨 Never write Executing → StartingExecution on recovery. That is the exact inverse of the _Exec commit edge (StartingExecution → Executing), and because BOTH the recovery observer and the exec round watcher are self-healing, the two volley under load — the re-dispatch ping-pong behind the resubmit / cold-load flake. Resume re-runs the round into its existing cell while Status stays Executing. The orphaned-claim rollback above is CAS-guarded precisely so it can never perform that inverse write.

Activities — recover from the owner's own state

Activities recover from the owner's state, where the owner hub is DISTINCT from the executor. NodeType compilation is the canonical case: the compile runs on a separate activity hub, so the NodeType hub coming up with CompilationStatus == Compiling on its first own-stream emission means the compile was interrupted. It re-requests by flipping CompilationStatus = Compiling → Pending so the compile watcher dispatches a fresh build.

Read the owner's OWN state — never probe the activity hub cross-hub. That read lags, and a false "still running" leaves the operation stranded (the rbuergi/CatBond "renders nothing" symptom).

🚨 Do NOT add a "first emission: Running ⇒ Failed/interrupted" recovery to a hub that IS the executor. Such a hub activates in order to run the work, so its own activity is legitimately Running the instant it comes up — a first-emission recovery would kill every freshly-started run. The kernel/script hub is exactly this case and deliberately has no such recovery. Restart-recovery for a hub-that-runs-its-own-work needs a separate supervisor, or the owner-state re-request pattern above.

Claim-handler pattern (clears intent in the same atomic transition)

hub.GetWorkspace().GetMeshNodeStream().Update(node =>
{
    var t = node.Content as MeshThread;
    if (t.Status != Idle) return node;              // already running
    if (t.PendingUserMessages.IsEmpty) return node; // nothing to do
    return node with { Content = t with {
        Status = StartingExecution,
        ExecutionStartedAt = DateTime.UtcNow
    }};
});

Why "each hub owns its own state": the intent field lives on the same node as the state field. The owning hub is the only writer; cross-hub coordination is impossible because no other hub knows about the field. Read-only views (UI, MCP) can show "request pending" by checking Requested<X> is not null && Status == Idle.

Ship the round-orchestration steps as small IObservable<Unit> builders — one step per Hub.Observe(..., target) or workspace.GetMeshNodeStream(path).Update(...), each followed by .Select(_ => Unit.Default), composed with Concat / SelectMany. (The builder names used in the snippets above are illustrative; the thread hub's real implementation is ThreadSubmissionServer.DispatchAfterClaim → CommitRoundAndExecute in src/MeshWeaver.AI/ThreadSubmission.cs.)

Anti-patterns to remove on sight

Finishing an activity — ActivityLog.Finish on every terminal write

A fresh ActivityLog starts at Status = Running (the enum default). Until something calls .Finish(version, status) on it, every consumer that reads the log — the response carrying it, the activity MeshNode's content stream, the UI overlay — sees Running. Long after the work is done, the activity looks live.

Every code path that owns the activity's terminal write — success OR failure — MUST call .Finish(version, status) on the log before the log escapes its caller. This includes:

• the success branch: .Finish(version, ActivityStatus.Succeeded)
• every catch / failure branch: .Finish(version, ActivityStatus.Failed)
• every early-return short-circuit (cache hit, "no work to do", validation reject) — the log is still escaping, and still needs a terminal state

Finish reads Messages and computes GetFinalStatus(): an Error message → Failed, a Warning → Warning, otherwise → Succeeded. The overrideStatus argument is a floor — the function returns MAX(overrideStatus, GetFinalStatus()). So Finish(v, Succeeded) after an AppendError correctly returns Failed. Pass the natural success status as the override; trust GetFinalStatus() to bump it on errors.

Anti-pattern: appending an error after Finish

// ❌ WRONG — log says Succeeded, has an Error message
var log = freshLog.Finish(v, ActivityStatus.Succeeded);
log = AppendError(log, "actually it failed");          // doesn't flip Status

// ✅ RIGHT — re-Finish so GetFinalStatus() reads the new Error
log = AppendError(log, "actually it failed");
log = log.Finish(v, ActivityStatus.Succeeded);         // Status → Failed

Anti-pattern: handing back a result whose log was never finished

// ❌ WRONG — log stays Running forever. Repro:
// CompileActivityLogTest.SuccessfulCompile_ReportsActivityLogWithSourceQueriesAndMatchedPaths
// (test/MeshWeaver.Hosting.Monolith.Test/CompileActivityLogTest.cs)
var log = new ActivityLog(ActivityCategory.Compilation) { HubPath = path };
return Observable.Return(new CompileResult(asm, configs, log));

// ✅ RIGHT — Finish on the way out, success or failure
var log = new ActivityLog(ActivityCategory.Compilation) { HubPath = path };
var result = BuildResult(asm, log);
return Observable.Return(result with
{
    Log = result.Log.Finish((int)hub.Version, ActivityStatus.Succeeded)
});

The same applies to every exception branch you can swallow. A try { … } catch (Exception ex) { return Fail(ex.Message); } that forgets to Finish leaves consumers polling forever.

Writing the terminal state back through GetMeshNodeStream

When the activity is a MeshNode (the canonical shape — _Activity/{id} nodes), the terminal write must land via the activity hub's own workspace.GetMeshNodeStream(activityPath).Update(...):

• It rides the data-sync protocol — every subscriber (UI overlay, agent, Activity Details panel) sees the transition as one tick of the stream.
• It debounces through MeshNodeTypeSource.Sample(200ms) like any other content edit — no extra SaveMeshNodeRequest hand-rolled.
• It cannot bypass the per-node hub's reducer, so there's no "wrote to persistence, hub still serving stale content" race.

hub.GetWorkspace().GetMeshNodeStream(activityPath!)
    .Update(current =>
        current?.Content is ActivityLog log
            ? current with
            {
                Content = log with
                {
                    Status = ActivityStatus.Succeeded,
                    End = DateTime.UtcNow,
                }
            }
            : current!)
    .Subscribe(_ => { }, ex => logger.LogWarning(ex,
        "Activity terminal write failed for {Path}", activityPath));

Do NOT write the activity's Status/Content from outside the activity hub (e.g. workspace.GetMeshNodeStream(activityPath).Update(node with { Content = … }) from another hub) — that bypasses the activity hub's reducer and the RequestedStatus control plane, and races the per-node hub's own view of its content. The terminal write above is fine because the activity hub issues it on its own stream; external callers flip RequestedStatus instead and let the reducer react.

Do NOT call IStorageAdapter.Write directly — same race, and worse because persistence is invisible to the per-node hub's MeshNodeReference cache. The next read off the node stream returns the pre-patch content.

🚨 The outside-write rule is not only about races — under Orleans it DEADLOCKS, silently. A cross-hub GetMeshNodeStream(activityPath).Update(…) is a round trip: it posts, and its response has to be processed by the hub that issued it. When the issuing hub is an Orleans grain, that hub has a single-threaded activation scheduler — so if the same turn is currently running the long operation, the turn cannot serve the response it is itself waiting on. The write never lands and nothing errors:

• the activity keeps only the first message (the one baked into CreateNode),
• every later progress line silently never appears,
• the terminal Status is never written, so the activity reads Running forever,
• meanwhile the operation itself completes — its own writes go through IIoPool under the System identity on a different path — so the side effects land and only the log looks hung.

Observed on memex 2026-08-02: GitSync activities frozen at message 1 while {space}/_GitSync.lastSyncCommitSha had already advanced to the new commit. Diagnosing this from the activity node alone is misleading — verify the operation by its own effect, not by its log. It reproduces under load (many heavy partitions, compiles in flight) and slips through on an idle mesh, which is why it survives review.

Two rules follow, and they are separate:

1. Progress must come FROM the activity hub. The activity hub owns its content; it appends its own Messages and writes its own terminal Status. Other hubs flip RequestedStatus and let the reducer react — they never patch the log.
2. Never run the activity's execution on the calling hub's turn. Hop the subscribe onto the drainable pool (IPooledSubscribeScheduler.SubscribeThroughPool, falling back to SubscribeOn(TaskPoolScheduler.Default)) so the command and its writes execute on pool threads while the hub's turn stays free to route their round trips. This is a scheduler fix, never a timeout — a bigger timeout just hangs longer. TaskPoolScheduler.Default is TaskScheduler.Default-backed, which is what makes it Orleans-safe; anything that captures TaskScheduler.Current inside a grain re-enters the activation and reproduces the deadlock. Purely reactive — no async/await, no Task.Run, no .Result.

Same failure shape and same remedy as LayoutAreaHost.ScheduleRenderSubscribe (a view generator that queries in-render wedging its own hub turn).

⚠️ ActivityRunner.Append / Finish (MeshWeaver.GitSync) still issue their writes from the calling hub's workspace — rule 1 above is not yet satisfied there. Rule 2 is (the execution is scheduled off the hub turn), which is what releases the deadlock; moving the writes into the activity hub is the remaining structural fix.

NodeTypeCompilationActivity.MarkSucceeded / MarkFailed in MeshWeaver.Graph.Configuration is the canonical implementation — copy its shape (Update through GetMeshNodeStream, with a best-effort try/catch that logs and swallows so observability never breaks the work).

When the activity is just a response field

Some handlers return an ActivityLog inline on a response (e.g. GetCompilationPathResponse.Log) instead of, or alongside, a _Activity MeshNode. The same rule applies: Finish before the response is posted. If multiple compile paths produce the response (fresh compile, cached hydration, pinned release), each must Finish at the end of its own branch. A shortcut path that hands the log straight to BuildResponse without a Finish is a "Running forever" bug.

Reporting status back to the UI

Status flows the other direction the same way — through the same content the user is patching. The owning hub writes Status (and Messages, and any other observable progress fields) on the activity's content, and every UI / agent / monitor subscribed to the node's MeshNodeReference stream gets the snapshot pushed within milliseconds.

The pattern inside scripts (and inside any worker hub doing long-running work):

• Use Log.LogInformation(...) at every coarse-grained step. Each call appends to Messages and ticks the workspace.

• For waits, subscribe reactively to whatever you're waiting on instead of looping or sleeping. The wait is naturally cancellable via the script's Ct global:

  await Mesh.GetWorkspace()
      .GetMeshNodeStream("rbuergi/downstream-job")
      .Where(n => (n?.Content as JobContent)?.Status == JobStatus.Succeeded)
      .FirstAsync()                         // ← not .Take(1): an empty completion must FAULT
      .ObserveCompletion(                   //   rather than settle silently
          ex => Log.LogWarning(ex, "downstream watch faulted AFTER the wait settled"),
          Ct);                              // ← cancels via Activity Control Plane
  

  🚨 ObserveCompletion (MeshWeaver.Messaging, imported by default in scripts), never .ToTask(Ct) — forbidden repo-wide as of 2026-08-30 — and never a bare await someObservable either. Rx's bridge and Rx's own awaiter both resume the rest of the script INLINE on the mesh thread that signalled; ObserveCompletion completes with RunContinuationsAsynchronously, so the hub thread is released the moment it signals.

• For compute-heavy steps with no natural log lines, an Observable.Interval heartbeat keeps the user informed:

  using var heartbeat = System.Reactive.Linq.Observable
      .Interval(TimeSpan.FromSeconds(5))
      .Subscribe(t => Log.LogInformation($"Still working — {t * 5}s elapsed"));
  try { await DoLongWork(Ct); } finally { heartbeat.Dispose(); }
  

• Never Thread.Sleep and never Task.Delay(ms) without Ct. Both ignore cancellation and are indistinguishable from a hung script while they wait.

Subscribers (UI stripes, activity-details views, agents watching their job) read the same content via workspace.GetMeshNodeStream(activityPath) and project whatever field they care about (Messages.Count, Status, RequestedStatus, your own progress field). One source of truth, one observation pattern, no parallel "events" channel.

Operations as scripts — the canonical shape for export, import, compile, …

Once you've internalised the property-driven control plane, the next step is: don't write a bespoke handler at all when the work is an operation with inputs, multiple steps, progress to report, or a meaningful output. Express the operation as a Code MeshNode template that the kernel runs as an Activity. This is how export, import, compilation, and any "user kicks off a job" surface should be modelled going forward.

Why "operation = script"

The shape end-to-end

Five pieces, all of which already exist in the framework — no new infrastructure required:

1. The script template — a Code MeshNode, e.g. Doc/Templates/Code/ExportPdf.
2. The form — a layout area that binds a node-bound editor DIRECTLY to the activity node's content, so each field edit persists through GetMeshNodeStream(path).Update with no intermediate copy (see Asynchronous Calls — "Reactive in click actions").
3. The trigger — a click handler that posts ExecuteScriptRequest against the Code template. Sync, fire-and-forget, no await.
4. The activity — created automatically by the script-execution path; see Script Execution.
5. The result panel — subscribes to the activity via GetMeshNodeStream(activityPath) and projects Messages plus the terminal output.

Worked example: export-as-script

1. The script template (a Code MeshNode seeded by the export module)

// ExportPdf.csx (lives as the Code property of a seeded MeshNode at e.g.
// Doc/Templates/Code/ExportPdf — IsExecutable = true).
//
// Inputs arrive via the activity's content (Title, IncludeChildren, MaxDepth,
// BrandNodePath). The script reads them off Mesh.GetMeshNode of its own activity.
var activity = await Mesh.GetMeshNode(Mesh.NodePath);
var inputs   = activity!.Content as ExportInputs ?? new ExportInputs();

Log.LogInformation("Loading source markdown {Path}", inputs.SourcePath);
var src      = await Mesh.GetMeshNode(inputs.SourcePath);
var md       = (src?.Content as MarkdownContent)?.Content ?? "";

Log.LogInformation("Resolving branding");
var branding = await Mesh.GetWorkspace()
    .GetMeshNodeStream(inputs.BrandNodePath)
    .Where(n => n is not null)
    .Select(n => (n!.Content as CorporateIdentity).ToOptions())
    .FirstAsync()
    // A top-level await is fine in a kernel Script; the BRIDGE is what is constrained —
    // ObserveCompletion, never Rx's ToTask and never a bare `await someObservable`.
    .ObserveCompletion(
        ex => Log.LogWarning(ex, "branding read faulted AFTER the wait settled"),
        Ct);

Log.LogInformation("Rendering");
var doc   = new DocumentBuilder().Build(src!.Name, [(src.Name, md)], inputs.Options, branding);
// The PDF renderer composes the document into print HTML and prints it with the headless
// browser, so it returns a COLD IObservable<byte[]> — the work runs on subscription.
var bytes = await Mesh.ServiceProvider.GetRequiredService<PdfDocumentRenderer>()
    .Render(doc).FirstAsync()
    .ObserveCompletion(
        ex => Log.LogWarning(ex, "PDF render faulted AFTER the wait settled"),
        Ct);

Log.LogInformation("Writing {Bytes} bytes to content collection", bytes.Length);
var outputPath = $"{inputs.TargetCollection}/{Sanitize(src.Name)}.pdf";
// … write bytes via the content-collection service …

return new ExportOutput(outputPath, "application/pdf", bytes.Length);

The script uses the public renderer types directly (PdfDocumentRenderer, DocumentBuilder) — no service layer in the middle. PdfDocumentRenderer is resolved from DI because it wraps the browser leaf; DocumentBuilder is a plain new. The kernel exposes Mesh, Log, Ct globals; that plus the public types is enough.

2. The form layout area — bind the inputs DIRECTLY to the node

🚨 Never replicate the activity node into a layout-area /data/{id} copy plus a save subscription. There is no SetupAutoSave helper in the framework, and there must not be one: two stores drift, and the save loop clobbers fields the form did not touch. Declare a node-bound editor and let the GUI read and write the node stream itself.

return Controls.Stack
    // ONE source of truth: the editor binds to the activity node's own content
    // and each field edit writes straight back through GetMeshNodeStream(path).Update.
    .WithView(MeshNodeContentEditorControl.ForType(activityPath, typeof(ExportInputs)))
    .WithView(Controls.Button("Export")
        .WithClickAction(ctx => Trigger(ctx, activityPath)));

ExportInputs' own property attributes ([Description], [UiControl<T>], [Editable(false)], [MeshNode("query")] for a node picker) decide the controls — the same Edit-macro surface every other form uses. The form never reads its own state inside the click action; the binding already persisted it. The click is O(1) (see Asynchronous Calls — "Reactive in click actions: use stream.Update, not `Take(1) + Subscribe + manual node write"). Full rules: GUI Data Binding.

3. The click handler — patch RequestedStatus = Running and submit

private static Task Trigger(ClickAction ctx, string activityPath)
{
    // Flip RequestedStatus → Running. The Activity hub's WatchControlPlane
    // subscription picks it up and submits the script to the kernel.
    ctx.Host.Hub.RequestActivityStatus(
        ctx.Host.Hub.Address.ToString(), ActivityStatus.Running);
    return Task.CompletedTask;
}

WatchControlPlane(...) wires the activity hub to react to RequestedStatus = Running by posting SubmitCodeRequest to its own kernel — exactly the same plumbing that powers Cancel, just for Start.

4. The result panel — subscribe to the activity stream

return host.Hub.GetWorkspace()
    .GetMeshNodeStream(activityPath)
    .Select(node => node?.Content as ActivityLog)
    .Where(log => log is not null)
    .Select(log => Render(log!));

Render switches on log.Status — for Running, show the streaming message list; for Succeeded, render the output (a download link to the bytes, or a link to the saved node's path); for Failed / Cancelled, show the terminal Messages and the reason. One subscription, one switch, all states handled.

Cancel, retry, status — all free

Because the operation runs as an Activity, all standard control-plane operations are inherited with no extra code:

• Cancel = hub.CancelActivity(activityPath) — writes RequestedStatus = Cancelled onto the activity node via its stream. The kernel-hosted run sees Ct flip and exits.
• Retry = create a new run by patching RequestedStatus = Running again on a fresh activity. The form panel does this by posting another ExecuteScriptRequest.
• Status / live progress = Status + Messages on the same content the form already binds to.

When to keep a static request handler instead

"Operations as scripts" is the right shape whenever the work is observable. It's not the right shape for everything:

If the operation has any of these: form inputs to collect, multiple steps, progress users want to watch, a meaningful output worth keeping in the activity history, or a "what's the status?" question — script execution is the canonical shape. Default to it.

Migration checklist — turning a request handler into a script-driven Activity

1. Identify the inputs. Whatever the request type carried becomes the activity's content record (ExportInputs, ImportInputs, …) — the same shape, at rest on a MeshNode instead of in flight on a message.
2. Write the template script. Move the handler body into a .csx-shaped string on a Code MeshNode; swap the request fields for (Mesh.GetMeshNode(Mesh.NodePath).Content as XxxInputs); replace hub.Post(response) with return result;.
3. Bind the form. Every input the handler used to read off the request becomes a property on the activity's content record, rendered by a node-bound editor (MeshNodeContentEditorControl.ForType(activityPath, typeof(XxxInputs))) that writes each edit straight back to the node. No /data replica, no save subscription.
4. Click submits. The click patches RequestedStatus = Running (or posts ExecuteScriptRequest against the template Code node). No await, no synchronous reads of form state.
5. Result panel subscribes. The bottom half of the layout area subscribes to the activity stream via GetMeshNodeStream(activityPath), switches on Status, projects progress and terminal output.
6. Delete the request type and handler. Or keep it as a transitional shim marked [Obsolete] if external callers still depend on it — but the new path is the script.

If a step in the migration feels awkward, that's a sign the operation isn't a good fit — re-check the table above.

Routing activities to a different partition

By default a script's activity lands at {partitionRoot}/_Activity/{guid} — the partition the Code node lives in. For shared or read-only partitions (the docs partition is the canonical example) you usually want each viewer's runs to land in the viewer's own home instead, so each user's activity feed shows their own history independent of who else is browsing.

Two layered hooks resolve this, in order:

1. Per Code node: set CodeConfiguration.ActivityParentPath on the Code node itself. The literal sentinel "{viewer}" expands to the calling user's home (their AccessContext.ObjectId); any other value is taken as-is.
2. Per partition: set PartitionDefinition.DefaultActivityParentPath on the partition's Admin/Partition/{Name} node. Applies to every Code node in that partition that doesn't override step 1. Same "{viewer}" sentinel rule.
3. Default: the partition root.

The partition lookup is reactive — PartitionRegistry.GetPartition(namespace) returns an IObservable<PartitionDefinition?> cached via Replay(1).RefCount(), composed into the create-activity chain. Updating a partition's DefaultActivityParentPath at runtime takes effect immediately for subsequent runs.

Wiring example: docs partition routes to user

The built-in MeshWeaver documentation partition does this exactly:

builder.AddMeshNodes(new MeshNode("Documentation", "Admin/Partition")
{
    NodeType = "Partition",
    Name = "MeshWeaver Documentation",
    State = MeshNodeState.Active,
    Content = new PartitionDefinition
    {
        Namespace = "Doc",
        DataSource = "EmbeddedResource",
        Versioned = false,
        DefaultActivityParentPath = "{viewer}"   // ← every script run lands in the caller's home
    }
});

After this, any executable Code node under Doc/... that a viewer triggers writes its activity to {viewer}/_Activity/{guid} — e.g. rbuergi/_Activity/abc... for the user rbuergi. The viewer sees the run in their own activity stripe; the originating Code node's LastActivityPath field still points back to the cross-partition activity for the Output pane.

Use this any time the partition is shared (docs, demos, reference data) and the runs are user-specific. Skip it for partitions where runs belong to the partition itself (a per-tenant data pipeline whose runs are tenant-scoped, not viewer-scoped).

When to break the rule

A few situations legitimately call for something other than the property pattern:

• Cross-hub one-shot triggers that don't belong to a long-lived entity (e.g. "render this report once") may still be plain request/response — there's no living state to patch.
• Internal hub-to-hub plumbing (e.g. the kernel executor's CancelScriptRequest) is fine as a message type because it's an implementation detail hidden behind the public content-driven surface.

If you're not sure: default to the property pattern. It scales better as the number of operations grows, and it forces you to think clearly about the state your nodes actually live in.

Recovery on activation — activities must self-heal too

The control-plane property pattern makes an activity cancellable and retryable, but it does not by itself make it crash-safe. An activity hub can activate onto a node a previous process left mid-run — Status = Running with no live worker behind it (portal restart, Orleans grain deactivation, a seeded post-crash node). If the watcher that drives Running → Completed/Failed only existed in the dead process, the activity is stuck Running forever and every observer (Take(1) on the activity stream, a parent waiting on it) parks.

Apply the same resurrection contract as threads (see ThreadOperations → resurrection on activation and DebuggingMessageFlow → resurrection on init) on the activity hub's init:

• Re-establish, never give up on the loaded-state read — no Take(1).Timeout(N)-then-abandon. If the observation faults before it drives the node to terminal, re-subscribe.
• On activation, drive any non-terminal Status to a valid one: a Running activity with no resumable worker → Failed/Cancelled (so observers unblock and the user can retry); a RequestedStatus left pending → honor it.
• Guarantee terminal with a no-progress watchdog so a wedged Running activity can never hang an observer indefinitely.
• If any observer dies before the activity reaches a terminal Status (Completed/Cancelled/Failed), restart the watcher.

The trace signature is identical to the thread case: a burst of work then silence (the worker finished or died and nothing reported the terminal status) is a missed observation, not a lock — fix the observer, don't bump the timeout.

Related

• Request via Stream Update — the general rule this page is the activity-shaped instance of.
• Transient Node Probes — the throwaway hubs this watcher is deliberately not installed on, and the three own-address read seams that answer directly.
• Logon Actions — per-user work at logon. Its run-once ledger is the same claim-before-you-mutate shape as the Idle → StartingExecution flip above, with the ledger key as the claim: the guard lives inside the update lambda so a rebased patch re-reads it and no-ops, which is what makes two concurrent logons apply a migration exactly once.