NodeType Release Redesign

This document is the original design proposal for first-class Release MeshNodes, which superseded the implicit edit-then-invalidate-cache compile flow with an explicit, observable, version-pinned release pipeline.

🚨 This design SHIPPED, under different names and a different trigger. Read the sections below as the historical rationale, not as an API reference — and never copy their code. What actually exists today:

In this proposal	What shipped
Release : ActivityLog	NodeTypeRelease (MeshWeaver.Graph.Configuration) — a plain record, not an ActivityLog. It mirrors the compile's terminal Status and links the run via CompilationActivityPath.
CreateReleaseRequest / CreateReleaseResponse	The trigger is a stream.Update control-plane field: set NodeTypeDefinition.RequestedReleaseAt (+ RequestedReleaseForce) via workspace.GetMeshNodeStream(nodeTypePath).Update(...), or call hub.RequestNodeTypeRelease(...). A CreateNodeTypeReleaseRequest/Response pair still exists as legacy plumbing — never post one from new code (see Request via stream.Update).
NodeTypeService.GetCachedConfiguration / GetActiveReleaseStream	NodeTypeService no longer exists. The active release is the NodeTypeDefinition.LatestReleasePath field on the NodeType node — read it directly, do not resolve the active release with a Query (that round-trip is exactly what the field replaced). RequestedReleasePath pins a specific historical release.
InvalidateCache on NodeTypeService	ICompilationCacheService.InvalidateCache(nodeName) — still present, on the cache service.
AssemblyPath as the durable artefact	AssemblyPath is a process-local hint; the cross-silo durable reference is AssemblyCollection + AssemblyContentPath (content-collection blob) plus AssemblyStoreVersion.

The C# blocks below violate current platform rules (Observable.FromAsync, a blocking .Wait(), an unsubscribed UpdateMeshNode). They are annotated in place and kept only to show what was proposed. See Asynchronous Calls and Controlled I/O Pooling.

Why the old flow breaks down

The current NodeType compile flow is implicit and entirely process-local:

1. A user edits a NodeType or one of its Source/ children.
2. The change-feed fires NodeTypeService.InvalidateCache(nodeTypePath), which clears in-memory dictionaries.
3. The next access triggers a Roslyn compile and loads the result into a process-local AssemblyLoadContext (ALC).

That sounds simple, but there are four failure modes that compound one another.

The ALC key mismatch. Release-based ALCs are keyed in _loadContexts by release.Path. InvalidateCache looks up by nodeName, finds nothing, skips the GC sweep, and File.Delete on the cached .dll throws UnauthorizedAccessException. This is what kept CodeEditRecompileTest (test/MeshWeaver.Hosting.Monolith.Test) skipped at the time. (It runs today — the test is no longer skipped.)

No observable feedback. Users have no signal while a compile is running, when it finishes, or whether it succeeded. Diagnostics are read on demand via GetCompilationError(nodeTypePath) — a polled, in-memory dictionary.

No rollback. Old assemblies are deleted before the new compile starts. If the new compile fails, there is nothing to fall back to.

No history. Once a compile succeeds, the previous version is gone. There is no audit trail of what changed and when.

The redesign goal: treat releases as first-class, observable, versioned MeshNodes so the framework's existing Activity Control Plane machinery handles progress, cancellation, diagnostics, and rollback automatically.

The model

Each Release is a MeshNode of type Release at {nodeTypePath}/Release/{version}. Versions are user-supplied or auto-stamped (timestamp + short hash). A Release owns its own .dll on disk at a path that is stable for the (nodeTypePath, version) pair. Releases accumulate; old ones remain as history.

Schema

🚨 Proposed shape, not the shipped one. The real type is NodeTypeRelease (MeshWeaver.Graph.Configuration) — a plain record that does not derive from ActivityLog. It carries Status as a mirrored string plus CompilationActivityPath (the link to the live message log), and adds AssemblyCollection / AssemblyContentPath / AssemblyStoreVersion for cross-silo activation and SourceVersions / TestVersions snapshots. Code against that type.

public sealed record Release : ActivityLog("NodeTypeRelease")
{
    /// <summary>
    /// The NodeType this release was built from. Stable across the release's
    /// lifetime; a release belongs to exactly one NodeType.
    /// </summary>
    public required string NodeTypePath { get; init; }

    /// <summary>
    /// User-supplied version label (e.g. "1.2.0", "feature-x"). When null,
    /// auto-stamped by the create handler with a timestamp + 8-char hash of
    /// the compilation inputs.
    /// </summary>
    public string? Version { get; init; }

    /// <summary>
    /// Release notes — markdown body the author writes to describe the
    /// release. Surfaces in the UI release history list and at the top of
    /// the Release detail view.
    /// </summary>
    public MarkdownContent? Notes { get; init; }

    /// <summary>
    /// Snapshot of the compilation inputs at release time. Stored on the
    /// release so a future replay can verify the inputs match. Same hash
    /// used to derive the disk path.
    /// </summary>
    public required string Code { get; init; }
    public string? HubConfiguration { get; init; }
    public IReadOnlyList<ContentCollectionConfig>? ContentCollections { get; init; }
    public required string FrameworkVersion { get; init; }
    public required string ContentHash { get; init; }    // 16-char base64

    /// <summary>
    /// Filesystem path of the compiled DLL. Set when the compile activity
    /// terminates with <c>Succeeded</c>; null on failure. Path is
    /// <c>{cacheDir}/{nodeTypePath-sanitized}/{version}/Release.dll</c>.
    /// </summary>
    public string? AssemblyPath { get; init; }
    public string? PdbPath { get; init; }

    // Inherited from ActivityLog:
    //   Status            — Pending → Compiling → Succeeded / Failed
    //   RequestedStatus   — control plane (e.g. set to Cancelled to abort)
    //   Messages          — Roslyn diagnostics during compile
    //   Start, End        — when compile started / finished
    //   ReturnValue       — JsonElement of the AssemblyPath (also set above)
}

Release derives from ActivityLog so the existing Activity Control Plane machinery — observable progress via workspace.GetMeshNodeStream(releasePath), cancellation via RequestedStatus = Cancelled, and real-time message streaming — comes for free.

Lifecycle

1. Create-release request

🚨 Superseded — do not write a request type for this. The shipped trigger is a stream.Update control-plane field: set NodeTypeDefinition.RequestedReleaseAt (with RequestedReleaseForce for "bypass the sources-unchanged short-circuit") through workspace.GetMeshNodeStream(nodeTypePath).Update(...), or call hub.RequestNodeTypeRelease(...). The per-NodeType hub's watcher reacts and dispatches the compile, idempotently, off the last-handled stamp.

// ❌ HISTORICAL shape. The legacy `CreateNodeTypeReleaseRequest`/`Response` pair still
//    exists as plumbing — never post one from new code.
public sealed record CreateReleaseRequest(string NodeTypePath, string? Version, MarkdownContent? Notes)
    : IRequest<CreateReleaseResponse>;

public sealed record CreateReleaseResponse(string ReleasePath, string? Error = null)
{
    public bool Success => string.IsNullOrEmpty(Error);
}

The handler at the mesh hub:

1. Reads the current NodeTypeDefinition and Source/ content of NodeTypePath.
2. Computes ContentHash over those inputs.
3. Creates a Release MeshNode at {NodeTypePath}/Release/{Version ?? autostamp()} with Status = Compiling.
4. Posts back CreateReleaseResponse with the release path immediately (just-start, matching the ScriptDispatch.StartScript pattern).
5. The Release's hub fires the compile asynchronously.

2. Compile activity (per release)

Each Release MeshNode's hub watches its own MeshNodeReference stream via hub.WatchControlPlane(...). When RequestedStatus = Compiling (set on create), it fires the Roslyn compile in the background:

🚨 Do not copy this block. Observable.FromAsync is forbidden outside IoPool (it runs the prologue on the subscribing thread — i.e. the hub's action block — with no concurrency bound); the Roslyn compile is a blocking leaf and belongs on pool.InvokeBlocking(...). And UpdateMeshNode here is never subscribed, so the write would silently not happen — the shipped code composes GetMeshNodeStream(path).Update(...).Subscribe(_ => { }, ex => …).

hub.RegisterForDisposal(hub.WatchControlPlane(requested =>
{
    if (requested != ActivityStatus.Compiling) return;
    Observable.FromAsync(ct => CompileReleaseAsync(hub, ct))   // ❌ FORBIDDEN — use IIoPool
        .Subscribe(
            assemblyPath =>
                hub.GetWorkspace().UpdateMeshNode(curr =>
                    curr.Content is Release r
                        ? curr with { Content = r with {
                            Status = ActivityStatus.Succeeded,
                            AssemblyPath = assemblyPath } }
                        : curr),
            ex =>
                hub.GetWorkspace().UpdateMeshNode(curr =>
                    curr.Content is Release r
                        ? curr with { Content = r with {
                            Status = ActivityStatus.Failed,
                            Messages = r.Messages.Add(new LogMessage(ex.Message, LogLevel.Error)) } }
                        : curr));
}));

Roslyn diagnostics flow into Release.Messages (inherited from ActivityLog) during the compile, using the same per-Activity logger pattern as the kernel.

3. Resolution: which release is active?

🚨 Superseded — and the proposal below is the shape that was rejected. Resolving the active release with a Query is eventually consistent (stale right after a compile) and costs a round-trip, and the .Wait() is a blocking sync-over-async read that deadlocks on a hub. What shipped instead: the answer is a field on the NodeType node — NodeTypeDefinition.LatestReleasePath, written by the compile watcher after a successful compile and preserved across failed compiles, so consumers keep loading the last-known-good release. Read it off GetMeshNodeStream(nodeTypePath); never query for it. RequestedReleasePath, when set, pins activation to a specific historical release instead (production pinning / rollback).

The proposal was for NodeTypeService.GetCachedConfiguration(nodeTypePath) to become a stream-backed read keyed off a release feed:

// ❌ HISTORICAL — do not copy. `.Wait()` blocks the caller; the Query below is
//    eventually consistent. Read NodeTypeDefinition.LatestReleasePath instead.
public NodeTypeConfiguration? GetCachedConfiguration(string nodeTypePath) =>
    GetActiveReleaseStream(nodeTypePath)
        .Take(1)
        .Select(release => release?.AssemblyPath is { } path ? LoadConfig(path) : null)
        .Wait(); // sync read for the cached path; observable variant for hot paths

private IObservable<Release?> GetActiveReleaseStream(string nodeTypePath) =>
    meshService.Query<MeshNode>(
            MeshQueryRequest.FromQuery($"namespace:{nodeTypePath}/Release nodeType:Release"))
        .Select(change => change.Items
            .Select(n => n.Content as Release)
            .Where(r => r is { Status: ActivityStatus.Succeeded, AssemblyPath: not null })
            .OrderByDescending(r => r!.Start)
            .FirstOrDefault());

The active release is always the latest Succeeded one. That property did survive: failed compiles never become active, and users keep running on the previous release until they ship a fix in a new one.

4. ALC management

CompilationCacheService becomes Release-keyed:

• GetOrCreateLoadContextForRelease(release) keys _loadContexts by release.Path (already does this, but loads from a version-stable folder rather than a hash-stable one).
• DLL path: {cacheDir}/{nodeTypePath-sanitized}/{version}/Release.dll. The path is stable for the same (NodeTypePath, Version) pair. Re-running a compile against an existing version overwrites in place but never deletes a different version's DLL.
• Switching active release: when a new Release becomes the latest Succeeded, the previous release's ALC stays in _loadContexts until explicitly unloaded. NodeTypeService calls cacheService.UnloadContext(prevRelease.Path) when the active release advances. The DLL on disk is kept — only the ALC is disposed. New per-node hub activations bind to the new release's ALC; existing per-node hubs stay on the previous ALC until they are recycled.
• InvalidateCache(nodeTypePath) is deleted. Releases are immutable and durable — there is nothing to invalidate. The replacement is "create a new release," which the user does explicitly.

5. UI surfaces

On submit, the form posts CreateReleaseRequest, navigates to the new Release's detail view, and the user watches the compile happen in real time via the Activity Control Plane subscription.

Migration plan

The redesign is invasive but strictly additive: Release MeshNodes are introduced alongside the existing cache, readers are flipped one consumer at a time, and the old implicit path is deleted last.

CodeEditRecompileTest was to un-skip at phase 3, rewritten as: create V1 release → read V1 → create V2 release → read V2 marker — exercising the explicit-release path with no InvalidateCache call and no file-delete race. It is un-skipped and running today.

Open questions for review

1. Version naming default. When the user omits Version, the suggested auto-stamp format is {yyyyMMddHHmmss}-{8charContentHash} — sortable and unique.

2. Garbage collection. Releases accumulate indefinitely. A TTL, "keep last N," or explicit-delete policy is probably needed, but deferred as a follow-up.

3. Cross-instance compilation. Releases are MeshNodes, so they replicate across instances. Compiled DLLs on disk are per-instance. A Release that succeeded on instance A still needs to compile on instance B. This should be idempotent: same inputs → same content hash → same release ID → same target path → already-compiled is a no-op.

4. Failed releases — keep or drop? Proposal: keep (Status=Failed) and surface them in history. The Notes and Activity messages explain why the compile failed, which is useful for triage.

5. Concurrent create-release. Two users creating a release for the same NodeType simultaneously will get different auto-stamped versions (timestamp differs). Both compile independently; the latest Succeeded wins active status. This is probably fine.

2026-09-02 — the post-condition at the settle (#781)

Symptom. Publish/Deck had compiled the current source and instances kept binding the previous day's assembly. The node's own state said it, and said it quietly:

lastCompileSucceededAt        2026-08-27T21:53:01.850Z
lastCompiledVersion           575
latestAssemblyPath            Publish_Deck/v575-s8929555-aadb349047af.dll

requestedReleaseAt            2026-08-27T21:52:59.898Z
lastReleaseRequestHandledAt   2026-08-27T21:52:59.898Z   ← consumed
latestReleasePath             Publish/Deck/Release/20260826065548-neI3XM25   ← the PREVIOUS DAY

compilationStatus: Ok, compiledSources identical to currentSourceVersions, an assembly built, a release path present. Healthy from every angle. Only comparing lastCompiledVersion against the release reveals it, and nothing was comparing them.

What was NOT the cause

The obvious reading — "a release request fired while a compile was in flight" — is wrong, and chasing it would have produced a second gate that already exists. InstallReleaseRequestWatcher has gated on a SETTLED status (not Pending and not Compiling) since long before the incident:

&& def.CompilationStatus is not CompilationStatus.Pending
                          and not CompilationStatus.Compiling

Checked out at 2026-08-01, 2026-08-20 and 2026-08-27T21:00 — present in all three, all before the 21:52:59 request. So at the moment the request was handled, nothing was in flight.

What it actually is: an ordering the request side cannot see

Two ways to reach the same state, and the fix must not care which one happened:

1. The request cut the build that was current at 21:52:59 — correctly, by its own contract — and the compile that finished at 21:53:01 started after it. Nothing revisits the result.
2. The release create simply did not land. TryCreateReleaseNode is best-effort by design (compile correctness must not depend on a MeshNode create), so an expired bound, a fault, or a refusal all emit null, logged at Warning and swallowed. The create runs under the REQUESTER's identity for attribution, so a partition the requester may not create in refuses it — and Access denied: user 'rbuergi' lacks Update permission on 'Publish/Deck' makes that the leading candidate for that night.

In both, ApplyCompileSuccess stamps releasePath ?? def.LatestReleasePath — the previous build's release — while LastReleaseRequestHandledAt was stamped on the DISPATCH commit and the trigger is therefore already spent. Nothing retries, and asking again cannot repair it: a second request is absorbed by the build already in hand (#1707 slice 3).

The fix: a post-condition, checked where the compile SETTLES

latestReleasePath must never be older than lastCompiledVersion while requestedReleaseAt has been consumed.

ReleasePostCondition (MeshWeaver.Compiler.Pipeline) evaluates it in the terminal chain of RunCompile, after the release create has answered and before the terminal stamp — the one moment at which every fact is in hand. The verdict is a pure function of (the definition at dispatch, the compile result, the release cut on this settle), so it is unit-testable with no mesh.

It fires only on EVIDENCE, and stays silent wherever the answer is inconclusive:

Why RE-CUT rather than only report

The remedy mints the missing release from the bytes this compile just produced — no recompile — under SYSTEM, with the requester cleared so it does not re-attempt the attribution that was refused.

• Reporting alone leaves the mesh in the incident's state: unrepairable from the outside, because the trigger is spent and a fresh request is absorbed by the build in hand. Only force escaped, and the person who needed it could not write the node.
• Un-consuming the trigger is worse. A release create that keeps failing would re-dispatch a compile on every settle — a reconcile fed by its own writes, unbounded.
• System is the credential that produced the bytes. The compile already runs as System precisely so it succeeds on a partition the caller cannot write; the requester passed the Compile gate at the entry point. Cutting the artefact that compile owed them widens nothing.

…and it stays LOUD

Silence is what made this invisible for a day, so the violation is an ERROR naming the type, the stale path and the build that moved — whether or not the re-cut succeeds — and the outcome is written to the compile _Activity, the official diagnosis surface. A re-cut that also fails says so explicitly: "the node advertises a build no release names".

Pinned by: ReleasePostConditionTest (10 cases, MeshWeaver.Compiler.Pipeline.Test) for the verdict, and — on a real mesh, in MeshWeaver.Plugins — ReleasePostConditionAtSettleTest, which reproduces the incident end to end: an Editor who holds Compile asks for a release on a type whose Release subtree denies Create, the compile succeeds as System, the attributed create is refused and swallowed, and the settle must still leave a release naming lastCompiledVersion.