"Mailbox not connected" is sometimes a lie

Issue Systemorph/MeshWeaver.Plugins#1398, measured on memex.meshweaver.cloud on 2026-09-06. An Executive Assistant round ran ~20 parallel SearchMail calls successfully, went silent, and then re-executed the same message from scratch — at which point every mail tool answered

I don't have access to your mailbox and calendar yet. Please connect them here: /auth/ea/connect

while Auth/_EaCredential/rbuergi was present, carried a refreshTokenEncrypted, held the right scopes, and had been refreshed two and a half minutes earlier. A new thread on the same portal, with no user action in between, worked again five minutes later.

This page records the reading of that path. The credential-read half is fixed and shipped — core MeshWeaver#3446 landed the seam and this repo's #1416 landed the consumer, both on 2026-09-06. A read that does not complete is now its own answer and no longer offers a consent link.

Where the architecture lives now. The durable description of the seam — the three connection states, why every entry point returns IObservable<T>, and the three sites that collapsed into one null — is core's /Doc/Architecture/ExecutiveAssistantCredentialReads. Read that for how it works. What is kept here is the incident reading: the measurement, what was excluded and how, and the parts of #1398 nobody has explained yet.

Still open. Plugins#1398 — what re-drove the round, and why the thread node never left Executing. Neither is explained here and neither is touched by the fix above.

The path a mail tool took, per call

(As it stood on 2026-09-06. The seam is now IObservable<EaGraphAccess> IEaGraphAuth.GetAccessToken. The …Async names below are the pre-#3446 ones; they survive as transitional default-implementation forwarders on the interface — deliberately not [Obsolete], so this repo's -warnaserror build does not break on them — and core's HubReachableAsyncGuard carries them as debt until the last caller moves.)

Every tool in ExecutiveAssistantPlugin opened with ClientAsync(), which called IEaGraphAuth.GetAccessTokenAsync. That lands in core, memex/Memex.Portal.Shared/Authentication/EaGraphAuth.cs, and did all of this, every time:

1. LoadAsync — read the credential node Auth/_EaCredential/{user};
2. PostTokenAsync — a fresh grant_type=refresh_token redemption against Entra;
3. on rotation (Entra returns a new refresh token essentially every time) StoreAsync — a write of that same node, itself preceded by another LoadAsync.

There is no token cache anywhere on this path. Nothing is remembered between calls — and that is still true after the fix, deliberately: see What landed below.

The lie (the code as it was)

// LoadAsync
node = await ws.GetMeshNodeStream(PathFor(userObjectId))
    .Take(1).Timeout(TimeSpan.FromSeconds(10)).FirstAsync() …
…
catch (Exception ex)
{
    logger?.LogWarning(ex, "EaGraphAuth: load failed for {User}", userObjectId);
    return (null, null);          // a FAULT, shaped exactly like an ANSWER
}

// GetAccessTokenAsync
var (_, cred) = await LoadAsync(userObjectId, ct);
var refresh = protector.Unprotect(cred?.RefreshTokenEncrypted);
if (string.IsNullOrEmpty(refresh)) return null;   // → "not connected"

A 10-second read timeout — or any other read fault — is caught, logged at Warning, and returned as (null, null), which is byte-for-byte indistinguishable from "this user has never connected". The caller has no way to tell the two apart, so a transient becomes a confident, wrong statement about the user's account, and the round's twenty successful searches are discarded to make it.

This is the fail-closed fallback that forges a correct-looking bug: a guard that fails safe must still fail legibly. Refusing to act on an unreadable credential is right; claiming the credential is absent is not. That is the defect #3446 removed.

A second mechanism — which the author later RETRACTED as required

The faulting round issued ~20 parallel SearchMail calls. By the shape above that is ~20 concurrent Entra redemptions and up to ~20 writes of ONE mesh node — whose owning hub is single-threaded — with ~20 more reads queued behind them. The credential write observed at 12:29:49.375 (version 101) is one of those, landing 372 ms after the round went silent at 12:29:49.003.

🚨 This page originally led with that reading, and leading with it pointed the fix at the wrong place. On MeshWeaver#3433 the issue author retracted it:

The root cause is async/await/Task<T> in hub-reachable code. Contention is not needed to explain the failure at all. […] My "~20 parallel calls contend on one single-threaded node hub" framing was a plausible-sounding second mechanism that is not required to produce the observed failure — and stating it first pointed the fix at a token cache when the fix is to stop being async.

The contention above is real arithmetic and is kept for that reason. It is not the explanation, and no token cache was added.

What was excluded, and how

The one measurement that settles it

Both surviving candidates log at Warning, from the same class, in the same window:

• EaGraphAuth: load failed for {User} (with a TimeoutException) — read starvation, or
• EaGraphAuth: token endpoint returned {Status} — the Entra path.

{app="memex-portal"} |= "EaGraphAuth", 2026-09-06 12:29:40Z–12:33:00Z. Whoever reproduces this should capture it before the window rolls — it names which of the two, and it costs one query.

🚨 It has NOT been run, and one nearby zero must not be quoted as if it had. No Loki read was made for this page. What was run is a mesh query against the Hosting/LogEntry mirror — namespace:Hosting scope:descendants nodeType:LogEntry EaGraphAuth → count: 0 — and that is not evidence the log line is absent: that mirror is populated by the module's ingest-logs script over a bounded window rather than by a live tail, so a zero there carries no denominator and no statement about what the window covered. When the Loki query is finally run, use an explicit start/end plus an oldest-line control: since= is capped in practice at about an hour and has returned a false zero for a window it silently never reached.

🚨 Can a re-executed round duplicate a user-visible effect?

Asked because #1398's other half is a round that re-executes from scratch on the same message.

• A second mail: no, under the default. Email:AgentSend defaults to DraftOnly, and in that mode SendMail/ReplyToMail are not in CreateTools() at all — the model never receives them, so no re-execution can reach a live send. On a deployment that explicitly sets Email:AgentSend=Send that protection is gone: there is no per-message confirmation and no idempotency key, so a re-executed round could send the same mail twice.
• A second draft: yes. DraftMail is a plain POST /me/messages and DraftReply a plain createReply — no idempotency key, no dedupe, no "have I already drafted this?" check. A round that composed a draft and then re-executed leaves the reviewer two near-identical drafts: exactly the confusion GetDraft/UpdateDraft (#1049) exist to remove, arriving through a door those tools cannot cover, because a re-executed round starts with an empty tool-call list and knows about nothing.

The 2026-09-06 round happened to be read-only, so nothing was duplicated. The same fault on a drafting round would have duplicated.

What landed

One defect, one fix, and it is not the one this page led with.

Stop reporting a fault as an answer. The read now distinguishes "there is no credential" from "I could not read the credential", and the tool says which. IEaGraphAuth answers IObservable<EaGraphAccess> carrying a three-state EaConnection — Connected, NotConnected, Undetermined — and ExecutiveAssistantPlugin.ClientAsync branches on it:

• Connected with a token → the Graph client;
• NotConnected → the one truthful consent offer, the /auth/ea/connect link;
• Undetermined, a source that completed with no value, or the contract violation of Connected-with-no-token → "I could not determine whether your mailbox and calendar are connected just now", carrying the diagnostic and no consent link.

Sending someone to re-consent is harmless in itself; telling them their mailbox is not connected when it is — and discarding the round's work to say it — is not.

All three collapsing sites went with it, not just the outer catch: the two catches and the _ => null switch arm, which was never an error handler at all but a hand-rolled shape test, now replaced by the house accessor. Core's page carries that table.

The seam is reactive end to end (IObservable<T>, IIoPool for the Entra POST, never Observable.FromAsync), and the one bridge left is the consent controller's, through ObserveCompletion — never .ToTask(). ExecutiveAssistantPlugin also reports a credential fault that arrives after its wait has settled, rather than orphaning it.

🚨 No token cache was added, and none is wanted for this. The "acquire once per round instead of once per tool call" item this page used to lead with was the retracted framing; per the author's own comment the fix is to stop being async, and contention is not needed to explain the failure. If a round-scoped acquisition is ever wanted it is a performance change on its own merits, argued separately — an instance cache keyed by the acting identity, misses sharing one in-flight acquisition through IIoPool (pool.Run in a promise slot — never a SemaphoreSlim, never a bare ConcurrentDictionary<key, IObservable<T>>), with no entry outliving the token's exp.

Coverage. ExecutiveAssistantConnectionStateTests (#1416) asserts the three states through ExecutiveAssistantPlugin with a stub IEaGraphAuth: collapse Undetermined into NotConnected and the tests go red. That is one of the module's two consumers of the seam — GraphEmailSender's probe and its SendAsUserAsync path are still uncovered, tracked on #1428.

Still unexplained (do not read this page as a full diagnosis of #1398)

• What re-executes the round. The visible stall guards — StreamStallGuardChatClient and the matching branch in ThreadExecution — abort with "stopped responding mid-stream; resubmit to retry", i.e. they ask a human. Something else re-drove the message. Worth instrumenting: a per-thread hub re-activation re-picking a message left pending, and any fault path that re-enters the round instead of failing it.
• Why the thread node never leaves Executing. The message reached Completed; the thread stayed Executing with the pre-fault lastActivityAt, indefinitely — so a later submit_message would queue behind a round that never ends. Two later threads on the same portal flipped to Idle normally, so this is specific to the re-executed round, not a broken projection in general.