CI artifact storage โ the bytes, the bill, and where they belong
Compute left GitHub; storage has not. Every private-repo job now runs on our ARC scale sets
(aks-silos / aks-silos-dind). Runner routing and billing limits are separate controls: a
positive Actions budget does not enforce self-hosted-only execution. What is still billed is
Actions storage, and it is the whole of the remaining Actions line.
Maintainer, 2026-09-17: "we still incur cost for github actions โฆ please see that it goes to 0" ยท "disable for any private repo" ยท "and when free capacity gone => defer to our infra".
The object-store changes described below do not by themselves remove every GitHub artifact handoff. PR artifacts on our infrastructure documents the named-artifact adapter, partial-rerun contract, shared-mount proof and remaining rollout gates. Do not treat a declared store variable, an admission check or an extra durable copy as proof of zero uploads.
๐จ READ THE BUDGET, NEVER REMEMBER IT โ the amount moves within the day, and the whole fleet's CI
hangs off it. GET /organizations/Systemorph/settings/billing/budgets (the classic
/orgs/{org}/settings/billing/actions endpoint answers 410 Moved) returns each budget's
budget_amount and prevent_further_usage. Measured at $0 on 2026-09-17 morning and at
$6,000 the same day at 16:18Z โ and the difference between those two numbers is whether any
private repo in the fleet can upload an artifact at all. At $0 the first upload of every
MeshWeaver.Plugins run failed from ~08:29Z with "Artifact storage quota has been hit", every gate
behind it reddened for want of its input, and nothing published
(MeshWeaver.Plugins#2016). A
storage-quota refusal is therefore a budget event, not a capacity event, and deleting artifacts
does not lift it until GitHub recalculates (every 6โ12 h).
The measurement (2026-09-17, REST only)
Org net Actions storage, 1โ17 September: $66.24, rising โ $8.70 on 09-16 alone, i.e. a
~$260/month run rate. The public repo is not in it: Systemorph/MeshWeaver averaged 4,353 GB
and was billed $0, because a public repository's Actions storage is free. Only the private
repos pay.
| repository | live artifacts | month-to-date |
|---|---|---|
| MeshWeaver.Plugins | 381.5 GB | $52.98 |
| MeshWeaver.SocialMedia | 12.0 GB | $1.16 |
| MeshWeaver.Reinsurance | 10.2 GB | $4.53 |
| MeshWeaver.Education | 8.4 GB | $4.59 |
| MeshWeaver.Manufacturing | 6.1 GB | $0.47 |
| MeshWeaver.Crm | 6.0 GB | $2.51 |
| MeshWeaver (public) | 4,353 GB avg | $0 |
๐จ Actions CACHE is not billed. Plugins holds 11.87 GB of cache and SocialMedia 11.38 GB against a 10 GB per-repo allowance, but the allowance is enforced by LRU eviction, not by an invoice โ SocialMedia's 11.38 GB of cache sits beside a 9 GB average storage charge, which is arithmetic only if cache is outside it. Cache is a capacity question (an idle entry pushes a live one out), not a cost one.
๐จ You pay for retention PLUS a deletion lag of 2โ6 days
The artifact inventory and the invoice disagreed by 2.4x, and the explanation is not compression and not logs. An artifact keeps costing money after it expires, until GitHub actually deletes it, and that lag is days.
Integrating every artifact's bytes over 2026-09-16 โ alive from created_at to expires_at โ gives
9,044 GB-hours for Plugins against 21,904 billed. Re-integrating with a deletion lag added to each
expiry, and fitting the lag per repository, reconciles every repo to within 1%:
| repository | best-fit lag | modelled GBh | billed GBh |
|---|---|---|---|
| MeshWeaver.Plugins | 2.8 d | 21,885 | 21,904 |
| MeshWeaver.Reinsurance | 5.0 d | 1,183 | 1,189 |
| MeshWeaver.Crm | 5.8 d | 730 | 729 |
| MeshWeaver.Manufacturing | 4.0 d | 228 | 228 |
| MeshWeaver.Education | 4.8 d | 1,249 | 1,259 |
| MeshWeaver.SocialMedia | 2.2 d | 584 | 573 |
The artifacts API corroborates it directly: 1,070 GB of Plugins artifacts are listed expired: true and have not been deleted, 506 GB of them expired within the last two days.
Three consequences, and they reorder what is worth doing:
- A 1-day artifact is billed for four to seven days. The scratch families โ
platform-refs-*,workspace-build-*,compile-check-refs,portal-hosts-bin-*โ are consumed inside a 40-minute run and declare the shortest retention GitHub allows, and they are still 52% of the fleet's daily storage cost. - Explicit deletion beats shortening
retention-days.DELETE /repos/{o}/{r}/actions/artifacts/{id}is immediate; expiry is followed by days of billed limbo. That is the argument for the fleet cleanup job (Memexactions-cleanup.yml), not just for smaller retention numbers. - Moving the bytes off GitHub avoids both, which is what the rest of this page is about.
๐จ An EXPIRED artifact is deletable, and the sweeper used to skip every one of them
The lag above is not GitHub's to end on its schedule โ it is ours to end with a request. Measured
2026-09-17T16:22Z: DELETE /repos/Systemorph/MeshWeaver.Plugins/actions/artifacts/10455640297,
on an artifact whose expired field already read true, answered 204 No Content, and a GET
of the same id then answered 404. So the 1,070 GB sitting expired: true and undeleted is a
pool a sweep can reclaim.
The size of that pool, read straight off the invoice: MeshWeaver.Plugins was billed 21,904 GB-hours on 2026-09-16 โ an average of 913 GB standing โ against a 381.5 GB live inventory. The 531 GB difference is the lag. It is the majority of what that repository pays for, and it is not live artifacts at all.
Memex's scripts/actions-cleanup.py skipped it, on the premise โ written into its own docstring โ
that "expired artifacts hold no storage". That premise is the exact opposite of the fit above, and
what it cost was structural, not marginal: its live rule selects an artifact that is live AND
older than --artifact-days (2), and a 1-day-retention artifact is expired: true before it is
two days old. So the four families that are only a handoff between the jobs of one run โ
platform-refs-*, workspace-build-*, compile-check-refs, module-pack-tool-* โ could never be
selected at all, by construction, and those are the families the table below prices at 44% of the
fleet's daily cost. The two rules, dry-run over the same repositories on 2026-09-17:
| repository | the live rule alone | live + expired within 7 days |
|---|---|---|
| MeshWeaver.Manufacturing (40 pages) | 108 artifacts / 1.71 GB | 616 / 9.22 GB |
| MeshWeaver.Plugins (its newest 1.2 days) | 0 | 194 / 7.97 GiB |
๐จ Expiry is the EARLIEST safe moment to delete a handoff โ and deleting one before it is NOT
safe. A download of an expired artifact already fails, so deleting it cannot break a re-run that
would otherwise have worked. Deleting a live handoff at the end of its own run โ the tempting
"it has been consumed, drop it" โ would: POST โฆ/rerun-failed-jobs re-runs the failed job but
not the succeeded producer behind it, so the consumer comes back looking for bytes nobody
re-uploaded. That is a live shape, not a hypothetical: see the misdirection below.
๐จ And a budgeted sweep must spend its budget on BYTES. The invoice is GB-hours; the sweeper's
budget is REQUESTS (--max-deletes, because GitHub wants โฅ1 s between mutative requests and the
caller is capped at 45 minutes). Measured over MeshWeaver.Plugins' newest 120 artifact pages โ
10,800 artifacts, 2026-09-16T11:37Z โ 2026-09-17T16:16Z, 163.5 GiB uploaded in 29 hours โ 595
rows (5.5%) carry 102 GiB (62%). Deleting in page order spends the budget on 300-byte receipts, so
the sweeper orders candidates by size, descending.
What one repository uploads in a day, by family
The live-inventory table further down is a stock; this is the flow that refills it, and the two answer different questions. MeshWeaver.Plugins, the same 29-hour window:
| family | GiB | rows | retention | read by |
|---|---|---|---|---|
platform-refs-catalog-* |
34.25 | 164 | 1 d | this run's pack legs, only when /opt/platform misses |
workspace-build-catalog-* |
33.81 | 143 | 1 d | this run's pack legs |
compile-check-refs |
31.11 | 130 | 1 d | this run's compile-check units and lanes |
bake-<sha>[-shard-*] |
11.58 | 542 | 3 d | the fold, in this run โ nothing in this repo |
portal-hosts-bin-* |
6.69 | 45 | 1 d | this run's portal-hosts-test shards |
module-pack-tool-catalog-* |
3.07 | 158 | 1 d | this run's pack legs |
module-bundle-* (60 modules) |
~35 | ~5,700 | 7 d | this run's gates and a later run's ledger |
๐จ A quota refusal surfaces one hop downstream, wearing no quota in its message
When the org's Actions budget refused uploads on 2026-09-17 (08:29Z onward), the producer job
failed at Run actions/upload-artifact@v7 with GitHub's own sentence โ and every consumer behind it
failed with:
##[error]Unable to download artifact(s): Artifact not found for name: portal-hosts-bin-network-133-โฆ
Please ensure that your artifact is not expired and the artifact was uploaded using a compatible
version of toolkit/upload-artifact.
Sixteen legs, one run, and not one of those messages names a quota, a budget or the producing job.
The author of a documentation-link PR reads "artifact not found" and goes looking in their own
diff. The steward's verdict on the producer was worse โ "step 'Run actions/upload-artifact@v7'
failed and NO named signature matched โ this is (or may be) a real red". MeshWeaver.Plugins#2017
adds a third class for it (INFRA: a named condition that is provably not the change under test
and that no retry can clear, reported with its cause and never retried), which is the right shape
and the one to copy into core's steward rather than adding the quota to a list that means retry.
And the declared retention is capped at 7 days anyway
Measured off expires_at: publication-inputs declares retention-days: 30 and is created with a
7-day expiry; so are teardown-stragglers (14) and test-evidence (14). The repositories'
"Artifact and log retention" setting is 7 days and silently clamps four declarations in the fleet โ
gate-coredumps (15) among them, so the SIGSEGV evidence everyone reaches for lives a week, not a
fortnight, whatever the YAML says.
What each family actually costs, per day, fleet-wide
With the lag included, the model totals $8.78/day against $8.70 billed:
| family | $/day | share |
|---|---|---|
module-bundle-* |
$2.40 | 27.3% |
bake-* |
$1.51 | 17.2% |
workspace-build-* |
$1.38 | 15.7% |
platform-refs-* |
$1.28 | 14.6% |
compile-check-refs |
$1.10 | 12.5% |
portal-hosts-bin-* |
$0.78 | 8.8% |
e2e-* (Education) |
$0.19 | 2.2% |
module-pack-tool-* |
$0.13 | 1.4% |
| everything else | $0.03 | 0.3% |
What the 381.5 GB in Plugins actually is
| bytes | artifact family | retention | who reads it |
|---|---|---|---|
| 247.1 GB | module-bundle-<module> |
7 d | both: this run's gate / compile-check / publish-bake, and a LATER run through the build ledger |
| 36.7 GB | bake-<sha>[-shard-*] |
3 d | the fold, in this run; bake-<sha> by MeshWeaver.Education's e2e jobs. Opt-out per caller (upload-bake); 3 d is a floor, not a habit โ see below |
| 28.2 GB | platform-refs-<lane> |
1 d | this run's pack legs, only when the /opt/platform mount misses |
| 28.2 GB | workspace-build-<lane> |
1 d | this run's pack legs |
| 24.4 GB | compile-check-refs |
1 d | this run's compile-check units and lanes |
| 13.8 GB | portal-hosts-bin-* |
1 d | this run's portal-hosts-test shards |
| 2.5 GB | module-pack-tool-<lane> |
1 d | this run's pack legs |
| 0.6 GB | gate logs, receipts, verdicts, change-set | 3โ30 d | humans and the callers' ratchets |
Two facts decide the design:
- Only ONE family is cross-run.
module-bundle-*is downloaded by another run โ by the build ledger's reuse leg (gh run download "$ART_RUN" -n "$ART_NAME"), bynode-repo-publication-reuse.py, and by core'ssatellite-compat-imagebaseline. Everything else is a handoff between the jobs of one run and is only kept for 24 hours because 1 day is GitHub's floor, not because anyone reads it that long. - Most of the 247 GB are DUPLICATES. A run that the ledger tells to REUSE a bundle downloads those bytes and then re-uploads them under the same name in its own run, because its own consumers read the artifact from this run. 754 live copies of one module's bundle were measured at once, at 8 MB each.
What became OPT-OUT โ and why it is not "removed outright"
node-repo-gate.yml uploads the gate's bake twice โ per shard, then folded โ "for publish-bake to
reuse (bake-run-id)". That reuse was never built: bake-run-id exists in no workflow, script
or input anywhere in the fleet, and node-repo-publish-bake.yml uploads no artifact at all and
bakes its own mount. Two copies of identical bytes (every shard bakes the whole mount, so the fold
is a union of duplicates), held three days โ 66.8 GB across the fleet, and ~100% of the
Reinsurance, Manufacturing, Crm and Education bills.
๐จ But "read by nothing" was WRONG, and the correction is the lesson. #4578 deleted both uploads
on a fleet-wide search that found no consumer; the search read local checkouts, and
MeshWeaver.Education's was eight commits stale. Its main downloads bake-<sha> twice โ in
e2e-install and in the four-shard e2e-mesh, neither with continue-on-error, both feeding the
blocking mesh-gate โ and it calls the lane at @main, so the removal would have reddened its next
run on an artifact that had simply stopped being produced. #4585 restored the uploads behind an
upload-bake input defaulting TRUE; Reinsurance, SocialMedia, Crm and Manufacturing pass
false (verified 2026-09-17 against each repo's REMOTE default-branch workflows), Plugins' opt-out
is MeshWeaver.Plugins#2024, and Education keeps it. That holds ~64.9 GB of the 66.8 and costs
Education nothing.
๐จ Never conclude "nothing reads this" from a working tree.
git rev-parse HEADagainstgh api repos/<o>/<r>/commits/main --jq .shafirst, or read the file through the API. An absent consumer and a stale checkout are byte-identical from here.
๐จ An upload whose reader does not exist is not a contract, it is a bill. If a cross-run bake reuse is ever wanted, it lands with its consumer.
๐จ A same-run handoff's retention floor is NOT 1 day โ it is the QUEUE, twice
The per-shard bake looks like the ideal candidate for GitHub's 1-day minimum: bake-<sha>-shard-<n>
has exactly one consumer, Collect every shard's bake, and nothing outside node-repo-gate.yml
names the sharded form. It was shortened to 1 on 2026-09-17 and reverted the same hour, because
"the consumer runs minutes later" is a description of the happy path, not a bound.
The fold needs: [plan, gate], so it cannot start until the last shard has finished โ and
GitHub's usage limit lets a job sit QUEUED for 24 hours before it is terminated.
timeout-minutes bounds execution and never the wait, so that bound applies twice:
first shard uploads โ last shard queued โค24 h + runs โค45 min โ fold queued โค24 h
โ 48.8 h maximum age at the moment the fold downloads it. 1 day and 2 days are both below
that; 3 is the smallest whole-day value above it, with ~23 h of margin. Shortening it converts a
gate whose shards all passed into Artifact not found in the fold โ a red manufactured by the
retention.
The general rule, and it applies to every retention-days on this page: the floor for a
same-run handoff is (the consumer's worst-case queue wait) + (the producer's worst-case wait and
run), not "how long a human thinks the run takes". The bytes are better recovered by the sweeper,
which deletes at expiry โ the one moment that provably cannot break a consumer that could still
have run.
The seam: ci-artifact-store.py
.github/scripts/ci-artifact-store.py gives a lane one place to say where bytes go:
gha GitHub artifacts โ the caller's own upload/download steps
file:<dir> a writable share the runner pod mounts (no credential)
azblob:<account>/<container>[/<prefix>] Azure Blob via the ambient azure/login (--auth-mode login)
resolve ยท put ยท get ยท probe ยท prune, and a --self-test that exercises all of it against a
fake az and a real directory. A locator is <store spec>/<key>#sha256=<hex>, so a reader that
resolved a different store refuses it rather than fetching the wrong bytes, and get verifies the
digest the record attests before the bytes are used.
The degrade rule โ the one thing to get right
๐จ A caller that declares no store gets gha and behaves exactly as it did before the file
existed. That is not a fallback, it is the default: the public repo, a fork PR and any runner
without our infra have no credential and must keep working. resolve prints the mode and the reason
into the job summary, so "it used GitHub artifacts" is never something you infer from an absence.
๐จ But a store that is DECLARED and cannot be used is RED. Every put/get failure is red,
naming the phase. There is no path where a store is configured, fails, and the lane quietly writes
somewhere else or rebuilds from source โ that is the fault-becomes-fact defect
(#2695), and it is how unchanged โ no
compile turns into sometimes compiles, nobody knows.
๐จ The path is not the identity โ a file: store is per-MOUNT, and mine may not be yours
file:/ci-artifacts names a directory, and the two runner pools mount two different Azure Files
shares there. MW_ARTIFACT_STORE was set to file:/ci-artifacts on 2026-09-18 and the shared
module-pack lane stopped completing for every repository that calls it
(#4761). The producer wrote a 1,459,865,600-byte
workspace-build.tar and succeeded; eight consumers asked for that exact path 13 seconds
later and got "is not there โ the record names an object this store does not hold". Both sides
printed byte-identical ARTIFACT_STORE and STORE_RUN_PREFIX, the same run, the same attempt.
The cluster manifests say why, in the header of the file that declares them โ "ONE Azure Files share per runner namespace":
| runner scale set | namespace | ci-artifacts PVC |
|
|---|---|---|---|
producer (prepare, build-workspace) |
aks-silos-dind |
arc-runners-dind |
its own dynamically-provisioned share |
consumers (pack) |
aks-silos |
arc-runners |
a different dynamically-provisioned share |
Two PersistentVolumeClaims with the same name in two namespaces are two different claims, and
with storageClassName: azurefile-csi and no volumeName each one provisions its own share. So the
write genuinely succeeded and the read genuinely found nothing: the file: backend never had a
successful cross-pool precedent at all. (The green runs cited as proof it worked had
ARTIFACT_STORE: gha โ a control on the other side of the variable.)
The fleet already owns the cure one volume over. ci-platform reaches both namespaces as two
static PersistentVolumes carrying ONE volumeHandle, i.e. one share addressed twice; the
node-repo-module-pack.yml select โ prepare edge is cross-pool by construction
(MW_RUNNER โ MW_RUNNER_DOCKER), so nothing in that lane can hand bytes over until ci-artifacts
is wired the same way or the variable goes back to gha. That half is a cluster change and is
tracked as Systemorph/Memex#420, which states both routes: one share behind two static PVs, or
the move to azblob: this page already recommends.
The stopgap that is in place, and what lifts it (measured 2026-09-20)
MW_ARTIFACT_STORE=gha is set as a REPOSITORY variable on eight repositories โ MeshWeaver,
.Plugins, .Education, .Reinsurance, .SocialMedia, .Manufacturing, .Crm, .Feedback โ
since 2026-09-19T05:35Z, on the maintainer's instruction, after ~9 h in which every
MeshWeaver.Plugins run was red and 20 pull requests could not go green. A repository variable
overrides the organisation one, so each is back on the GitHub-artifact path. It was verified on both
sides of the change: Plugins run 35418370866 attempt 1 (org value) went red at 93 s at the first
put with the two-share refusal, attempt 2 (repo override) completed; and the control still holds โ
run 35534037490 (2026-09-20T19:58Z) ran the same topology, producer aks-silos-dind โ consumers
aks-silos, on ARTIFACT_STORE: gha, every Module bundle green.
Three facts about it that are not in the variable's value:
- The organisation variable's reach is two repositories, not eight. It is untouched โ
file:/ci-artifacts, created 2026-09-18T21:04:23Z, never modified โ and it isvisibility: selected, with the selectionMeshWeaver.PluginsandMeshWeaver.SocialMedia(measured 2026-09-20 over REST). The other six overrides therefore override nothing today; they are there so that widening the selection later cannot re-expose a repository without a change in that repository, and the public repository's is correct for a second reason โ it must never be pushed onto private infrastructure (the script's header says so). - ๐จ So deleting an override does two different things, and only one of them is a rollback. On
MeshWeaver.PluginsorMeshWeaver.SocialMediathe organisation value applies again and the repository is back on the two-share path, RED at the run's firstputโ not silently broken, but not working either. On the other six there is then no variable at all, andresolvemaps""/gha/noneto the same branch ("no object store is declared for this run; artifacts stay on GitHub"), so nothing changes today. Read the selection before reading an override as protection. - Nothing will go red when the stopgap outlives its cause. Once
ci-artifactsis one share the organisation value is correct again, and the eight overrides will keep every repository onghawith nothing to warn โ the object store simply never gets used by exactly the repositories whose storage bill it exists to remove. A stopgap has to be removed by the same act that closes the cause.
So the lift is three steps, in order, and the third is not optional:
Systemorph/Memex#420 closes. It states two routes, and step 1 is a different act on each โ because only one of them has a mount to compare at all:
- A โ one share, the
ci-platformshape. Two staticPersistentVolumes over onevolumeHandle, oneclaimRefper runner namespace; a PVC'svolumeNameis immutable, so the claim that moves is deleted and recreated. The prerequisite is thenresolveprinting the samestore-idon both pools. Mind the shape:mount_identityemits<fstype>:<mount source>:<bind root>, so a whole share mounted at/ci-artifactsreadscifs://<account>.file.core.windows.net/<share>:โ today the share segment differs,pvc-ebde47ec-โฆonaks-silos-dindagainstpvc-42c3b45b-โฆonaks-silos. - B โ
azblob:. There is no mount identity to unify:Store.store_id()returns the spec and onlyFileStoreoverrides it, because an account plus a container name the same bytes from every runner, soazblob:<account>/<container>is identical on both pools by construction and can never discriminate between them. Step 1 here is provisioning plus a variable move โ the container,Storage Blob Data Contributorfor the CI identity, and thepull_requestfederated subject listed under "Where the bytes should live" below, then the organisation variable set to theazblob:spec rather than a path.
๐จ On neither route is step 1 a handoff proof. One agreed identity says both pools mean the same store; it says nothing about bytes having crossed it. That is step 2, and it is the same on both routes.
- A โ one share, the
One run proves the handoff on the store that was chosen โ a run whose
ARTIFACT_STOREline reads the new spec (file:/ci-artifacts, or theazblob:one) and whoseModule bundleconsumers are green, the producer having run on the other pool. A green run onghaproves nothing about either route; that is the control above, on the other side of the variable.Lift an override only for a verified private pilot. The incident issue is closed, but its closure is not evidence that the mount or the complete graph works. Keep each repository on
ghauntil its cross-pool proof and retention controller pass; leave public callers on their existing GitHub backend. Removing an override is a rollout act, not a documentation cleanup.
Why nobody could see it: put's success line could not be wrong. The byte count and the sha256
both came from the source file and dst was never stat-ed or read back, so the producer was
green by construction and the failure necessarily presented as a consumer problem โ the same
family as a gate that cannot fail on its own input. Both halves are now closed:
putreads the destination back. After the rename it asserts thatdstexists, is a regular file, has the source's length, re-hashes to the digest, appears in its own directory listing, and left no staging file behind. The bytes arefsynced before the rename, because on a network filesystem the server โ and so every other mount โ holds them only once the client has flushed. Four deliberate breakages of the swap (the destination vanishes, is truncated, holds different bytes, or the "rename" was really a copy) are self-test rules, each proved red.- A store has an IDENTITY, read from the kernel.
store_id()returns the mount source from/proc/self/mountinfoโ for cifs//<account>.file.core.windows.net/<share>โ which is identical for two pods on one share and different for two shares.resolveemits it asstore-id, and the lane hands it to everyput/getas--expect-store-id, so a runner standing on a different share is RED at the first store operation of the run, naming the mechanism and both remedies, instead of writing a handoff that reads as an absence eight jobs later. An empty--expect-store-idis red too: a check wired up with nothing to check against is a check that passes on no evidence.
The identity is derived, never minted โ no marker file to lose, nothing to keep in sync, and a symlinked or trailing-slash spelling of the same directory is the same store (a self-test rule, because that negative control is what stops the check reddening a correct run).
๐จ A mount source is only an identity for a SHARED filesystem. overlay is the source of every
container's root filesystem and tmpfs of every tmpfs, so comparing sources alone would answer
"same store" for two pods that share nothing โ the exact false pass the check exists to refuse. If
/ci-artifacts were ever a plain directory on the pod's own root (a volume that never mounted, a
spec that lost its volumeMounts entry), reachable() accepts it, because it exists and is
writable. So an identity outside SHARED_FSTYPES (cifs/smb3/nfs/โฆ) carries this machine's node
name: two pods can then never agree about a node-local directory, one process always agrees with
itself, and the refusal says check volumeMounts rather than unify the shares โ a different fault
with a different remedy.
๐จ Named artifacts must check the producer's identity before looking for a manifest.
ci-run-artifacts.py and the upload-artifact / download-artifact composites use the same
FileStore. The composites require expected-store-id in file: mode and the adapter validates
it before every operation, including an empty pattern or collection download. A legitimate empty
selection still succeeds, but an empty selection on the wrong share is RED, never evidence of a
successful handoff. The resolve-artifact-store composite derives the identity at the first
producer; subsequent jobs and reusable workflow calls carry that identity unchanged. A reusable
workflow requires a caller identity in file mode before resolving its own mount, and compares it
before publishing outputs. Reusable defaults stay gha, regardless of organization variables;
only an explicitly migrated complete graph selects own storage. Tests execute
the actions' actual shell and mutate graph bindings so that dropping the check cannot pass.
This is code-level protection, not proof that the infrastructure is repaired. The Plugins override
was still gha when inspected on 2026-09-22. The shared workflow and caller migration is staged;
the live switch still requires the cross-pool and retention proof below.
The general form, for the third time on this volume class: a publish atomically, then swap
protocol is only atomic if the swap is a real rename on that filesystem, and a shared store is
only shared if both ends are on the same one. The portal learnt the first half as File.Move
copying on Azure Files (#2190 โ
#4547); this is the second half, in CI.
How the module bundle splits in two
node-repo-module-pack.yml takes artifact-store and artifact-store-id inputs. In file: mode:
- the durable copy goes to the store, keyed
modules/<repo>/<module>/<build-key>/<file>โ the ledger's build key is a content address, so identical bytes are written once however many runs want them, andputprobes by sha256 before uploading anything; - the record gains
bundleStore.locatorbesidebundleArtifact, never instead of it; - the named artifact also goes to our store, through the shared upload/download actions. Named module bundles retain 7 days: publication reuse reads them across runs without a ledger key, so the durable ledger copy does not justify shortening that retention;
- the reuse leg fetches from the store when the record names one and this run resolved the same
store and the object is actually there (
module-build-ledger.pyasks the store, never the record's word); otherwise it can use a verified named artifact on that same store. Missing historical bytes permit a rebuild. A corrupt archive, unavailable declared store or wrong physical identity fails RED. None of those cases falls back to GitHub artifact storage.
Same-run handoffs โ named artifacts, keyed by run and attempt
platform-refs-<lane>, workspace-build-<lane> and module-pack-tool-<lane> exist only to cross a
job boundary inside one run. Their earlier GitHub copies contributed to the measured storage bill
above. The shared actions now select exactly one backend: GitHub for an unset store or explicit
gha, our mounted store for explicit file:. This also covers the workflow's other named
artifacts: receipts, gate logs, test output and publication input attestations.
Manifests and digest-verified archives live below named/, separated by repository, run, attempt
and artifact name. The consumer chooses the newest available attempt no later than its own. This
is required for GitHub's re-run failed jobs semantics: a successful producer is not rerun, and
its previous-attempt artifact remains the correct input. A full rerun's newly uploaded artifact
wins. Cross-run publication reads use the source run's attempt, not the consuming run's attempt.
Lane-qualified names prevent two reusable calls in one run overwriting one another.
The platform-refs fetch keeps its fallback semantics exactly: the runner's /opt/platform mount
first, then whichever handoff this run used, and a refusal that now names both.
๐จ One expression, two readers. The retention-days: on the ten upload slots and the
--retention-days the ledger record states were two independent literals until 2026-09-17. They now
read one expression, so a record can never outlive the artifact it names.
Supported backend and retention
The named-artifact adapter currently supports file: only, alongside the compatible gha
backend. The lower-level durable-object helper also has an azblob: implementation, but that is
not a named-artifact backend and cannot be selected for this graph. This migration does not grant
pull requests production bake credentials or add federated identities. An authenticated remote
store would be a separate design, not a fallback when the mounted store fails.
In own-store mode the migrated workflow graph disables actions/cache and setup-node's automatic
package-manager cache as well as GitHub artifact uploads. Existing runner-local caches remain
usable. GitHub still orchestrates these workflows and retains logs/checks; moving artifacts does
not by itself move the build queue or eliminate every GitHub cost.
Lifecycle
The retention controller must understand the named manifest's expiry; it must not apply the old two-day handoff rule to every named artifact:
| prefix | holds | delete after |
|---|---|---|
modules/<repo>/<module>/<key>/ |
the cross-run reuse copy | 14 days |
named/ |
named archives and manifests, including cross-run publication inputs | each manifest's expiry; module bundles 7 days |
runs/ |
legacy handoffs from the earlier workflow implementation | legacy 2-day rule, never applied to named/ |
What the migration still needs (maintainer)
- One backing share, proved across pools. A producer on one runner pool and a consumer on the other must report the same physical identity and read the same digest-verified bytes. Exercise the real filesystem's locking and atomic replacement too; local-directory tests do not prove SMB semantics. Provisioning is tracked in Memex#420.
- Expiry-aware cleanup. Install and verify bounded pruning for named manifests and archives without deleting unexpired cross-run publication inputs or a transfer protected by its lock.
- Land the compatibility foundation before its callers. The shared actions, resolver and
helper CLI contracts must be on core
mainbefore workflows call them at@main; the shared reusable workflows must land before the Plugins caller passes their new inputs. - Enable one private pilot and measure a real PR. Only after the infrastructure proof, lift
that repository's
ghaoverride. Verify the complete graph, a partial rerun and cross-run reuse, with no new GitHub artifact or Actions-cache writes. The queue's own rollout is separate. - Treat historical GitHub cleanup separately. The existing inventory remains billable until retention/deletion completes. Inspect the sweeper's current mode and protection set before any cleanup change; this code migration does not authorize deleting existing retained artifacts.
See OwnPrArtifacts for the staged rollout contract and evidence required before calling the private PR path enabled.
See also
- ArtifactRetentionInterlock โ the same sentence for the container registry: a cleanup whose protection set is stale deletes something still in use.
- ModuleBuildArchitecture โ the lane this seam sits inside.
- BuildCoordination โ the ledger that decides what is rebuilt at all.