Content Chunk Navigation

When a content file is indexed, its extracted text is sliced into overlapping chunks and each chunk is embedded for similarity search. Most retrieval surfaces hide the chunk and answer at the file level — "which Document matches?" — which is the right answer for navigation. This page covers the layer underneath: reading the chunks themselves, by similarity and by index, so an agent can pull the exact passage it needs and step through a file's neighbouring windows.

The chunking model

ContentIndexingService splits each file's extracted text into fixed-size character windows:

Property Value
Window size 1000 characters
Overlap 150 characters (each window repeats the trailing 150 chars of the previous one)
Index chunk_index — 0-based, per file, contiguous

The overlap means a sentence straddling a window boundary still lands wholly inside at least one chunk, so a semantic hit is never split across two windows. Every chunk of one file carries the same whole-file content_hash (the hash gate's "did this file change?" key); per-chunk identity is chunk_index.

Chunks live in a per-partition Postgres content_chunks table — one schema per partition, exactly like mesh_nodes and the satellite tables (see Postgres Schema Architecture). The columns are collection_path, file_path, chunk_index, content_hash, chunk_text, metadata, embedding, page, bbox, last_modified. The in-memory store (InMemoryChunkedContentVectorStore) backs the unit-tested core with the identical contract.

The page + bbox columns are source provenance — see Source provenance below. They are added by an idempotent ADD COLUMN IF NOT EXISTS in the (self-provisioning, per-partition) schema script, so an existing partition gains them on its next provision — there is no separate DbVersion migration.

Retrieval vs. extraction

There are two different jobs, and they want different granularities:

🚨 Full text lives at the file path, not the Document node. A search hit is the Document node at {collectionPath}/_Documents/{slug}, and Get on that returns only the node's metadata + the AI Summary (~a few hundred chars) — never the text. The complete extracted text is served by Get on the file path {collectionPath}/{filePath}, through the registered content transformers (#396). Derive the file path from the hit's own collectionPath + filePath fields (search_chunks returns both; the Document node carries them too) — do not reconstruct it by editing the node path, because {slug} is a lossy encoding of filePath (directory separators and non-ASCII collapse to -). Getting the Document node, seeing only a summary, and concluding "the full text is unavailable" is the failure that sends an agent stepping through every chunk — exactly the chunk-boundary corruption this section warns against.

The chunk tools are deliberately not a substitute for a full-document read. If you need every row of a table, a windowed chunk is the wrong tool — it may start or end mid-table.

The three tools

Tool Granularity Returns Use when
search Document node the file's Document node (chunk index dropped) "which file is about X?" — navigation, linking
search_chunks chunk {documentPath, collectionPath, filePath, chunkIndex, rank, snippet, page?, bbox?} per hit "find the passages about X" — gather context
get_chunk one chunk {text, prevIndex, nextIndex, totalChunks, page?, bbox?, …} read a known chunk + step to neighbours

search — Document-level

Node search (and the autocomplete document: prefix) runs the same cosine search over chunks but resolves each hit up to its Document node via DocumentPaths.For(collectionPath, filePath) and dedupes by file — one result per file, keeping the best-scoring chunk's snippet. The chunk index is discarded. This is the right answer for "take me to the file". On Postgres, bare-text tokens in a node search route through the same HNSW vector path described in Vector Search.

search_chunks — chunk-level

search_chunks(query, scope?, limit=20) embeds the query and runs the cosine search across the in-scope collection(s), but keeps the chunk coordinate. Results are not deduped by file — chunk-level granularity is the whole point — and are capped at limit. Each hit carries the (collectionPath, filePath, chunkIndex) triple you feed back into get_chunk, plus a documentPath for linking and a rank (0-based, best-first). The store returns hits most-similar-first but does not surface the raw cosine distance, so rank is the relevance signal rather than a fabricated score.

Two scope models — the engine (ContentChunkSearch in the indexing core) dispatches by query shape:

Form How collections are resolved Use when
Anchored — scope is a node path the scope path itself plus each ancestor prefix (part/Space/Sub → part/Space/Sub, part/Space, part) you don't know exactly which collection holds the content — "what's relevant to where I am" (the agent's context anchor)
Targeted — namespace:<node>/<collection> in the query the one named collection, resolved by an optional scope: qualifier you know the collection and want only it

For the targeted form the scope: qualifier selects:

When a query carries a namespace: token the targeted form wins and the scope parameter is ignored. With neither a scope path nor a namespace: token there is no collection to anchor on, so the tool returns an empty result with a hint to pass one rather than guessing.

get_chunk — read + step

get_chunk(collectionPath, filePath, chunkIndex) reads exactly one chunk and reports its neighbours:

{
  "found": true,
  "collectionPath": "ACME/Reports",
  "filePath": "pension/2025.txt",
  "chunkIndex": 4,
  "text": "…the full 1000-char window…",
  "prevIndex": 3,
  "nextIndex": 5,
  "totalChunks": 12,
  "page": 4,
  "bbox": { "x": 0.12, "y": 0.34, "w": 0.61, "h": 0.08 }
}

prevIndex is null at index 0; nextIndex is null at the last chunk; totalChunks lets the caller bound the range. An out-of-range index (or a file that was never indexed) returns {found:false, totalChunks, message} carrying the valid range, never an error. This is how an agent reads a search_chunks hit in full and then walks forward or backward through the document a window at a time. page + bbox are present only when the source carried a layout (PDFs) — see Source provenance.

Where the tools live

The search itself is one engine — ContentChunkSearch in the storage-agnostic indexing core (MeshWeaver.ContentCollections.Indexing). Every surface calls it, so they stay in sync by construction — including the GUI's "this maps to a tool call" claim, which is honest precisely because the GUI runs the same code the agent does:

The Explore-index GUI

The Content Indexing settings tab is the in-portal surface for the index. Besides status + "Re-index all content", its Explore index section lets a user search the space's content interactively:

The store reads (GetChunk, GetChunkCount) are on IChunkedContentVectorStore. Like every read in the indexing core they are reactive and cold (IObservable<T>), and the Postgres implementation runs the DB round-trip through the cap-1 pg:vector I/O pool (see Controlled I/O Pooling) — never a bare Observable.FromAsync. When content indexing is not wired into a host, the store/embedder are absent and the tools degrade to a clear "not available" envelope instead of throwing.

Why a search failed — the four answers that must stay apart

search_chunks has to embed the query before it can rank anything, so a refusal from the embeddings backend stops the search after it started. The tool has four genuinely different things to report, and the envelope keeps them apart by construction:

What happened searched count error Who fixes it
The backend refused the credential (401/403) false absent embedding-credential-rejected an operator — Embedding:ApiKey is wrong, expired, or has no source
The backend rejected the request (400/404/422) false absent embedding-request-rejected a developer — Embedding:Model names something it does not serve, or the payload shape is wrong
The backend could not answer (5xx, 429, unreachable, timeout) false absent embedding-unavailable nobody — retry, then check Embedding:Endpoint
No embedding provider is configured false absent search-not-performed nobody — a legitimate state, not a failure
The search ran and matched nothing true 0 absent nobody — this is a real answer

Two properties do the work, and both are load-bearing:

Where the diagnosis is produced

The classification happens at the only layer that can make it — the HTTP response — and is carried down, never re-derived:

OllamaEmbeddingProvider          reads the status AND the response body → EmbeddingRequestException(Failure, Endpoint, Model, StatusCode, ResponseBody)
  → EmbeddingProviderChunkEmbedder   maps EmbeddingFailure → the error discriminator → ChunkEmbeddingException
    → ContentChunkSearch             folds it into a `searched:false` result carrying that discriminator
      → search_chunks / get_chunk / the Explore-index GUI   all three, because all three call the one engine

🚨 EnsureSuccessStatusCode() is the wrong tool here and is deliberately not used: it discards the response body — the only place an OpenAI-compatible server states what it objected to — along with the endpoint identity and any class a consumer could branch on. The bare HttpRequestException it throws instead escaped the MCP tool as "'search_chunks' threw an unhandled exception", which left a caller unable to tell a rejected key from a rejected model name. Both statuses were in fact observed seconds apart from one deployment's two pods.

🚨 Nor may a refusal become an empty result. Returning no hits would make "I could not search" byte-identical to "I searched and found nothing" — strictly worse than throwing, because it is silent. A provider that cannot embed therefore throws a named failure; only a provider that legitimately has no embeddings to offer returns null, and the node-level query path reads that null as its cue to fall back to ILIKE (see Vector Search).

The failure is still reported to operators: ChunkNavigation logs the genuinely-broken arms once per call, at the same level the escaping exception used to be logged at. The "indexing is off" and "you passed no query text" arms are answers rather than incidents and are deliberately not logged.

Source provenance (page + position)

Every chunk carries where it came from in the source document, so a consumer can cite the page and open the source page and mark the exact region — not just quote text.

Extraction is positional. For PDFs the extractor (TextExtractor) builds the text word-by-word from PdfPig's laid-out words and records one span (page + box, in PDF points → normalized top-left) per word. The chunker (TextChunker.ChunkPositioned) then attributes each character window to the page it begins on and unions the boxes of the words inside the window on that page — so a chunk that straddles a line break still gets one tight box, and a chunk that crosses a page boundary is pinned to its starting page. Formats with no layout (txt/markdown/docx) carry a null page/bbox and degrade gracefully.

Marking in the viewer. The Document node's Source area renders the original PDF (PDF.js) and, given the deep-linked chunk's page + bbox, scrolls to that page and overlays a highlight rectangle at the exact region — precise and robust, independent of whether the chunk text can be re-found by string match. When the position is absent it falls back to the verbatim text-match highlight. The block reader also shows · page N in each block's header.

Backfilling existing collections. The plain re-index is hash-gated — an unchanged file is skipped, so it would never gain provenance. The Content Indexing tab's Rebuild button (and ContentIndexingObserver.ReindexAll(..., force: true)) bypasses the hash gate to re-extract, re-chunk and re-store every file, populating page/bbox on files indexed before the feature existed.

Reading a document end to end

A typical agent loop:

  1. search_chunks("accrued benefit obligation", scope: "ACME/Reports") → a ranked list of chunk hits.
  2. Pick the top hit's (collectionPath, filePath, chunkIndex).
  3. get_chunk(...) to read the full window, then follow nextIndex / prevIndex to read the surrounding context.
  4. If the goal is to extract a whole table rather than gather context, Get the file path {collectionPath}/{filePath} (built from the hit's own fields) for the complete text — not the hit's documentPath (the Document node), which returns only metadata + summary (see the note under "Retrieval vs. extraction").

Related: CQRS — Queries vs. Content Access for read semantics, Vector Search for the node-level semantic path.

Reconnecting…
The server was updated. Reloading the page to pick up the latest version.