Chunk identity across four retrieval libraries: four different answers, and no shared contract to inherit

Four libraries, four answers

Four open-source retrieval libraries were read at immutable commits rather than from documentation, because this behaviour lives in default arguments and decorators rather than in prose. No claim is made here about their relative adoption — the conclusion rests on how far apart their answers are, not on how many people use them.

LlamaIndex assigns a fresh random identifier every time. That is not a mistake — it is unique, which is what a library holding the chunk and the vector in one process needs. It is simply not an address: rebuild the corpus and every stored vector is orphaned, not because anything changed but because the identifiers were never a function of anything.

Unstructured is the closest thing here to a durable answer, and it is worth being precise about it. The Element.id property does fall back to a UUID — but that is the bare-object fallback, not what partitioning produces. The partition pipeline is decorated with process_metadata(), which reads unique_element_ids (default False) and calls assign_and_map_hash_ids() when it is false. So deterministic hashed identity is the default and UUIDs are the opt-in. The hash covers filename, text, page number and per-page sequence, which makes it reproducible and indifferent to any model — but also positional, so inserting content earlier in a document can move the identifiers after it.

Haystack repays careful reading, and punished careless reading here. Its _create_id() does hash the embedding and sparse embedding along with the content and metadata. It does not follow that changing model re-identifies anything: __post_init__ is self.id = self.id or self._create_id(), and the embedders attach vectors with replace(doc, embedding=...), which carries the existing id straight through. Embedding and re-embedding preserve identity. What is true is narrower: a document whose id is generated while a vector is already attached has an identifier that is a function of that vector — a real coupling between identity and representation, on one construction path rather than as a lifecycle guarantee.

LangChain takes no position. Document is page content, a free-form metadata dict and an optional id. That is a reasonable interchange shape and it is not a contract; nothing about provenance is defined, so nothing about provenance is guaranteed.

What that heterogeneity means

The useful result is not that any of these is wrong. Each is coherent for a framework that owns the chunk, the vector and the index together.

The result is that there is nothing here to inherit. Four libraries give four different answers to "what is this passage called", ranging from random to positional to conditionally model-dependent to undefined. A corpus that expects to outlive whichever runtime is currently reading it cannot adopt one of these as its identity contract, because none of them is offered as a contract between systems in the first place.

So identity has to be an explicit property of the corpus. And once it is written down deliberately, it turns out one identifier cannot do the job.

Why one identifier is not enough

The obvious correction to random identity is content addressing: derive the identifier from the text and it becomes reproducible. That fixes the rebuild problem and creates a worse one. If the identifier is the text digest, then correcting a typo moves the address. A reader who cited a section last year finds their reference no longer resolves — not because the section was withdrawn, but because a comma changed.

The tension resolves by noticing that two different things were being asked of one identifier:

• a public referent — the publication and its section anchor — which is what a citation points at, which corresponds to a real anchor in the published HTML, and which survives rewording, repartitioning and a rename of the heading itself;
• a derived chunk identity — the referent, the position within that section, and a digest of the exact text — which is what a vector is keyed by, and which moves whenever either the text or the partitioning moves.

The public referent is deliberately not unique: every chunk cut from one section shares it, because they are all in that section and a reader citing the section means the section. Uniqueness is the derived identifier's job.

The second identifier's property is what pays for the arrangement. A vector stored against it is self-invalidating: after an edit the old vector is attached to an identifier the corpus no longer contains. Nothing has to remember to expire it, and no separate freshness table has to be maintained and trusted.

Neither identifier contains a model name, a vector, a cluster, a timestamp or a random component.

What was demonstrated

The design was implemented over the real Masters publication corpus and tested against published pages rather than fixtures.

Editing a sentence of published prose moves the derived identity and leaves the public referent untouched. Editing page furniture — an eyebrow label, a date line — moves nothing at all, which matters more than it sounds: an editor adjusting a caption should not silently invalidate a page's embeddings.

The sharper test is repartitioning. Inserting a whole paragraph earlier in a long section changes how the section groups: the positions after it shift and several chunks' text changes. The derived identities move, as they must. The public referent does not, because the anchor did not — a citation recorded before the edit still lands exactly where it did. A single-word edit never exercises that, which is why the first version of this test was not enough.

The sharpest test exposed a defect rather than confirming a design. The public referent was only as durable as the anchor, and the anchors were being slugified from the heading's wording — so renaming a heading silently moved every citation aimed at it, and no test noticed because no test renamed a heading. The fix belongs in the publication layer rather than in the corpus: the constrained Markdown grammar now carries an explicit section id,

## Four libraries, four answers {#four-libraries-four-answers}

which is consumed rather than displayed, and a published record whose body headings lack one fails the build. Display text and address are now independent things, and the test that proves it renames a heading and checks that the anchor, the section URL and the locator all stay put while the visible heading and the heading path change.

A second result came out of the same work. The site already had a deterministic extractive passage layer for its reference desk, with hard-won rules about what counts as published prose and what is furniture. Rather than write a second parser with its own opinion, the policy was shared and the implementations left separate — and the two extractors were then shown to produce identical text across all 119 sections of all 14 public pages. Sharing the boundary rule while keeping the extractors independent proved better than consolidating them, because the older layer's integrity test depends on re-deriving text through a genuinely different code path.

What this does not settle

No embedding was generated. The claim that a text-derived identity makes a stale vector visible is a property of how the identifier is constructed, not an observation of a migration that was performed. That comes next, and this order is deliberate: a contract a model attaches to should exist and be stable before the model is chosen, or the model's output shape quietly becomes the schema and the first migration becomes a migration of the corpus.

Nor does it say whether the chunks are good — whether the unit suits retrieval, whether the boundaries fall where a reader would put them. Those need an evaluation with an unseen holdout, over a corpus larger than three publications.

There is also a methodological result worth keeping. The first pass of this review got two of the four libraries wrong, in the same way both times: it read a default in isolation instead of the path that actually runs. Unstructured's UUID fallback is real but is not what partitioning does; Haystack's id formula is real but does not imply the lifecycle that was inferred from it. Reading a data class is not the same as reading a lifecycle, and a prior-art review that stops at the first plausible line of code will produce confident, wrong comparisons.

What the corrected evidence does settle is narrower and load-bearing: an institution that expects to change retrieval models should not let the retrieval library decide what its passages are called. Identity is a property of the record, not of the tool currently reading it.