{
  "author": "Joseph Walker",
  "confidence": "provisional",
  "date_created": "2026-09-05",
  "date_modified": "",
  "date_published": "2026-09-06",
  "dek": "The four open-source retrieval libraries examined here disagree about what a chunk is called, what its name is a function of, and whether it survives a rebuild. None offers a cross-runtime identity contract — so a corpus expected to outlive its retrieval stack has to own that contract itself.",
  "finding_id": "MCG-F-0003",
  "in_ask_corpus": true,
  "kind": "finding",
  "limitations": [
    "This is a source review of four libraries at four commits, plus an implementation over one small corpus. It is not a benchmark and says nothing about retrieval quality.",
    "Identity behaviour is configurable in most of these libraries. The finding is about what an identifier is a function of on the default path, not about what each library can be made to do.",
    "No embedding model was run and no library was installed. The claim that a text-derived identity makes a stale vector visible is a property of the construction, not an observation of a migration.",
    "The Haystack lifecycle result comes from a standard-library reproduction of the pinned source rather than from executing the library, and only one embedder implementation was read. It establishes the dataclass semantics, which is the load-bearing step, and the behaviour of the reviewed embedding path — not the behaviour of every embedder and document store in the ecosystem.",
    "Unstructured's deterministic identity was read rather than executed; the claim is about what the code does on the default partition path.",
    "The corpus tested contains three publications. Chunking behaviour at larger scale or over structurally different material is unmeasured.",
    "Library versions move quickly. Every statement here is pinned to a commit and is a fact about that commit only."
  ],
  "methods": [
    "Select four open-source retrieval libraries whose document models a corpus might otherwise inherit. No claim is made about their relative adoption; the conclusion rests on the differences between them, not on how many people use them.",
    "Resolve each to an immutable commit and read the identity code there, rather than reading documentation or a moving branch.",
    "For each, ask two questions: what is the identifier a function of, and does an independent rebuild reproduce it?",
    "Where reading the formula could not settle the lifecycle, reproduce the dataclass semantics from the pinned source with the standard library and observe the result directly.",
    "Implement the resulting two-identifier design over the real Masters publication corpus.",
    "Demonstrate the required properties against published pages: edit prose, edit page furniture, and repartition a section by inserting a block earlier in it."
  ],
  "negative_results": [
    "No open interchange convention for provenance-preserving chunk identity was demonstrated in this review. What the search returned was library-internal data models, none of which is offered as a contract between systems. That is a weaker prior-art position than the claim and evidence layers examined in MCG-F-0001, where W3C PROV and the Wikibase statement model exist as standards.",
    "None of the four libraries is a substitute for a corpus contract, and none was adopted as a runtime. Doing so would take a Python dependency tree into a toolchain that has none, in exchange for reading HTML, grouping blocks and writing JSON — which is not the difficult part.",
    "Content addressing alone does not work, despite being the obvious correction to random identity. Deriving the identifier only from the text makes citations fragile: fixing a typo would move an address a reader had already recorded.",
    "The first pass of this review reached two wrong conclusions, both from reading a default in isolation rather than the path that actually runs. It reported Unstructured as defaulting to UUIDs, when the partition pipeline defaults to deterministic hashes and UUIDs are opt-in; and it inferred from Haystack's id formula that changing model re-identifies every document, when an existing id is preserved through the embedding step. Reading a data class is not the same as reading a lifecycle.",
    "The reviewed libraries are not badly designed. A framework that owns the chunk, the vector and the index in one process has no reason to separate identity from representation. The coupling only becomes a defect when the corpus is expected to outlive the framework."
  ],
  "reconsider_if": [
    "An open interchange convention for provenance-preserving chunk identity emerges with real adoption, which would move this from 'define it yourself' to 'export to it'.",
    "A reviewed library adopts a durable, model-independent identity on its default path, which would narrow the heterogeneity this Finding describes.",
    "A migration between embedding models is actually performed and the self-invalidating identity fails to make stale vectors visible in practice.",
    "A retrieval experiment shows the chosen chunk unit is wrong for Masters prose, which would change the chunking policy without touching the identity conclusion.",
    "The corpus grows to a size where per-chunk digests become an operational cost rather than a convenience."
  ],
  "related_claims": [],
  "related_findings": [
    "MCG-F-0001"
  ],
  "related_sources": [],
  "research_question": "What must remain stable in a RAG-ready publication corpus if embedding models and retrieval runtimes are expected to change?",
  "result": "The four libraries give four materially different answers, and none of them is a contract another system could adopt. LlamaIndex generates a random UUID per node unless the caller supplies one, so an independent rebuild does not reproduce identity. Unstructured's partition pipeline defaults to a deterministic hash of filename, text, page number and per-page sequence — reproducible and independent of any model, but positional, so structural insertion can move it. Haystack preserves an existing id on the reviewed embedding path, but an id generated while a vector is present is a function of that vector, which couples identity to representation on that construction path. LangChain leaves the id optional and imposes no provenance semantics at all. The consequence is not that these libraries are wrong for their own purposes: it is that there is no shared identity contract to inherit, so a corpus expected to outlive its retrieval runtime must define identity explicitly, and the useful shape is two identifiers rather than one — a stable public referent that citations point at, and a revision-specific derived identity that a stored vector is keyed by.",
  "schema_version": 1,
  "scope": "The chunk and document identity semantics of four open-source retrieval libraries, read at immutable commits, assessed against one question: what is an identifier a function of, and does it survive an independent rebuild? Retrieval quality, chunking quality and embedding model quality were all out of scope. No embedding was generated, and no library was installed; the one lifecycle question that could not be settled by reading was settled with a standard-library reproduction of the pinned source.",
  "sources": [
    {
      "accessed": "2026-09-05",
      "note": "MIT. BaseNode declares id_ with default_factory=lambda: str(uuid.uuid4()), so identity is random unless the caller supplies one. NodeRelationship is SOURCE/PREVIOUS/NEXT/PARENT/CHILD and RelatedNodeInfo carries the target's hash.",
      "title": "LlamaIndex — llama_index/core/schema.py at v0.14.24",
      "url": "https://github.com/run-llama/llama_index/blob/9ba74b8628712e68d16955d9492b5192bd7e6f00/llama-index-core/llama_index/core/schema.py"
    },
    {
      "accessed": "2026-09-05",
      "note": "Apache-2.0. OBJECT FALLBACK: the Element.id property lazily assigns uuid.uuid4() when no id was set. id_to_hash computes sha256(filename + text + page_number + sequence_number)[:32].",
      "title": "Unstructured — Element.id and id_to_hash at 0.27.5",
      "url": "https://github.com/Unstructured-IO/unstructured/blob/d68ab0fda9bcac317b7c033d42412d716ba1d790/unstructured/documents/elements.py"
    },
    {
      "accessed": "2026-09-06",
      "note": "NORMAL PARTITION OUTPUT: the decorator reads unique_element_ids (default False) and calls assign_and_map_hash_ids(elements) when it is false, so deterministic hashed identity is the default and UUID identity is opt-in. Sequence numbers are assigned per page by grouping element page numbers.",
      "title": "Unstructured — process_metadata partition default at 0.27.5",
      "url": "https://github.com/Unstructured-IO/unstructured/blob/d68ab0fda9bcac317b7c033d42412d716ba1d790/unstructured/documents/elements.py#L620"
    },
    {
      "accessed": "2026-09-05",
      "note": "Apache-2.0. _create_id hashes content, blob, mime type, meta, embedding and sparse_embedding together. __post_init__ is self.id = self.id or self._create_id(), so an existing id is preserved.",
      "title": "Haystack — haystack/dataclasses/document.py at v3.1.1",
      "url": "https://github.com/deepset-ai/haystack/blob/d60cce01a778bc3498a02866fe288b3c07398649/haystack/dataclasses/document.py"
    },
    {
      "accessed": "2026-09-06",
      "note": "The one embedder implementation read for this review. It attaches vectors with replace(doc, embedding=...), which carries the existing id through __post_init__ unchanged. Other embedders at this version were not inspected.",
      "title": "Haystack — OpenAIDocumentEmbedder at v3.1.1",
      "url": "https://github.com/deepset-ai/haystack/blob/d60cce01a778bc3498a02866fe288b3c07398649/haystack/components/embedders/openai_document_embedder.py"
    },
    {
      "accessed": "2026-09-05",
      "note": "MIT. Document is page_content plus a free-form metadata dict and an optional id defaulting to None; no provenance or identity semantics are imposed.",
      "title": "LangChain — langchain_core/documents/base.py at langchain-core 1.6.2",
      "url": "https://github.com/langchain-ai/langchain/blob/8215039dea978372bd3fd95b88663a11b0159043/libs/core/langchain_core/documents/base.py"
    },
    {
      "accessed": "2026-09-05",
      "note": "Derivation and attribution relationships; the same instinct as recording what an edge pointed at rather than only that it pointed.",
      "title": "W3C PROV-O: The PROV Ontology",
      "url": "https://www.w3.org/TR/prov-o/"
    }
  ],
  "status": "published",
  "superseded_by": "",
  "supersedes": [],
  "systems_evaluated": [
    "LlamaIndex — llama-index-core 0.14.24",
    "Unstructured 0.27.5",
    "Haystack — haystack-ai 3.1.1",
    "LangChain — langchain-core 1.6.2"
  ],
  "title": "Chunk identity across four retrieval libraries: four different answers, and no shared contract to inherit",
  "topics": [
    "retrieval",
    "chunk identity",
    "provenance",
    "content addressing",
    "build versus reuse"
  ],
  "truth_boundary": "This is a provisional Finding. The result held under the stated method and scope, and has not yet been repeated or tested against a wider sample.",
  "url": "https://mastersconsultinggroup.org/research/findings/chunk-identity-in-rag-ready-publication-corpora/",
  "withdrawn_reason": ""
}