Load an Agnos RAG export into LangChain or LlamaIndex

Agnos, a Windows app made by Gidion, cleans documents and exports them as a RAG corpus: a JSONL file of passages, each with an id, the file it came from and where in that file. This guide reads that file into LangChain and into LlamaIndex with a few lines of Python, keeps every field as metadata, builds a small index in memory and runs one query. Every script on this page was run on a real export, and the output shown is what it printed.

What you start with

In Agnos, open Export, choose RAG corpus and press Build and download. You get one JSONL file: one JSON record per line, one record per piece of text. The examples here use the export of the four-page handbook PDF from the headers and footers guide. Download it (1.5 KB) to follow along. Its one record, laid out across several lines and with the text shortened:

{"text": "Northwind Tools Staff Handbook\nThis handbook covers the everyday rules … nobody keeps a copy after\nleaving.",
 "chunk_id": "d733867d65aa37daf2259108e3323ff6_000000",
 "source": "northwind-staff-handbook.pdf",
 "file_id": 1,
 "uploaded_at": "2026-09-23T23:02:03.677203Z",
 "quality_score": 0.8541,
 "language_confidence": 0.0,
 "content_type": "prose",
 "chunk_index": 0,
 "page_start": 1,
 "page_end": 4}

Both libraries want the same two things for each piece: its text, and a dictionary of metadata. The fields map like this:

Which fields a record has depends on the file: page_start and page_end for PDFs, slide_start and slide_end for decks, heading_path for Word, Markdown and web pages, sheet for Excel, and record (with record_id when the record has its own id) for each JSON record and each spreadsheet or CSV row read as Records. A field with nothing to say is left out, so the code below never assumes one is there. Every field is described in Export formats.

The handbook is short, so Agnos kept it as one piece, and the searches below have only that piece to find. A longer export is read by exactly the same code.

Set up

Tested on 17 October 2026 with Python 3.14.3 on Windows 11, and these versions:

pip install langchain-core==1.6.12 numpy==2.5.4 llama-index-core==0.14.25

Install only the one you use: langchain-core and numpy for LangChain (the stand-in embedding and the in-memory store both need numpy, which langchain-core does not install), or llama-index-core for LlamaIndex. The scripts need no API key and no model download. Save each one next to the export file and run it from that folder, or set EXPORT to the file’s full path, written with an r in front: EXPORT = r"C:\Users\you\Downloads\agnos-dataset_20260923_141502.jsonl".

Into LangChain

import json
import uuid

from langchain_core.documents import Document
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.vectorstores import InMemoryVectorStore

EXPORT = "northwind-staff-handbook-rag.jsonl"


def load_agnos(path):
    docs = []
    with open(path, encoding="utf-8") as f:
        for line in f:
            if not line.strip():
                continue
            record = json.loads(line)
            text = record.pop("text")
            record.pop("token_ids", None)  # only there if you ticked Token ids
            docs.append(Document(id=record.get("chunk_id") or str(uuid.uuid4()), page_content=text, metadata=record))
    return docs


docs = load_agnos(EXPORT)
print(len(docs), "document(s)")
print("id:", docs[0].id)
print("metadata:", docs[0].metadata)

embeddings = DeterministicFakeEmbedding(size=256)  # a stand-in: put your embedding model here
store = InMemoryVectorStore(embeddings)
store.add_documents(docs)

question = "How many days a week can staff work from home?"
for doc, score in store.similarity_search_with_score(question, k=3):
    print(f"{score:.4f}  {doc.id}  {doc.metadata['source']}")

What it printed

1 document(s)
id: d733867d65aa37daf2259108e3323ff6_000000
metadata: {'chunk_id': 'd733867d65aa37daf2259108e3323ff6_000000', 'source': 'northwind-staff-handbook.pdf', 'file_id': 1, 'uploaded_at': '2026-09-23T23:02:03.677203Z', 'quality_score': 0.8541, 'language_confidence': 0.0, 'content_type': 'prose', 'chunk_index': 0, 'page_start': 1, 'page_end': 4}
0.0133  d733867d65aa37daf2259108e3323ff6_000000  northwind-staff-handbook.pdf

• text becomes page_content. Every other field goes into metadata exactly as the export wrote it, chunk_id included.
• The Document’s id is the chunk_id, and InMemoryVectorStore stores each piece under that id, so store.get_by_ids(["d733867d65aa37daf2259108e3323ff6_000000"]) returns this piece.
• A piece you put back with Keep this anyway has no chunk_id, so the script gives it a random id, a new one each time it runs.
• token_ids is only in the file if you ticked Token ids on Export. It is a long list of numbers that no search needs, so the script drops it.
• LangChain embeds only page_content. The metadata travels with each result, for citing and filtering.
• DeterministicFakeEmbedding turns each text into random numbers, the same ones every time. It lets the code run without a model, but it cannot match by meaning, so the score of 0.0133 means nothing. The search asks for three results and gets one, because the export has one piece.

To use the store in a chain, store.as_retriever(search_kwargs={"k": 3}) gives you a retriever over the same pieces.

Into LlamaIndex

import json
import uuid

from llama_index.core import VectorStoreIndex
from llama_index.core.embeddings import MockEmbedding
from llama_index.core.schema import MetadataMode, TextNode

EXPORT = "northwind-staff-handbook-rag.jsonl"

# Fields the embedding model and the LLM see. The rest stay on the node.
SHOWN = {"source", "page_start", "page_end", "slide_start", "slide_end", "heading_path", "sheet", "record", "record_id"}


def load_agnos(path):
    nodes = []
    with open(path, encoding="utf-8") as f:
        for line in f:
            if not line.strip():
                continue
            record = json.loads(line)
            text = record.pop("text")
            record.pop("token_ids", None)  # only there if you ticked Token ids
            hidden = [key for key in record if key not in SHOWN]
            nodes.append(TextNode(
                id_=record.get("chunk_id") or str(uuid.uuid4()),
                text=text,
                metadata=record,
                excluded_embed_metadata_keys=hidden,
                excluded_llm_metadata_keys=hidden,
            ))
    return nodes


nodes = load_agnos(EXPORT)
print(len(nodes), "node(s)")
print("id:", nodes[0].node_id)
print("embedded as:", repr(nodes[0].get_content(metadata_mode=MetadataMode.EMBED)[:90]))

embed_model = MockEmbedding(embed_dim=256)  # a stand-in: put your embedding model here
index = VectorStoreIndex(nodes, embed_model=embed_model)
retriever = index.as_retriever(similarity_top_k=3)

question = "How many days a week can staff work from home?"
for hit in retriever.retrieve(question):
    print(f"{hit.score:.4f}  {hit.node.node_id}  {hit.node.metadata['source']}")

What it printed

1 node(s)
id: d733867d65aa37daf2259108e3323ff6_000000
embedded as: 'source: northwind-staff-handbook.pdf\npage_start: 1\npage_end: 4\n\nNorthwind Tools Staff Hand'
1.0000  d733867d65aa37daf2259108e3323ff6_000000  northwind-staff-handbook.pdf

• Each record becomes a TextNode whose id is the chunk_id, or a random id for a piece put back with Keep this anyway, with every field in metadata. VectorStoreIndex(nodes, ...) indexes the nodes as they are, so the pieces stay as Agnos cut them.
• Pass embed_model, or set Settings.embed_model once. Without either, LlamaIndex reaches for OpenAI’s embeddings: with only llama-index-core installed, it stops with an ImportError asking for llama-index-embeddings-openai.
• MockEmbedding gives every text the same numbers, so every score is 1.0. Like the LangChain stand-in, it is only there so the code runs.

Choose which fields the model sees

LlamaIndex writes a node’s metadata in front of its text, both when it makes the embedding and when it sends the piece to an LLM. Without the two excluded_ lists, the text it embeds for the handbook starts like this:

chunk_id: d733867d65aa37daf2259108e3323ff6_000000
source: northwind-staff-handbook.pdf
file_id: 1
uploaded_at: 2026-09-23T23:02:03.677203Z
quality_score: 0.8541
language_confidence: 0.0
content_type: prose
chunk_index: 0
page_start: 1
page_end: 4

Northwind Tools Staff Handbook
This handbook covers the everyday rules for everyone who works at Northwind Tools. It is reissued

An id, an upload time and a quality score do not help a search, and they cost tokens every time the piece is sent to a model. The script lets the model see only the file name and the place in the file; the output above shows the embedded text now starts with the source. The two lists are separate, so you can embed one set of fields and show the LLM another. Every field stays in node.metadata either way.

TextNode or Document

LlamaIndex treats a Document as a whole file to be cut up. Give Agnos records to VectorStoreIndex.from_documents as Documents and LlamaIndex runs its own splitter over each one and gives every node it makes a new random id. The chunk_id survives only as the node’s ref_doc_id and in its metadata. Use Document if you want LlamaIndex to cut the text again; use TextNode, as above, to keep Agnos’s pieces and ids.

To have a model write an answer from the matches, index.as_query_engine(llm=...) takes your LLM. That needs a model, so it is not part of the tested code.

Put in a real embedding model

Each script marks the one line to change. Replace the stand-in with the embedding class for your model, from its integration package:

We checked that these classes are in the releases of those packages current on 17 October 2026, but we did not run them, because they need an API key or a model download. Embed the pieces and the questions with the same model, and build the index again if you change it.

Show where an answer came from

Every piece carries its place in its file, so an answer can cite it. This turns the metadata into a short label, and works with either library:

def span(name, first, last):
    return f"{name} {first}" if first == last else f"{name}s {first}-{last}"


def where(meta):
    if meta.get("page_start") is not None:
        return span("page", meta["page_start"], meta["page_end"])
    if meta.get("slide_start") is not None:
        return span("slide", meta["slide_start"], meta["slide_end"])
    if meta.get("heading_path"):
        return " > ".join(meta["heading_path"])
    if meta.get("record") is not None:
        place = f"record {meta['record']}"
        return f"sheet {meta['sheet']}, {place}" if meta.get("sheet") else place
    if meta.get("sheet"):
        return f"sheet {meta['sheet']}"
    return ""

Call it with doc.metadata in LangChain or hit.node.metadata in LlamaIndex. For the handbook piece it returns 'pages 1-4'. For the Word record shown in Export formats, it returns 'Northwind Tools Staff Handbook > 1. Working hours'. For a spreadsheet row read as Records it returns a label such as 'sheet Orders, record 3'.

Things to know

Metadata some stores refuse

The in-memory stores above take any metadata. Some vector stores accept only single values, such as text and numbers, and refuse a list. heading_path and restored_from are lists, and record_fields is an object. This turns them into text before you add the pieces:

import json


def flatten(meta):
    flat = {}
    for key, value in meta.items():
        if key == "heading_path":
            value = " > ".join(value)
        elif isinstance(value, (list, dict)):
            value = json.dumps(value, ensure_ascii=False)
        flat[key] = value
    return flat

Use metadata=flatten(record) in either loader. On the Word record, heading_path becomes 'Northwind Tools Staff Handbook > 1. Working hours'. The handbook record has no lists, so it comes back unchanged.

When a file changes

Remove the old file’s pieces, then add the pieces from a new export. After the scripts above:

# LangChain
old = [doc.id for doc in docs if doc.metadata["source"] == "northwind-staff-handbook.pdf"]
store.delete(ids=old)

# LlamaIndex
old = [node.node_id for node in nodes if node.metadata["source"] == "northwind-staff-handbook.pdf"]
index.delete_nodes(old, delete_from_docstore=True)

Then add the new pieces with store.add_documents(...) or index.insert_nodes(...).

One id, one piece

chunk_id is unique within an Agnos project. Both in-memory stores keep one entry per id: add a piece whose id is already there and it replaces the old one, with no warning. If you load exports from several projects into one index, check that no id appears twice.

No need to split again

Agnos has already cut the text into pieces, so there is no text splitter in these scripts. If a piece is too long for your embedding model, split it, and keep in mind that each part keeps the whole piece’s metadata: every part of the handbook piece would still say pages 1 to 4.

Ready-made JSON readers

Both projects publish JSON readers too. LangChain’s JSONLoader, in langchain-community, reads JSONL with json_lines=True and jq_schema=".", a content_key for the text and a metadata_func for the other fields; it needs the jq package. LlamaIndex’s JSONReader, in llama-index-readers-json, reads JSONL with is_jsonl=True, but it writes each whole record, field names included, into the text rather than into metadata. For an Agnos export, the plain json loop above does the same job with nothing extra to install.

Agnos makes the export on your computer, and nothing is uploaded. Once it is saved, the file is yours to load into any tool. For what each field means and how the pieces are made, see PDF to JSONL for RAG and fine-tuning.