Export formats

What is inside each kind of export, field by field. How to choose an export and set its options is in Export.

File names

An export is named from the File name option (agnos-dataset unless you change it), followed by the date and time it was built:

agnos-dataset_20260923_141502.jsonl
agnos-dataset_20260923_141502.zip

Characters other than letters, digits, dots, dashes and underscores become a dash. If the name changes, the field shows the final one after Saved as.

RAG corpus

One JSONL file: one JSON record per line, one piece per record, in document order. This line is from a Word document cleaned with the RAG / Retrieval Corpus purpose:

{"text": "## 1. Working hours\n\nThe workshop is staffed from 08:00 to 18:00. Anyone working outside those hours tells their team lead first, in writing.", "chunk_id": "9f2c41d7a0b34e6c8d15f0a2b7c93e61_000001", "source": "staff-handbook.docx", "file_id": 12, "uploaded_at": "2026-09-23T09:41:07.218532Z", "quality_score": 0.84, "language_confidence": 0.0, "content_type": "prose", "chunk_index": 1, "heading_path": ["Northwind Tools Staff Handbook", "1. Working hours"]}

Which fields appear depends on the document and the project's Output settings. A field with nothing to say is left out: a Word document has no pages, so its pieces have no page numbers.

Cleaned documents

A zip with one plain text file per document, named after the original, and a manifest:

agnos-dataset_20260923_141502.zip
  faq.txt
  returns-policy.txt
  staff-handbook.txt
  manifest.json

If two documents share a name, the second becomes staff-handbook_2.txt. A document you changed in Review is rebuilt from its pieces, so it holds only what you kept.

Upload pack for an AI tool

A zip of cleaned documents as Markdown, made to fit the file limits of the tool chosen under The AI tool you will upload to, with a note and a manifest:

agnos-dataset_20260923_141502.zip
  faq.md
  returns-policy.md
  staff-handbook.md
  READ ME FIRST.txt
  manifest.json

The text is the same cleaned text as in Cleaned documents, and a document you changed in Review is rebuilt from its pieces in the same way. The tools and their limits are in Export.

One file per document, while that fits

Each document gets a .md file named after it: staff-handbook.docx becomes staff-handbook.md. Two documents with the same name become notes.md and notes 2.md. Characters other than the letters A to Z, digits, spaces, dots, dashes, underscores, round brackets and plus signs become an underscore, and the part of the name taken from the document is cut at 60 characters.

Past the file limit

When there are more documents than the tool takes files, neighboring documents are combined, as evenly as possible and keeping their order. A combined file is named after its first document and how many follow it, such as faq + 3 more.md. Inside it, a line holding --- separates one document from the next.

Past the word limit

A document longer than the tool's word limit is split at paragraph breaks, into files such as staff-handbook (part 2 of 3).md. A single paragraph longer than the limit is cut between words.

A heading on every document

Each document starts with its original file name as a # heading, such as # staff-handbook.docx, so a combined file still says where each passage came from.

READ ME FIRST.txt

A note to read before you upload. It names the tool and the limits the pack was made for, and lists:

• what is in each file, with word counts;
• any document left out because no text was found in it;
• what Agnos took out of each document, by rule, such as “52 lines (running headers and footers 40, page numbers 12)”;
• a NOTE to upload in batches, or export fewer documents at a time, when the documents need more files than the tool takes.

Upload the .md files. READ ME FIRST.txt and manifest.json are for you rather than for the tool.

The upload_pack block

The pack's manifest.json has everything the manifest always has, and an upload_pack block:

target, target_label

The tool chosen: any, chatgpt, claude, gemini or notebooklm, and its name as the note gives it.

max_files, max_words

The limits the pack was made to. max_files is 0 when the tool sets no count, as for Claude: every document keeps a file of its own.

fits

true when the pack needs no more files than the tool takes.

files[]

One entry per .md file: its name, its word count, and the documents in it, each with its part number and word count.

documents_with_no_text

The documents left out because no text was found in them.

Training data

A zip of records that hold only the text, one per line:

{"text": "## 1. Working hours\n\nThe workshop is staffed from 08:00 to 18:00. Anyone working outside those hours tells their team lead first, in writing."}

Without a validation split, the zip holds data.jsonl and manifest.json. With one, it holds train.jsonl, val.jsonl and manifest.json.

How the split is made is in Export. The manifest's leak_check counts any text that still ended up on both sides.

Everything

agnos-dataset_20260923_141502.zip
  data.jsonl               (train.jsonl and val.jsonl with a split)
  rejected.jsonl
  duplicates.jsonl
  near_duplicates.jsonl
  settings/<hash>.json
  manifest.json
  dataset_card.md

data.jsonl

The dataset, in the same record shape as the RAG corpus.

rejected.jsonl

Every piece not in the dataset, with its reasons and the stage that removed it: cleaning, quality, or review for pieces you left out (reason removed_by_reviewer). A line from a PDF carries its page, and a line marked in Review's Check these first carries check, the reason it might be content (such as "table or figure label"). Cleaning settings matches each reason to the label Review shows.

duplicates.jsonl

Every piece left out because the same text was already there from another file, with the file it was kept in and the file it was dropped from.

near_duplicates.jsonl

Up to 5,000 pairs of nearly identical pieces, with how alike they are and whether the repeat was removed. It is written either way, so you can see what Keep one copy, counting near matches would remove before you choose it.

settings/<hash>.json

The settings each file was cleaned with, one file per distinct set. The manifest says which file used which.

A line from each list: two from rejected.jsonl, then one from duplicates.jsonl and one from near_duplicates.jsonl (texts shortened):

{"chunk_index": 5, "text": "Questions? Write to support.", "reasons": ["exact_duplicate"], "stage": "quality", "heading_path": ["Returns"], "source": "faq.md", "file_id": 13}
{"text": "We use cookies to improve your experience. Accept all", "reasons": ["web_boilerplate"], "stage": "cleaning", "source": "support-article.html", "file_id": 15}
{"text": "Returns are accepted within 30 days of delivery when the item is unused and in its original packaging.", "kept_in": "returns-policy.md", "kept_file_id": 11, "dropped_from": "faq.md", "dropped_file_id": 13, "chunk_index": 4}
{"similarity": 0.9412, "same_file": false, "removed": false, "first": {"file_id": 11, "file": "returns-policy.md", "chunk_index": 2, "page_start": null, "page_end": null, "heading_path": ["Returns"], "text": "..."}, "repeat": {"file_id": 13, "file": "faq.md", "chunk_index": 6, "page_start": null, "page_end": null, "heading_path": ["Returns"], "text": "..."}}

The manifest

manifest.json is in every zip. It records what the dataset is, what was taken out, and which settings made it, so anyone handed the dataset can answer those questions later.

generated_at, agnos_version, profile, dataset_name, options

When and how it was built. profile is rag, documents, upload_pack, training or bundle (Everything).

total_files, total_chunks, total_tokens, tokens_estimated

The totals. tokens_estimated is true when tokens were not counted and the total is estimated at about four characters per token.

quality

The lowest and the mean quality score.

files

One entry per file: pieces kept and removed, repeats found, tokens, the settings used and whether they are out of date, your Review decisions, and the side of the split. source_sha256 is the SHA-256 of the file you imported, lines_removed_in_cleaning counts the whole lines cleaning took out, and lines_removed_by_rule splits that count by rule.

audit

The record for an audit, in one place: when the export was made, the Agnos version, where the hashes and the removal counts are, and that the files were processed on this computer with nothing uploaded. With rejected.jsonl from Everything, it says what went in, what came out, and which rule took out each line.

duplicates, near_duplicates

How repeated text was handled, how much was found and removed, and from which files.

split

Whether and how the files were split, what is on each side, leak_check, and what the split cannot detect.

reviewer_decisions

How many pieces a person kept or left out by hand, counted apart from the rules.

stale_files, notes

Files cleaned with settings the project has since changed, and plain sentences about anything worth knowing.

The Cleaned documents manifest also lists each text file under documents. The upload pack's has the upload_pack block.

The dataset card

dataset_card.md comes with Everything. It is the manifest written for a person, in Markdown. It starts like this:

# agnos-dataset

Generated: 2026-09-23T14:15:02.114873Z
Agnos version: 2.3.0
Export profile: bundle

## Summary
- **Files**: 5
- **Chunks**: 19
- **Tokens**: about 1,078
- **Quality** (mean / min): 0.78 / 0.49
- **Languages**: unknown: 19

It then lists the options used and notes on anything worth knowing. When they apply, it also covers repeats removed, decisions made in Review, and the split with its leak_check.