Cleaning settings

What each purpose does, what every cleaning setting changes, and what each reason in Review means.

Purposes

A purpose sets every cleaning setting at once for one kind of job. How to choose or switch one is in Preview and clean.

RAG / Retrieval Corpus

Documents an AI tool will search and answer questions from (RAG). Keeps headings, tables, lists, code, numbers, short answers and non-English text exactly as written. Removes navigation, cookie banners, page headers and footers, empty pieces, and passages that repeat one already kept.

General Documents

Notes, articles, reports and knowledge bases. Keeps prose that the stricter model-training rules would drop. Removes markup and duplicates.

Default

Balanced settings for most uses. Removes very short pieces, duplicates and obvious markup.

LLM Training

Clean text for training a language model. Keeps long, clean prose. Removes short text, lists, boilerplate and anything the quality rules reject.

Paragraph Split

One piece per paragraph, for articles and reports. Removes duplicates and very short paragraphs.

Deep Clean

The most cleaning, for messy files. Repairs broken characters, and removes scanning debris, symbol lines, page markers and more.

Code / Source Files

Source code, cut by length with indentation intact. Removes very little.

No Chunking

Each file becomes one piece, which suits short documents. The cleaning still runs.

Large Files

Very large documents: bigger pieces and a longer time limit per file. Removes the same as Default.

Fast

The file as it arrived: no splitting and no filtering. New files wait in Clean until you press Clean.

Settings

Essentials gathers the most-used settings from the groups below. How to change and save settings is in Preview and clean; Recipes, History and Appearance are in Projects and settings.

Splitting

How files are cut into pieces.

Split files into pieces

Off, each file is one piece.

How to split

Words, Sentences, Characters, Paragraphs, Sections, Tokens (estimated) or Do not split. Sections cuts at headings, and is the only one that keeps tables and code blocks whole.

Piece size

How much goes in a piece, in the unit of the split. Too small splits a thought across pieces; too large may not fit what a model can read at once.

Overlap

How much of the end of a piece is repeated at the start of the next, so a thought cut in half still reads whole in one of them.

Start each piece with its headings

Sections only. Each piece opens with the headings it sits under, so it still makes sense when found on its own.

Do not cut inside a sentence, Do not cut inside a word

Moves each cut to the nearest sentence end, or the nearest space.

Drop pieces smaller than

Removes pieces with fewer words than this after cutting.

Read at most

Stops reading a file after this many characters; 0 reads it all. Sections always reads the whole file.

Quality

Which pieces are good enough to keep.

Filter out low-quality pieces

Off, no piece is removed for its length or its score. Repeats are still removed while Remove duplicates is on.

Quality bar

Every piece gets a score from 0 to 1. Pieces below the bar are removed. Higher is stricter.

Drop pieces shorter than, Drop pieces longer than

Word limits for a piece. 0 means no limit.

Judge the writing only in pieces of at least

Shorter pieces, such as headings and table rows, are judged on length and repetition only.

Remove duplicates

Removes pieces that repeat, exactly or nearly, text kept earlier in the file. This includes a clause repeated on purpose, such as a standard disclaimer.

Clean-up

Rules that remove clutter before the text is cut into pieces. Run the clean-up rules turns this group, and Privacy, on or off.

Privacy

Replaces details with a placeholder before the text is cut into pieces: Hide email addresses (<EMAIL>), Hide phone numbers (<PHONE>), Hide API keys and secrets (<API_KEY>) and Hide card numbers (<CC_NUMBER>).

Text

Small changes so the same thing is always written the same way. Tidy the text turns the group on or off.

Characters

Unicode form (NFKC is recommended), Use straight quotes, Use plain hyphens, Use plain punctuation, Use ordinary spaces, Remove invisible characters, Decode escaped characters, Remove accents (loses information) and Make everything lowercase (names and acronyms lose their capitals).

Spacing

Collapse spaces, Remove spaces at line ends, and Keep line breaks (off puts the text on one line).

Placeholders

Replace web addresses (<URL>), Replace email addresses (<EMAIL>) and Replace long numbers (<NUM>).

Limits

Cut each page or record at: longer pages, slides or records are cut short before splitting. 0 means no limit.

Reading files

How text comes out of each kind of file, before cleaning. Each setting is described in Supported files.

Output

Output format

JSONL (recommended), JSON, Plain text or CSV. Keep JSONL: exports only include files cleaned to JSONL, and Review cannot read Plain text or CSV.

On each piece

Source file details, Quality score, Token count, and Token ids (much larger files; only useful if you train with the same tokenizer).

Counting tokens

With Count real tokens off, tokens are estimated as a quarter of the characters. Tokenizer and Encoding choose what counts them. Cut pieces longer than cuts a piece with more tokens than this.

Standards

A last check each cleaning run must pass before its results can be exported. Turn on Check each run against these standards, then set the minimum pieces, tokens and average quality, and the most that may be removed.

When a run falls short either holds the output back, or keeps it with a warning. Nothing is deleted either way.

Processing

Clean files as soon as they are added

Off, a new file waits in Clean until you press Clean.

Skip a file already in this project

A file with exactly the same contents as one already here is not added twice.

Keep the original file

Off, the file you added is deleted once it is cleaned, and cannot be cleaned again with new settings.

Try again automatically

A run that fails for a passing reason is queued again, up to Tries before giving up (2 unless you change it).

Time limit per file

Bigger files already get more time; this can only raise the limit.

Why text was removed

In Review, on Removed, every piece names the rule that removed it. The first column below is the exact label Review shows. The last is the reason's name in rejected.jsonl, part of the Everything export.

Lines removed while cleaning

These lines are removed before the file is cut into pieces, so they cannot be put back one at a time. To keep lines like them, switch the setting off and clean the file again.

Length, score and repeats

Press Keep this anyway in Review to put one of these back in the export.

Quality rules

These rules judge the shape of a piece. The first four always run while Filter out low-quality pieces is on. The rest come from rule sets published with the Gopher, C4 and FineWeb-Edu datasets, and only judge pieces at least as long as Judge the writing only in pieces of at least.

RAG / Retrieval Corpus and General Documents turn the published rule sets off. RAG / Retrieval Corpus also turns off Too few words, Mostly non-letters and Words too short.

A rule that fires after the second clean-up pass is named rule_after_clean_… in rejected.jsonl. Review shows the same label either way.

To see what turning rules off would give back, see Test rules on a file.