How TokenCompressor works
A lightweight, deterministic pre-processor that shortens code and technical prompts without breaking their meaning — so your LLM bill drops on every API call.
1. Compress
Strip comments, normalize whitespace, apply a domain dictionary, and rename long repeated identifiers.
2. Decompress in the LLM
Send the one-time Context Prompt so the model expands abbreviations and uses the original identifier names in its response.
Techniques
| Technique | What it does | Avg savings | Reversible? |
|---|---|---|---|
| Strip comments | Removes //, /* */, #, <!-- --> | 5–15% | No |
| Normalize whitespace | Collapses spaces, trims lines | 8–20% | No |
| Dictionary substitution | Replaces common terms with short codes | 10–25% | Yes (via context prompt) |
| Identifier renaming | Replaces long repeated names with _a, _b… | 5–15% | Yes (via #MAP header) |
Quick start
- Open the Context Prompt page and copy the system prompt for your model.
- Send it as the first / system message of your LLM session.
- From now on, paste your code into the Compressor, copy the compressed output, and send that.
- The model decodes it transparently and replies in normal code.
FAQ
Does it work with any LLM?
Yes — any frontier model (Claude, GPT-4, Gemini, Llama) can decode the dictionary once you give it the Context Prompt. Smaller models may need a few examples first.
Is the compression lossy?
Semantically lossless: comments are removed and identifiers renamed via a #MAP table the model uses to recover the original names in its answer.
Where is my data stored?
Nowhere. Everything runs in your browser. History is kept only in localStorage.
How accurate is the token count?
We use the cl100k_base BPE tokenizer (same as GPT-4 and Claude 3+) running locally in your browser via js-tiktoken. Results match the real billable tokens within ±1%. The chars/4 estimator is only used as an instant fallback if the tokenizer fails to load.
What is Playground mode?
An advanced view where you toggle each compression technique individually, see the per-technique token impact in real time, and add your own substitution rules. Open it at /playground.
Can I add my own compression rules?
Yes — in Playground you can add unlimited 'long term → short alias' rules. They are saved in your browser and automatically appended to the Context Prompt so the LLM can decode them.