LongLLMLingua

Naive perplexity compression is question-agnostic — it keeps the noise. LongLLMLingua makes compression question-aware, selecting tokens by question-conditioned contrastive perplexity and reordering documents by relevance to fight lost-in-the-middle.

From the LongLLMLingua paper (Jiang et al., Microsoft, 2023, arXiv:2310.06839v2, “LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression”). It represents the hard-prompt branch (token-level pruning), complementing this section’s Compaction — one is a research-grade token mechanism, the other a harness-level operation.

Method

LongLLMLingua builds on LLMLingua, so start with the base. LLMLingua uses a small model M_S to score each token’s perplexity and drops the tokens with low perplexity — low perplexity means “least surprising,” so removing them barely changes the model’s overall entropy gain. This is a direct application of the “LM is compression” idea.

The flaw: this pure perplexity / information-entropy compression is question-agnostic. In long contexts the question-relevant information is sparse and dynamically located, so pure perplexity retains a lot of noise — on NaturalQuestions the paper measures it performing worse than zero-shot. LongLLMLingua’s thesis is therefore one line: compression must be question-aware. It contributes four mechanisms plus a recovery strategy.

1. Question-aware coarse-grained compression (document level). Score each document x^doc_k with an importance r_k and keep only the high-r_k ones. The trick is the conditioning direction: not “perplexity of the document given the question” (documents carry too much irrelevant text to be distinct), but perplexity of the question given the document:

r_k = -(1/Nc) · Σ_i log p( x^que,restrict_i | x^doc_k )

with a restrictive statement (“We can get the answer to this question in the given documents”) appended after the question as a regularizer against hallucination. The paper shows r_k’s recall beats BM25 / OpenAI-embedding / Cohere-Rerank and other retrievers.

2. Question-aware fine-grained compression (token level). When token-pruning the retained documents, measure each token’s relevance to the question by contrastive perplexity:

s_i = perplexity(x_i | x_<i) − perplexity(x_i | x_que, x_<i)

i.e. the shift in perplexity caused by conditioning on the question. The paper proves this is equivalent to conditional pointwise mutual information (PMI). Plain perplexity looks nearly random; high-s_i tokens instead cluster around the answer-bearing document — contrastive perplexity develops “relevant to the question” out of the noise.

3. Dynamic compression ratio. LLMLingua applies one ratio to all documents; LongLLMLingua uses the coarse-grained r_k ranking to allocate budget dynamically — more relevant documents get a lower compression ratio (keep more), via a linear scheduler — bridging the coarse and fine stages.

4. Document reordering. After coarse compression, reorder documents by r_k, putting the most relevant at the front. This targets position bias (the “lost in the middle” effect) directly — the same information is used far less when it lands in the middle of a long prompt.

5. Subsequence recovery. Token-level deletion mangles entities (names, places, numbers) — models tend to copy entities from the prompt, and a mangled entity produces errors. The recovery step does longest-common-subsequence matching across response / compressed prompt / original prompt (accelerated with prefix trees / sequence automata) and swaps mangled entities in the response back to the original’s full spans (paper Algorithm 1).

Paper and Results

  • Setup: target LLMs GPT-3.5-Turbo-0613 and LongChat-13B-16k; the small compression model is LLaMA-2-7B-Chat; datasets NaturalQuestions (multi-doc QA), LongBench, ZeroSCROLLS, MuSiQue (multi-hop), LooGLE (long-dependency).
  • Baselines: retrieval (BM25, Gzip, SentenceBERT, OpenAI-embedding) plus compression (Selective Context, LLMLingua).
  • Results: on NaturalQuestions (answer document at the 10th position), +21.4% performance at ~4× fewer tokens; 94% cost reduction on LooGLE; 1.4×-2.6× end-to-end latency speedup on ~10k-token prompts at 2×-6× compression.
  • The instructive comparisons:
    • Compression baselines (Selective Context, LLMLingua) do poorly on noise-heavy tasks because they are question-agnostic entropy compression that keeps the noise — sometimes below zero-shot.
    • Retrieval methods are fine at low ratios but degrade as the ratio rises (2×→4×) because recall drops.
    • LongLLMLingua is the most robust across tasks and ratios, even gaining slightly at higher ratios — question-aware compression reaches higher key-information density precisely when the ratio is aggressive.
    • Reordering helps every method, not just this one — independent evidence that position bias is real.

Limitations

  • Assumes a well-defined question. The whole apparatus conditions on x_que, so it fits QA / retrieval-shaped tasks best; in multi-turn agent loops “the current question” is not always clear.
  • Needs a small model to compute perplexity — cheaper than the target LLM but not free (extra forward passes).
  • Output is degraded natural language: token deletion sacrifices fluency and breaks entities, requiring subsequence recovery as a patch; the compression ratio has a ceiling (the paper’s sweet spot is 2×-6×).

What It Means for Our Compaction

This paper does not describe our mechanism (we don’t do token-perplexity pruning), but several principles transfer:

  1. Question-awareness is a lever — but mind which tier it lands on. Our compaction is already partly goal-aware at the summary tier: the FLOOR summary is a rolling task checkpoint that always preserves the user’s requests (P0) and the goal, demoting rather than deleting. What is genuinely question-agnostic is the other tier — tool-result reduction, which truncates by a fixed keep-head/keep-tail heuristic, not by relevance to the current task. That is exactly where LongLLMLingua applies: rank tool outputs by relevance to the active goal and trim accordingly, instead of uniform truncation — which is also what Compaction’s preservation policy and Anthropic’s “recall-first, precision-second” recipe aim at.
  2. Ordering is a second lever beyond “what to keep.” Our preservation policy says what survives but not where the survivors sit. This paper shows position drives utilization. Takeaway: after compaction, put the most relevant content at the ends of the window (especially the front), not buried in the middle.
  3. Protect the exact tokens. The subsequence-recovery lesson: any lossy compression destroys IDs, paths, numbers, error codes — things that must be verbatim and can’t be entrusted to a summarizer. This maps directly onto our keep-head/keep-tail tool truncation and keep-the-path compact references — treat exact spans separately from summarizable narrative.
  4. It also backs a cost choice we make: a cheap small model is enough to locate the key information, so compacting with a Haiku-grade model is theoretically sound.
  • Compaction — the section’s practical spine; LongLLMLingua is the research backing for the hard-prompt branch of its compression spectrum.
  • Compaction → The Research Lineage — the other branch, soft prompt (ICAE → 500xCompressor, encoding text into KV values the model reads directly), is summarized there.

Sources

Was this page helpful?
Edit on GitHub