LLMLingua-2 BERT-base multilingual MeetingBank
current baselineSmall extractive token-classification checkpoint used by the service today.
Prompt Compression Research
Working bibliography for base models, compression papers, checkpoints, and implementation resources used or considered by this project.
Current Decision
- Keep LLMLingua-2 BERT-base as the runtime baseline until another candidate wins on this repo's evals.
- Benchmark LLMLingua-2 XLM-RoBERTa-large as the nearest quality-oriented alternative.
- Use PCToolkit as a benchmark/reference source, not as a production runtime dependency.
- Treat generative or abstractive compressors as offline teachers or challengers until they prove safe for IDs, JSON, schemas, code, and tenant-specific wording.
- Use tenant profiles first, then optional LoRA/adapters; avoid full per-tenant checkpoints by default.
PCToolkit Assessment
- Useful for benchmark shape: unified compressor interface, dataset/task taxonomy, and metric examples.
- Candidate ideas worth borrowing: LLMLingua-2 model switching, Selective Context as a simple extractive baseline, and secondary metrics such as BLEU, ROUGE, BERTScore, edit/fuzzy similarity, QA F1, retrieval scoring, and code similarity.
- Do not import it into the production API directly. The toolkit brings a broad dependency set, multiple model families, hard-coded API-key placeholders in the runner, high thread-count evaluation code, and task-specific prompts.
- Keep this repo's role-aware message handling, protected spans, JSON/TOON handling, and tenant audit logic as the production path.
Runtime Baselines And Checkpoints
LLMLingua-2 XLM-RoBERTa-large MeetingBank
benchmarkLarger LLMLingua-2 checkpoint to compare for quality versus latency and memory cost.
Core LLMLingua Lineage
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
EMNLP 2023Original LLMLingua prompt compression work.
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
ACL 2024Long-context variant focused on improving long-context information processing and RAG-style prompts.
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression
ACL 2024 FindingsExtractive token-classification approach used as this project's baseline.
SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compression
CoLM 2025Security-oriented prompt compression from the LLMLingua research line.
Newer Compression Research To Track
An Empirical Study on Prompt Compression for Large Language Models
2025Comparative study across several prompt-compression methods and datasets.
Prompt Compression in the Wild
2026Latency, rate-adherence, and quality study. Useful for deciding when compression actually pays for itself.
Prompt Compression in Diffusion Large Language Models: Evaluating LLMLingua-2 on LLaDA
2026Evaluates whether LLMLingua-2 compression transfers to diffusion LLMs.
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
2025Attention-aware task-agnostic compression candidate. Relevant if an implementation becomes practical.
SCOPE: A Generative Approach for LLM Prompt Compression
2025Generative chunking-and-summarization compressor. Good challenger or teacher, but risky as default runtime.
Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor
2025Small-LLM compressor trained for compression-rate adherence and downstream quality.
CompactPrompt: A Unified Pipeline for Prompt Data Compression in LLM Workflows
2025Broader prompt and file-level compression pipeline. Useful for structured-data ideas.
Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
2024Sentence-level context-aware compressor with released code and dataset.
ACoRN: Noise-Robust Abstractive Compression in Retrieval-Augmented Language Models
2025RAG-focused abstractive compressor with noise-robust training steps.
Implementation Repositories And Package Resources
Microsoft LLMLingua
current compressorLibrary used by the runtime compressor wrapper.
Toolkit for Prompt Compression
benchmark referenceResearch toolkit linked by the empirical prompt-compression study. Use for benchmark design and candidate wrappers, not as a production runtime dependency.
Hugging Face Transformers
model loadingUnderlying model/tokenizer stack for LLMLingua-2 checkpoints and possible direct adapter work.
Hugging Face PEFT
LoRA candidateLikely adapter/LoRA library if we move beyond rules-only tenant profiles.
PyTorch
ML runtimeTensor and model runtime used through Transformers and LLMLingua.
FastAPI
HTTP APIAPI framework used by `app/main.py`.
Pydantic
schemasRequest and response schema validation.
Uvicorn
ASGI serverLocal and container ASGI server for FastAPI.
Hugging Face Datasets
training dataDataset utilities available for future training and evaluation workflows.
Requests
HTTP clientSimple HTTP client dependency used by scripts and smoke tests.
TOON Format
structured dataCompact structured-data format used by this repo's JSON-to-TOON compression path.