Offline Notebook LM
Ask questions about your own documents and get answers, entirely offline. No cloud, no data leaving your machine.
Why I Built This
Most AI assistants need an internet connection and send your data to someone else's servers. That's a dealbreaker for anyone working with sensitive documents or in air-gapped environments. I wanted something where you could drop in your files, ask questions, and get answers grounded in your own documents — without anything leaving your machine.
How It Works
- A routing agent classifies each query and decides whether to search at the summary level or chunk level across a 2-stage retrieval pipeline
- LLM backend selection benchmarks models on domain-specific queries and uses knowledge distillation to compress larger model outputs into efficient Phi-3 and Mistral inference
- Document ingestion supports 7+ file types with adaptive chunking, sentence-transformers embeddings, and ChromaDB vector storage
Built with Electron, React, and TypeScript on the frontend and FastAPI with Python on the backend. Local models run through Ollama and llama.cpp, with Apple Silicon acceleration through llama.cpp and ONNX Runtime. The model management interface supports switching models. The Docker backend has pytest and Ruff checks in CI, and a health/config endpoint supports diagnostics. Documents include PDF, DOCX, Markdown, and plain text; answers cite their sources.
Results
- 2-3x faster query times compared to naive retrieval, with lower memory overhead
- Approximately 100 to 200 chunks/second during embedding generation
- A 10-page PDF ingests in under 12 seconds; ChromaDB search takes under 50 ms for collections of up to 10,000 chunks
- Everything runs locally, with no telemetry or cloud dependency for any feature.