When self-hosting beats the API: a fine-tuned internal assistant

Challenge

A client needed an internal Q&A assistant over a proprietary knowledge base, but data-residency requirements and per-query cost at their volume ruled out sending the corpus and traffic to a hosted API.

Approach

We fine-tuned LLaMA 2 13B on the internal knowledge base with LoRA/PEFT — adapting the model on consumer-scale hardware without touching the full weights — and served it with vLLM for high-throughput inference under concurrent load. The whole system runs inside the client's own environment, which is what made self-hosting the right call over an API for this workload.

Architecture

Results

• —Answers grounded in the client's own knowledge base, with the data never leaving their infrastructure
• —vLLM serving sustains high-throughput inference under concurrent use
• —Lower marginal cost per query than a hosted API at the client's volume