Thoth

Thoth is a proof-of-concept AI interview and retrieval system built for a T-Mobile-sponsored hackathon, designed to capture knowledge from subject-matter experts and make it available through grounded responses.

The project placed first among competing teams on instructor black-box benchmarks with an approximately 75% pass rate.

Capturing institutional knowledge requires more than storing interview transcripts. The write path needed explicit ownership and review controls, while the read path needed to prevent unsupported answers and preserve the lineage of every retrieved claim.

I architected the interview write path and its 3-state approval lifecycle: draft, SME review, and administrator approval. I also designed the retrieval organization, grounding requirements, and tiered model strategy.

After the benchmark win, I took the initiative to form a cross-functional student group with diverse backgrounds, pulling together interested students from formerly competing teams, with the aim of piloting employee-facing feature integrations and handing T-Mobile's engineering team a working baseline. The project concluded in September 2026 at the end of its first phase.

We secured a mentor at T-Mobile to guide the effort, and we pooled our separate internship experiences across the team to structure our work the way a professional engineering team would operate.

Thoth is an orchestrator: it does not answer questions directly. A classifier scores each question and routes it to a subject-scoped agent that retrieves only from approved, SME-authored knowledge. Confidence of 0.7 or higher gets an answer, 0.4 to 0.7 gets a clarifying question, and anything lower escalates to an administrator.

The system uses per-subject ChromaDB collections across six domains with all-MiniLM-L6-v2 embeddings.

Grounded responses require citation lineage containing the SME name, entry ID, and topic. Models are assigned per step: Haiku for the short classify, clarify, and route decisions, Sonnet for answers and interview turns, and Opus for synthesis, which runs once per entry and is not latency-bound. That split keeps P95 latency under 8 seconds.

After the benchmark I wanted to know why an answer failed, not only how often. Every existing test scored the final answer, so a miss could not be traced to retrieval or to generation. I built a retrieval-only evaluation that needs no server, API key or model call: it seeds the knowledge base, runs the pipeline's own search, and scores the result against 119 labelled questions, with half of the new questions held out as a test set. It runs in about 15 seconds and gates every pull request in CI.

The measurements changed the diagnosis. Ranking was sound: on the held-out questions it scored 0.86 MRR, and the right entry reached the model's context 94% of the time. The weak point was the gate that decides whether to answer at all. It relies on a fixed embedding distance, and a quarter of answerable questions fell outside it while a third of out-of-scope questions fell inside.

Two changes came out of that. Search had been embedding each question once per subject collection; embedding it once cut mean retrieval time from 406 ms to 80 ms with identical results. I also swept the index chunk size instead of guessing it, checking each size against a long-document test corpus a teammate had built. Chunks of 150 words kept all 43 of its facts retrievable with half the context of the 300-word setting, and below 150 words coverage collapsed because a long table no longer fit in the 20 chunks the model receives.

The same measurements exposed a loss on the write path. Storing only the LLM-written summary of an expert's material dropped specific facts such as fee tables, dates, and long lists. Indexing the raw source chunks alongside the summaries raised fact recall on long documents from 33% to 100%, from 14 to 43 of 43 gold facts.

Generation is scored separately. RAGAS rates live answers at 0.95 faithfulness. Tracing every query in LangSmith showed where the time went, and routing classification to Haiku and answers to Sonnet, instead of one model for every step, cut p95 latency from 19.3 s to 6.9 s. Responses under 8 seconds went from 50% to 99%, at 59% lower cost.

The smaller chunk size is still a recommendation. It was measured offline and has not yet been confirmed against live model answers.

The central challenge was enforcing a closed-book guarantee so the model could not fill knowledge gaps from its pretrained memory. Approval state, retrieval scope, citations, and model routing all had to reinforce the same grounding contract.

A strong technical result buys attention, but it does not sustain a project on its own. Momentum after a win has to be actively built, which means treating team formation and structure with the same intentionality as the technical design.

I also came away convinced that trust in a system, whether it is a RAG pipeline or something else entirely, comes from every layer reinforcing the same explicit guarantee rather than any single component being clever.