Self-serve experiment analysis at Uber
I built a Python and LangChain agent that helps product managers and scientists explore A/B experiment results by driver and courier traits.
Conceptual diagram; implementation details are simplified for clarity.
The problem
Understanding experiment results by driver and courier traits depended on a manual workflow across teams. Product managers and scientists needed a faster way to ask focused questions and get useful, trustworthy slices of experiment data.
What I owned
I built a Python agent with LangChain to make experiment analysis more self-serve. The experience supports report filtering and sorting, AI-generated summaries, and conversational follow-up questions.
The technical approach
The key design choice was to retrieve relevant schema metadata and business context before generating SQL. Grounding the model in the available data gives it a concrete vocabulary for the query instead of asking it to guess table and field names.
The resulting query feeds a report that people can filter and sort. A summary and follow-up conversation help users move from a broad question to a more specific view of the experiment.
- Retrieve schema and business context first.
- Generate a query against the available experiment data.
- Present results in a filterable, sortable report with summaries and follow-ups.
What changed
The workflow moved toward self-serve reporting: product managers and scientists could explore experiment segments without relying on the same manual, cross-team handoff for every question.