Implementing a retrospective agent: how it works

In my previous post I described why we're trying to measure where engineering time is spent, and how a small LLM agent could help integrate that measurement into our teams' sprint retrospectives. A few months later, we've rolled out the agent across our teams and it has helped spark discussions that weren't happening before.

This post focuses on how the agent works, how it receives data, and how the prompts evolved. The data platform underneath, including how the DORA metrics are derived, gets its own post next.

Overview

The agent retrieves metrics and activity data from our productivity metrics data platform and generates a pre-read retrospective report to summarise the sprint and suggest potential discussion points.

Rather than flooding the model with all the raw data, the agent is given a summary of metrics across three sprints and tools to look up details on specific tickets, merge requests and failed releases.

Agent design

The initial prompt

The initial prompt summarises metrics across the three sprints, tagging each as [BASELINE] or [LATEST - FOCUS SPRINT]. The metrics cover Jira delivery and DORA metrics:

All of the metrics are computed in the data platform with deterministic code, we don't ask the model to attempt to calculate them from raw data. The prompt is explicit about this: "the metrics summary is authoritative, do not attempt to recalculate or second-guess it".

The summary also includes a list of tickets and merge requests that are outliers (e.g. above the 75th percentile for cycle time or lead time), and a list of any failed releases in the focus sprint. The agent is instructed to use these lists as starting points for its analysis. Tickets are named inline as KEY (16.8d, 3.0SP), and MRs as project!iid (308.7h: coding 116.3h, review 170.0h, deploy 22.4h).

The lookup tools

Three tools are registered on the PydanticAI agent, allowing the agent to look up details:

@agent.tool
def lookup_jira_ticket(ctx: RunContext[AgentDeps], ticket_key: str) -> str:
    """Look up a Jira ticket by its key (e.g. PROJ-1363).
 
    Returns summary, status, assignee, story points, sprint, dates, cycle time,
    and full status transition history.
    """
 
@agent.tool
def lookup_merge_request(ctx: RunContext[AgentDeps], query: str) -> str:
    """Search merge requests by title substring, Jira key, or MR reference (e.g. !1049).
 
    Returns up to 10 matching MRs with title, author, dates, lead time, and release info.
    """
 
@agent.tool
def lookup_failure(ctx: RunContext[AgentDeps], query: str) -> str:
    """Search detected failed releases by tag, project name, evidence, or alert tiny id.
 
    Returns up to 10 matching failures with signal (alert/revert), evidence,
    resolution time, and correlated alert detail where available.
    """

The docstrings double as the schema the model reasons over, and the return payloads are how the agent begins to build a narrative. A ticket comes back with its full status_history as a readable chain (e.g. "In Progress → Code Review → Awaiting Release → Done"), helping to identify tickets that flapped in and out of "Blocked". An MR has its coding/review/deploy time split and the release that shipped it, helping to identify common patterns between MRs and their impact on the release cycle.

The system prompt encourages the agent to use these tools: "use these tools proactively ... look it up before drawing conclusions".

The output structure

The agent's output type is a plain string - no structured output schema, no post-processing. The only thing shaping the report is the prompt's task definition: a concise summary of the latest sprint, 2-4 notable trends compared to the baselines ("both positive and negative"), and 3-5 concrete talking points or discussion questions, under three required headings: Sprint Summary, Notable Trends, Retrospective Talking Points.

Specifying 2-4 trends and 3-5 talking points forces selectivity, and framing the third section as questions rather than conclusions helps keep the report a pre-read instead of a verdict, trusting that teams have the full context to interpret the trends and decide what to do about them.

A typical run

Here's an illustration of a typical agent run. The summary named 11 tickets and MRs, and the agent made 11 lookups in parallel to get the details it needed. The report was generated in a single LLM call, and the output was a three-section markdown report.

The lookups gave more detail to the report. The worst cycle-time outlier (16.8 days) turned out to have spanned two sprints, and its merge request took 308.7 hours from first commit to release (split as 116.3 hours coding, 170.0 hours review, 22.4 hours deploy). Another MR spent 99.5 of its 99.7 hours in review.

The report identified the bottleneck was review latency (median 58 hours, roughly three times the previous sprint), not deployment, and was able to point to specific tickets/MRs to discuss. It also flagged that releases had dropped from 23 to 6 while the change failure rate improved from 8.7% to zero, and asked whether the slower flow was deliberate risk management or just changes queueing behind review.

The whole run was two LLM calls: roughly 11,400 input tokens (a third of them cache reads on the second call) and 2,200 output. At GPT-5.4's API pricing at the time of writing that's about $0.05, most of it the output tokens.

Prompt engineering in practice

Don't name engineers

In the initial prototype the agent would often call out engineers by name when discussing slower tickets. In the interest of preserving a blame-free retrospective environment, I added a guardrail to the prompt to avoid this. The agent was instructed to focus on team-wide trends and the work, rather than individual performance.

This was also something I could cover with a deterministic eval. The eval extracts every assignee and MR author from the input and fails if any appear in the output. The prompt plus the eval together enforce the rule.

The schema is part of the prompt

The agent reasons over your field names, so naming is important. An early version would misidentify a ticket's status and whether it was done or not based on the Jira status_category field. Adding an is_done boolean to the ticket schema fixed that, using deterministic logic and the data structure to encode the semantics rather than relying on the model to interpret an ambiguous field name or value.

Structure beats instructions

The prompt never uses the word "hypothesis", even though hypotheses are the product. The third section is talking points and discussion questions, not conclusions. Retros should be human-led, and the report format encodes that rather than asking the model to remember it.

The original prompt also asked for a glanceable summary table of the key metrics, and the model would produce a different formatted summary each time. By building a web UI around the agent the summary table became a deterministic component next to the generated narrative. If part of the output should be the same every time, don't ask an LLM to generate it.

Conclusion

The retrospective agent is a small but useful tool for our teams, helping to spark discussion and reflection within our engineering retrospectives. It demonstrates how prompt engineering and use of lookup tools can create a valuable AI assistant that respects team dynamics and focuses on actionable insights.

A big part of its usefulness is the metrics data platform underneath, which provides the authoritative data the agent reasons over. That requires careful choices about what metrics to track and how to calculate them from your development workflows. In the next post, I'll dive into how that platform works, how we derive the DORA metrics, and how it all ties together to support our engineering teams.