GitLab Well Architected
Cookbook navigation

Example: interpret a throughput increase during a Duo trial

Your trial participants are merging more MRs, and you want to understand what changed. Comparing them with other contributors helps put the increase in context. You'll also need to investigate whether Duo contributed to it and whether the extra activity delivered better results.

Start with the scope of the counts

When using the original prompt, supply your top-level group, participant usernames, and two equal-length periods. Before interpreting the answer, check what its queries counted.

The prompt first asks for merged MRs, then asks for both created and merged rates. If the query counts creation dates within that merged set, it will leave out MRs that remain open or were closed without merging. To include all creation activity, ask for a separate count:

For MRs created per week, count all MRs created within each period, regardless of
their current state. For MRs merged per week, count MRs merged within each period,
even if they were created earlier. Show those query scopes separately.

Keep the same participant list in both periods. Otherwise, a change in who is counted can look like a change in throughput.

Work through a small comparison

These invented totals let you practice reading the comparison without querying a group. Each period is four complete weeks, and the two counts were collected separately as described above.

Group Baseline created Trial created Baseline merged Trial merged
Trial participants 24 36 20 30
Other contributors 40 60 32 48

You can paste the table into your conversation and ask Duo to calculate the weekly rates and percentage changes. These are the results, shown as baseline → trial:

Group Created/week Merged/week Change
Trial participants 6 → 9 5 → 7.5 +50%
Other contributors 10 → 15 8 → 12 +50%

Both groups increased by the same percentage, so this comparison leaves Duo's contribution unresolved. A busier delivery period, smaller MRs, staffing changes, or a different mix of work could help explain what you're seeing. You'd need more information to judge whether Duo affected the result.

Ask what the numbers don't answer

Try a follow-up:

Both groups increased by 50%. What additional information would help us decide
whether the trial improved delivery, rather than simply increasing MR activity?

Look for questions about the result you wanted from the trial: whether a customer problem was resolved sooner, whether quality held up, and how much review or rework the changes required. Those questions need information beyond MR counts. To assess return on investment, you'd also need to compare the benefit with the full cost of the approach.

For an actual group, the Data Analyst agent guide explains how to start an analysis. GitLab Query Language explains the query format so you can check what the queries counted alongside the summary.

Back to the prompt