GitLab Well Architected
Cookbook navigation

Compare reviews with and without project instructions

Download example
On this page

What does Duo already catch, and what does it need to know about your project? You'll review an API change twice, add a focused instruction between the runs, then correct the change and check the next review.

The flow guide explains how to choose useful instructions. This exercise uses a small Order API so you can recognize the problem yourself: a new response field is missing from its OpenAPI schema. The sample files are only for following along; your own project keeps its code and tests.

Before you begin

Use a blank private Premium or Ultimate project with a README, working runners, and Code Review Flow enabled as described in the setup steps. You'll need permission to commit files to its default branch and create merge requests. Leave automatic reviews off so each request starts when you're ready. Use the same account to request each review and check that each one creates a review session; Enterprise seat holders need the group routing choice described in the setup.

You'll do the comparison in GitLab. cases.yml is a checklist, not a command to run. If you'd like to keep the results together, create an issue and copy the evaluation record into its description.

1. Start with a passing project

Open the Web IDE and copy these files to the default branch, preserving their paths. Each link opens a copyable file without leaving this page.

Commit the files and wait for the test job to pass. It runs the Python unit tests without installing dependencies:

python3 -m unittest discover -s tests

Open openapi/orders.yml. The Order response declares order_id and total_cents, with additionalProperties: false. That last setting is important: adding another response field without updating the schema makes the implementation and schema disagree.

2. Review the change without an instruction

Create example/response-before-instructions from the default branch. Replace these two files, commit them, and open a merge request:

The serializer now returns resource_type: "order", and the test expects it. The pipeline should pass: the Python test agrees with the new response, but it doesn't check OpenAPI.

Post this comment on the merge request:

/assign_reviewer @GitLabDuo

When the review finishes, read the summary and the comment on the new field. Our review identified the mismatch and suggested updating the schema or leaving out the field:

Duo identifies the response field missing from OpenAPI before any project-specific review instruction is added.

Record what your review found. A passing pipeline doesn't settle this question, and a comment that only asks for more tests hasn't explained the missing schema entry. Leave the merge request open and unmerged for comparison.

3. Add the team's context and repeat

Return to the default branch. Copy the review instructions to .gitlab/duo/mr-review-instructions.yaml and commit it there.

The example convention says that the team's clients use the OpenAPI schema. It asks the reviewer to compare changed response fields with that schema and the tests. The practice project doesn't include generated clients; you can check the schema mismatch directly without building an SDK.

Create example/response-with-instructions from this updated default branch. Apply the same two changed files from step 2, open a second merge request, wait for CI, and request Duo again with the same quick action.

Duo's second review names Order response compatibility and explains the same missing field using the project instructions.

Compare both comments. In our runs, the instruction made the explanation more specific to the team's clients, but both reviews found the problem. The severity changed from Major to Critical; that isn't another defect or proof of a better review. Decide whether the additional explanation is useful, and check claims about your clients against how they actually work.

4. Fix the schema and request a fresh review

On the second merge request's branch, replace openapi/orders.yml with the corrected schema. It declares resource_type as a required string with the value order. Commit it and wait for the new pipeline to pass.

Post this as a new top-level comment:

@GitLabDuo please review the latest changes.

Our fresh review recognizes the correction and finds no further issue:

After a fresh review request, Duo explains that the schema now matches the serializer and test.

Check the explanation against all three files: implementation, test, and schema. The original discussion may still be open even though the new review recognizes the fix. Resolve it after checking the change, just as you would a teammate's comment. Neither a clean review nor a passing pipeline merges the request.

Use the comparison on your own code

Pick a convention your reviewers spend time explaining. Start with Duo's existing feedback, add the missing context, and repeat the same change. Then try a corrected version. You're looking for helpful comments on real problems and a review that recognizes when those problems have been addressed.

Keep instructions that improve the explanation or catch something important. If they only repeat general advice or complain about harmless edits, shorten or revise them and compare again. Expected outputs summarizes what to look for without requiring the same wording or severity each time.

Walkthrough files

Download example
Files18
  • evals18
    • fixtures15
      • changes3
        • documented-field1
          • openapi1
        • undocumented-field2
          • src1
            • orders1
          • tests1
      • project11
        • openapi1
        • src5
          • orders4
        • tests3
    • templates1
evals/fixtures/project/README.mdmarkdown · 10 lines
# Order API

Small order response, database query, and shipping client examples. The public response is described in `openapi/orders.yml`. The unit tests check the Python behavior; they don't validate that schema.

Run the tests with:

```shell
python3 -m unittest discover -s tests
```

Source file

File