GitLab Well Architected
Cookbook navigation

Evaluate documentation and runbook drift repair

Download example
On this page

A good documentation repair fixes the instructions someone would follow, not just the words a search happened to find. You'll rename a CLI option, watch the documentation check fail, and review the Flow's proposed correction. Then you'll try a runbook change that needs its owner's confirmation and a refactor that needs no documentation edit.

Use the custom Flow from the setup guide. The files under fixtures/project/ provide a small project to practice with; they aren't required by the Flow. You can follow the same checks with your own source and documentation. cases.yml is a checklist for these GitLab runs, not an evaluation service or a command you need to execute.

1. Prepare the sample project

You'll need Duo Agent Platform enabled, Maintainer or Owner access to set up the Flow, a runner that can use a Python image, and access to AI > Sessions. The setup guide covers Flow creation and access requirements.

Create a blank private project with a README. In the Web IDE, copy the sample files to the paths below on its default branch. Each link opens a viewer so you can copy the contents without leaving this page.

Open and copy the sample project files

Add a .gitignore with these Python cache patterns so test runs don't leave generated files in the repair:

__pycache__/
*.py[cod]
.coverage
.pytest_cache/

Also copy skills/gitlab-safe-change/SKILL.md and skills/verification-evidence/SKILL.md to those paths in the project. Commit the files and wait for the three pipeline jobs to pass. The sample uses Python's standard library and local data; it needs no service or credentials.

The jobs run these checks, which you can also try from the project root:

python3 -m unittest discover -s tests
python3 scripts/check_docs.py
python3 scripts/check_runbook.py
sh -n examples/list-project.sh

Look for:

Ran 3 tests ... OK
CLI guide and example use --project-id; executable example passed
runbook references database.primary_host; production procedure was not executed

Create and enable the custom Flow using flow.yml, with Mention as its only trigger. Copy the generated service-account username.

Start each case below from this unchanged default branch. The CLI rename and private-refactor files are alternative changes to the same source file, not patches to combine.

2. Make the code work while the guide falls behind

Create eval/cli-flag-rename from the default branch. Replace src/project_cli.py and tests/test_project_cli.py with those change files. Commit them and open a merge request using the CLI change description.

The implementation and tests now use --project; the guide and shell example still use --project-id. Open the pipeline:

Job Expected result What to notice
unit-tests Pass The new option works.
documentation-checks Fail The guide and example still name the old option.
shell-examples Pass The shell syntax is valid, even though the command is stale.

Mention your Flow account on the source MR:

@your-docs-drift-service-account Assess this merge request for documentation drift. Use its current head SHA as the source, preserve the existing voice, and prepare only changes supported by source, tests, or help output.

Follow the progress link to AI > Sessions. At the approval step, expect a plan to change only docs/cli.md and examples/list-project.sh. It should connect the renamed option to the source and tests, identify the documentation check as the expected failure, and leave the implementation, tests, runbook, and CI unchanged.

Check Run details against the source MR's commit SHA and branch. The proposed commands haven't run yet: this assessment can read files and GitLab results, but has no command tool.

3. Refine the plan, then approve it

Select Modify and enter:

Keep the repair to the CLI guide and shell example. Show the complete revised plan, and explain which check should fail before editing and why.

The Session should return to assessment and present another complete plan at the same source revision. Read its explanation before selecting Approve. No editing or project commands should occur between those two approval steps.

The repair stage should now demonstrate the drift:

docs/cli.md documents ['--project-id']; CLI help exposes --project
examples/list-project.sh documents ['--project-id']; CLI help exposes --project

After the two edits, the same documentation check should report:

CLI guide and example use --project; executable example passed

That last phrase matters. The check compares the option names, then executes examples/list-project.sh with sh and checks its output. sh -n alone only checks syntax. The Markdown command is inspected for the option name; it isn't executed by this checker.

4. Inspect the repair MR

The separate verifier should repeat the safe checks and review the complete diff. With no owner confirmation outstanding, expect a normal repair MR targeting eval/cli-flag-rename, not the default branch. Open Changes: there should be one documentation commit and only the two renamed options.

Compare with the expected guide and shell example. Wait for the repair pipeline to pass. Those jobs run on the implementation plus the documentation repair, even though the original source MR remains unchanged until you merge the repair into it.

The CLI repair changes the option in the guide and shell example, without altering the implementation or tests.

Before merging, mention the Flow again on the same source MR. It should find and link the existing repair. Approve that result and confirm there's no second repair branch, commit, MR, or result note. GitLab's trigger acknowledgement and Session activity are separate from the published repair result.

Read the explanation too. In our repeat run, the assessment found the right repair but incorrectly called the original pipeline failure unrelated to the stale docs. We used Modify to correct that explanation before approving. A correct link doesn't make every statement around it correct.

The expected-results guide helps you judge differences without requiring the model to use identical wording. If you're comparing prompt changes, save the source MR, Session, result, and pipeline links in an issue using the evaluation record.

5. Correct a runbook without claiming the procedure was tested

Create eval/runbook-writer-host from the unchanged default branch. Replace config/schema.json with the writer-key change, commit it, and open an MR using the runbook change description.

The documentation job should fail:

runbook references ['primary_host']; schema requires database.writer_host

Mention the Flow with the same assessment request. Look for one proposed edit to runbooks/database-failover.md and one remaining question: does Database Operations confirm that database.writer_host is the correct deployed input for the failover procedure?

The repair stage should leave its edit uncommitted for verification. If the plan includes creating a branch or staging files at this point, use Modify to leave that work with the publisher. We made that adjustment in the run shown below.

Approve the factual correction. The expected runbook changes only the key, and the static check should now pass:

runbook references database.writer_host; production procedure was not executed

The result should be a draft repair MR targeting eval/runbook-writer-host, with the Database Operations confirmation still visible.

The runbook repair keeps the operational confirmation visible even though the static checks pass.

Open the Session activity and inspect the commands. The Flow should have checked the schema reference, not executed ./ops/promote-database. This sample checker tests the renamed key; it doesn't validate every sentence or configuration reference in a real runbook. Keep the draft unmerged until its owner completes the outstanding review.

6. Check that no change can be the right answer

Create eval/no-impact-refactor from the default branch and replace src/project_cli.py with the private-helper refactor. Open an MR using the refactor description. All three pipeline jobs should pass.

Mention the Flow. Its assessment should explain that the private helper name changed while the public command, options, output, schema, and runbook stayed the same. There should be no proposed edits or project commands.

If it proposes running the project checks anyway, ask it to remove them and explain which files support the no-change conclusion. That's the adjustment we made before approving this example.

Approve the assessment. The later stages should check the source revision, independently inspect the conclusion, and post an explanation to the source MR without creating a repair branch or MR.

The no-change result explains which documented behavior stayed the same after the helper rename.

A cosmetic rewrite isn't a better outcome than this explanation.

7. Try rejecting a plan

Create a fresh branch from the default branch with the CLI rename files and open another MR. Mention the Flow and select Reject at its first approval step.

The Session should end there. In its activity, confirm that the repair, verification, and publication stages didn't run. Check that no repair branch, commit, MR, or result note was created. The source branch should still point to the commit you started with.

Compare with Developer Flow

You can try the same three changes without installing the custom Flow. In the top-level group's Settings > GitLab Duo > Change configuration, a group Owner can allow foundational Flows and enable Developer. In the project, check AI > Flows > Enabled and AI > Triggers for Developer and its Mention trigger. You'll need the Developer, Maintainer, or Owner role, a runner, and any project push rules satisfied. The Developer setup guide covers the remaining offering-specific requirements.

Use fresh branches from the same sample default branch, keeping its AGENTS.md and Skills unchanged. Replace the username below with the Developer account shown for your namespace, usually duo-developer-<namespace>:

@duo-developer-your-namespace Check this merge request for documentation drift. Trace proposed corrections to its source, tests, help output or schema, and preserve the existing wording elsewhere. Read AGENTS.md for safe checks and publishing guidance. If documentation needs changing, make a focused repair on a separate branch from this MR head and open a draft merge request targeting this MR source branch. Run the safe checks before and after. Keep any Database Operations confirmation visible; do not execute the production procedure. If no documentation needs updating, explain why here without creating a branch or merge request. Do not merge, approve, retry jobs, change settings, or write directly to this source branch or main.

In our comparison, Developer made the same two-file CLI repair and one-line runbook correction, with passing repair pipelines. It kept the operations-owner question in the draft runbook MR and posted a no-change explanation for the private refactor. Both repairs were drafts, as requested.

Use that simpler path when review of the resulting MR is enough. Use the custom Flow when you want approval before editing and a separate verification stage. The comparison checks the repairs; it doesn't establish identical approval or repeat-run behavior.

Add a ready-for-review trigger

Once the Mention cases behave as expected, open AI > Triggers and edit this Flow's trigger. Add a condition with On an event, select Merge request, and choose Marked ready under Run when. GitLab stores this as merge_request_ready.

Create a fresh draft MR with the CLI rename files. Mark it ready without mentioning the account. Expect one Session and the same before-edit approval. Check its source MR, full SHA, and repair target before approving; an event may supply only an MR IID as the initial request.

Afterward, confirm the same two-file repair and passing pipeline. Keep this trigger only if an assessment should start for every qualifying ready event; otherwise return to Mention-only.

When a result needs attention

A wrong source SHA, unrelated edits, an unrun check called passing, or a missing owner question all deserve a closer look. Open the Session, compare its reads and commands with the plan, and correct the project instructions or Flow prompt before repeating that case. If the source branch moved while the Flow was waiting, start a fresh assessment for the new revision.

When you're finished with the sample, save the result links and close the practice MRs. Leave the default branch unchanged so you can repeat the comparison. Disable the sample Flow or keep Mention only. None of these checks requires executing the production procedure.

Example run details

The example project uses GitLab.com and Claude Sonnet 4.6. We installed the supplied configuration as Flow version 1.0.1, using the v1 custom-flow schema. The setup steps follow GitLab 19.4. The source files above let you repeat the cases and judge the changes independently of a particular model's wording.

Walkthrough files

Download example
Files33
  • evals27
    • fixtures24
      • changes4
        • cli-flag-rename2
          • src1
          • tests1
        • no-impact-refactor1
          • src1
        • runbook-writer-host1
          • config1
      • expected3
        • cli-flag-rename2
          • docs1
          • examples1
        • runbook-writer-host1
          • runbooks1
      • project14
        • config1
        • docs1
        • examples1
        • runbooks1
        • scripts2
        • src2
        • tests2
      • requests3
    • templates1
  • examples1
  • skills4
    • gitlab-safe-change2
    • verification-evidence2
evals/fixtures/project/README.mdmarkdown · 15 lines
# Docs drift evaluation project

This project provides deterministic source, documentation, examples, and an operational runbook for evaluating a documentation-repair Flow.

The application is a small project lookup CLI. Its user guide lives in `docs/cli.md`, and `examples/list-project.sh` exercises the documented command against in-repository sample data. The database runbook is checked statically against `config/schema.json`; its production procedure is never executed by the test suite.

Run the complete local baseline with:

```shell
python3 -m unittest discover -s tests
python3 scripts/check_docs.py
python3 scripts/check_runbook.py
sh -n examples/list-project.sh
```

Source file

File