[{"data":1,"prerenderedAt":1162},["ShallowReactive",2],{"/blog/reducing-pager-fatigue-and-improving-on-call-life":3,"navigation-en-us":410,"banner-en-us":848,"footer-en-us":856,"blog-post-authors-en-us-Steve Azzopardi":1103,"blog-related-posts-en-us-reducing-pager-fatigue-and-improving-on-call-life":1117,"blog-share-wrapper-en-us":1142,"next-steps-en-us":1152},{"id":4,"title":5,"authors":6,"body":8,"category":390,"date":391,"description":392,"extension":393,"externalUrl":394,"faq":394,"featured":395,"heroImage":396,"hidePromo":395,"meta":397,"navigation":398,"path":399,"seo":400,"slug":404,"stem":405,"tags":406,"template":408,"updatedDate":394,"__hash__":409},"blogPosts/en-us/blog/reducing-pager-fatigue-and-improving-on-call-life.md","How we improved on-call life by reducing pager noise",[7],"Steve Azzopardi",{"type":9,"value":10,"toc":384},"minimark",[11,28,94,102,109,112,118,127,132,150,153,163,177,183,192,198,201,205,219,244,250,271,274,287,294,327,346,350,353,364,370],[12,13,14,15,21,22,27],"p",{},"To monitor the health of GitLab.com we use multiple\n",[16,17,20],"a",{"href":18,"rel":19},"https://sre.google/workbook/implementing-slos/#what-to-measure-using-slis",[],"SLIs","\nfor each service. We then page the on-call when one of these SLIs is not\nmeeting our internal ",[16,23,26],{"href":24,"rel":25},"https://sre.google/workbook/implementing-slos/#decision-making-using-slos-and-error-budgets",[],"SLOs and burning through the error\nbudget","\nwith the hopes of fixing the problem before too many of our users even notice.",[12,29,30,31,36,37,42,43,51,52,57,58,63,64,69,70,75,76,81,82,87,88,93],{},"All of our services SLIs and SLOs are defined using ",[16,32,35],{"href":33,"rel":34},"https://jsonnet.org/",[],"jsonnet"," in\nwhat we call the ",[16,38,41],{"href":39,"rel":40},"https://gitlab.com/gitlab-com/runbooks/-/tree/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog",[],"metrics-catalog","\nwhere we specify a service and its SLIs/SLOs. For example, the ",[16,44,47],{"href":45,"rel":46},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet",[],[48,49,50],"code",{},"web-pages","\nservice has ",[16,53,56],{"href":54,"rel":55},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L22",[],"an apdex SLO of 99.5%","\nand multiple SLIs such as ",[16,59,62],{"href":60,"rel":61},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L60",[],"loadbalancer",",\n",[16,65,68],{"href":66,"rel":67},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L80",[],"go server",",\nand ",[16,71,74],{"href":72,"rel":73},"https://gitlab.com/gitlab-com/runbooks/-/blob/168397dee0ad955bfb473fd0ddb9146667eeaa13/metrics-catalog/services/web-pages.jsonnet#L113",[],"time to write HTTP headers",".\nHaving these in code we can automatically generate Prometheus ",[16,77,80],{"href":78,"rel":79},"https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/",[],"recording rules","\nand ",[16,83,86],{"href":84,"rel":85},"https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/",[],"alerting rules","\nfollowing ",[16,89,92],{"href":90,"rel":91},"https://sre.google/workbook/alerting-on-slos/#5-multiple-burn-rate-alerts",[],"multiple burn rate alerts",".\nEvery time we start burning through our 30-day error budget for an SLI too fast\nwe page the SRE on-call to investigate and solve the problem.",[12,95,96,97,101],{},"This setup has been working well for us for over two years now, but one big\npain point remained when there was a service-wide degradation. The SRE on-call\nwas getting paged ",[98,99,100],"strong",{},"for every SLI"," associated with a service or its\ndownstream dependencies, meaning they can get up to 10 pages per service since\nthe service has 3-5 SLIs on average and we also have regional and canary SLIs.\nThis gets very distracting, it's stress-inducing, and it also doesn't let the\non-call focus on solving the problem but just acknowledges pages. For example\nbelow we can see the on-call getting paged 11 times in 5 minutes for the same\nservice.",[12,103,104],{},[105,106],"img",{"alt":107,"src":108},"web-pages alert storm","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398914/blog/Content%20Images/reducing-pager-fatigue/web-pages-alert-storm.png",[12,110,111],{},"What is even worse is when we have a site-wide outage, where the on-call can\nend up getting 50+ pages because all services are in a degraded state.",[12,113,114],{},[105,115],{"alt":116,"src":117},"site wide outage alert storm","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398914/blog/Content%20Images/reducing-pager-fatigue/site-wide-outage-alert-storm.png",[12,119,120,121,126],{},"It was a big problem for the quality of life for the on-call and we needed to\nfix this. We started doing some research on how to best solve this problem and\nopened an issue to ",[16,122,125],{"href":123,"rel":124},"https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15721",[],"document all possible\nsolutions",".\nAfter some time we decided to go with grouping alerts by service and\nintroducing service dependencies for alerting/paging.",[128,129,131],"h2",{"id":130},"group-alerts-by-service","Group alerts by service",[12,133,134,135,137,138,143,144,149],{},"The smallest and most effective iteration was to group the alerts by the\nservice. Taking the previous example where the ",[48,136,50],{}," service paged the\non-call 11 times, it should have only paged the on-call once, and shown\nwhich SLIs were affected. We use ",[16,139,142],{"href":140,"rel":141},"https://prometheus.io/docs/alerting/latest/alertmanager/",[],"Alertmanager"," for\nall our alerting logic, and this already had a feature called\n",[16,145,148],{"href":146,"rel":147},"https://prometheus.io/docs/alerting/latest/alertmanager/#grouping",[],"grouping","\nso we could group alerts by labels.",[12,151,152],{},"This is what an alert looks like in our Prometheus setup:",[154,155,161],"pre",{"className":156,"code":158,"language":159,"meta":160},[157],"language-text","ALERTS{aggregation=\"regional_component\", alert_class=\"slo_violation\", alert_type=\"symptom\", alertname=\"WebPagesServiceServerApdexSLOViolationRegional\", alertstate=\"firing\", component=\"server\", env=\"gprd\", environment=\"gprd\", feature_category=\"pages\", monitor=\"global\", pager=\"pagerduty\", region=\"us-east1-d\", rules_domain=\"general\", severity=\"s2\", sli_type=\"apdex\", slo_alert=\"yes\", stage=\"main\", tier=\"sv\", type=\"web-pages\", user_impacting=\"yes\", window=\"1h\"}\n","text","",[48,162,158],{"__ignoreMap":160},[12,164,165,166,169,170,173,174,176],{},"All alerts have the ",[48,167,168],{},"type"," label attached to them to specify which service they\nbelong to. We can use this label and the ",[48,171,172],{},"env"," label to group all the\nproduction alerts that are firing for the ",[48,175,50],{}," service.",[12,178,179],{},[105,180],{"alt":181,"src":182},"grouping alerts by the type and env labels","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398910/blog/Content%20Images/reducing-pager-fatigue/group-alerts-by-service.png",[12,184,185,186,191],{},"We also had to update our Pagerduty and Slack templates to show the right\ninformation. Before we only showed the alert title and description but this had\nto change since we are now alerting by service rather than by 1 specific SLO.\nYou can see the changes at ",[16,187,190],{"href":188,"rel":189},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4684",[],"runbooks!4684",".",[12,193,194],{},[105,195],{"alt":196,"src":197},"Before and after on pages","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398909/blog/Content%20Images/reducing-pager-fatigue/before-after-slack-alerts.png",[12,199,200],{},"This was already a big win! The on-call now gets a page saying \"service\nweb-pages\" and then the list of SLIs that are burning through the error budget - we went from 11 pages to 1 page!",[128,202,204],{"id":203},"service-dependencies","Service Dependencies",[12,206,207,208,63,211,214,215,218],{},"However we still had the problem that when a downstream service (such as the database)\nstarts burning through the error budget, it has a cascading effect where ",[48,209,210],{},"web",[48,212,213],{},"git",", and ",[48,216,217],{},"api"," will also start burning through the error budget and page the\non-call for each service. That was the next thing that we had to solve.",[12,220,221,222,224,225,228,229,231,232,237,238,240,241,243],{},"We needed some way to not alert on the ",[48,223,217],{}," service if the ",[48,226,227],{},"patroni","\n(database) service was burning through the error budget because it's clear if the\ndatabase is degraded the ",[48,230,217],{}," service will end up degraded as well. We used\nanother feature of Alertmanager called\n",[16,233,236],{"href":234,"rel":235},"https://prometheus.io/docs/alerting/latest/alertmanager/#inhibition",[],"inhibition","\nwhere we can tell Alertmanager to not alert on ",[48,239,217],{}," if some alerts on ",[48,242,227],{},"\nare already firing.",[12,245,246],{},[105,247],{"alt":248,"src":249},"visualization of how inhibit rules work","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398910/blog/Content%20Images/reducing-pager-fatigue/inhibit-alerts.png",[12,251,252,253,256,257,262,263,270],{},"I've mentioned that all of our SLIs/SLOs are inside of the\n",[16,254,41],{"href":39,"rel":255},[],"\nso it was a natural fit to add dependencies there, and this is exactly what\nwe did in ",[16,258,261],{"href":259,"rel":260},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4710",[],"runbooks!4710",". With this\nwe can specify that an SLI depends on another SLI of a different service which\nwill automatically create\n",[16,264,267],{"href":265,"rel":266},"https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule",[],[48,268,269],{},"inhibit_rules","\nfor Alertmanager.",[12,272,273],{},"Since inhibit rules could potentially prevent alerting someone, we've used\nthese sparingly. To avoid creating inhibit rules too broadly, we've implemented\nthe following restrictions:",[275,276,277,281,284],"ol",{},[278,279,280],"li",{},"An SLI can't depend on an SLI of the same service.",[278,282,283],{},"The SLI has to exist for that service.",[278,285,286],{},"We only allow equal operations, no regex on SLIs.",[12,288,289,290,293],{},"After that it was only a matter of adding the ",[48,291,292],{},"dependsOn"," on each service for example:",[275,295,296,307,317],{},[278,297,298],{},[16,299,302,304,305],{"href":300,"rel":301},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4735",[],[48,303,210],{}," depends on ",[48,306,227],{},[278,308,309],{},[16,310,313,304,315],{"href":311,"rel":312},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4750",[],[48,314,217],{},[48,316,227],{},[278,318,319],{},[16,320,323,304,325],{"href":321,"rel":322},"https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4813",[],[48,324,50],{},[48,326,217],{},[12,328,329,330,332,333,336,337,339,340,342,343,345],{},"The ",[48,331,50],{}," inhibit rule shows a chain of dependencies from ",[48,334,335],{},"web-pages -> api -> patroni",", so if ",[48,338,227],{}," is burning through the error budget it will\nnot page for ",[48,341,217],{}," and ",[48,344,50],{}," services anymore!",[128,347,349],{"id":348},"how-its-working","How it's working",[12,351,352],{},"We have been using alert grouping and service dependencies for over a month now, and we have already seen some improvements:",[275,354,355,358,361],{},[278,356,357],{},"The on-call only gets paged once per service.",[278,359,360],{},"When there is a large site-wide outage they only get paged 5-10 times since we have external probes that also alert us.",[278,362,363],{},"There is an overall downward trend on pages for the on-call as seen below.",[12,365,366],{},[105,367],{"alt":368,"src":369},"pages trend","https://res.cloudinary.com/about-gitlab-com/image/upload/v1782398913/blog/Content%20Images/reducing-pager-fatigue/pages-trend.png",[12,371,372,373,378,379],{},"Cover image by ",[16,374,377],{"href":375,"rel":376},"https://unsplash.com/@yaoqiqiqilai",[],"Yaoqi"," on ",[16,380,383],{"href":381,"rel":382},"https://unsplash.com/photos/7iatBuqFvY0",[],"Unsplash",{"title":160,"searchDepth":385,"depth":385,"links":386},2,[387,388,389],{"id":130,"depth":385,"text":131},{"id":203,"depth":385,"text":204},{"id":348,"depth":385,"text":349},"engineering","2022-07-19","Too many pages? Here's how we tackled on-call SRE quality of life by grouping alerts by service and only paging on downstream services.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749682368/Blog/Hero%20Images/cover.png",{},true,"/en-us/blog/reducing-pager-fatigue-and-improving-on-call-life",{"title":5,"description":392,"ogTitle":5,"ogDescription":392,"noIndex":395,"ogImage":396,"ogUrl":401,"ogSiteName":402,"ogType":403,"canonicalUrls":401},"https://about.gitlab.com/blog/reducing-pager-fatigue-and-improving-on-call-life","https://about.gitlab.com","article","reducing-pager-fatigue-and-improving-on-call-life","en-us/blog/reducing-pager-fatigue-and-improving-on-call-life",[407],"production","BlogPost","6VMZbWshA6GxifvqGntA67PGJlHuBfqc_cLjnQQ7BkM",{"logo":411,"freeTrial":416,"sales":421,"login":426,"items":431,"search":767,"minimal":799,"duo":818,"switchNav":827,"pricingDeployment":838},{"config":412},{"href":413,"dataGaName":414,"dataGaLocation":415},"/","gitlab logo","header",{"text":417,"config":418},"Get free trial",{"href":419,"dataGaName":420,"dataGaLocation":415},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":422,"config":423},"Request a demo",{"href":424,"dataGaName":425,"dataGaLocation":415},"/request-a-demo/","sales",{"text":427,"config":428},"Sign in",{"href":429,"dataGaName":430,"dataGaLocation":415},"https://gitlab.com/users/sign_in/","sign in",[432,518,606,611,691,745],{"text":433,"left":398,"config":434,"menu":436},"Platform",{"dataNavLevelOne":435},"platform",{"type":437,"link":438,"columns":442,"feature":507},"lists",{"text":439,"config":440},"Explore the Platform",{"href":441,"dataGaName":435,"dataGaLocation":415},"/platform/",[443,466,489],{"title":444,"items":445},"DevOps",[446,451,456,461],{"text":447,"config":448},"CI/CD",{"href":449,"dataGaLocation":415,"dataGaName":450},"/solutions/continuous-integration/","ci/cd",{"text":452,"config":453},"Source Code Management",{"href":454,"dataGaLocation":415,"dataGaName":455},"/solutions/source-code-management/","source code management",{"text":457,"config":458},"Agile Delivery",{"href":459,"dataGaLocation":415,"dataGaName":460},"/solutions/agile-delivery/","agile delivery",{"text":462,"config":463},"Artifact Management",{"href":464,"dataGaLocation":415,"dataGaName":465},"/artifact-central/","artifact management",{"title":467,"items":468},"Security & Governance",[469,474,479,484],{"text":470,"config":471},"Application Security Testing",{"href":472,"dataGaLocation":415,"dataGaName":473},"/solutions/application-security-testing/","application security testing",{"text":475,"config":476},"Governance & Compliance",{"href":477,"dataGaLocation":415,"dataGaName":478},"/solutions/software-compliance/","governance and compliance",{"text":480,"config":481},"Secrets Management",{"href":482,"dataGaLocation":415,"dataGaName":483},"/gitlab-secrets-manager/","secrets management",{"text":485,"config":486},"Supply Chain Security",{"href":487,"dataGaLocation":415,"dataGaName":488},"/solutions/supply-chain/","supply chain security",{"title":490,"items":491},"Agentic AI & Context",[492,497,502],{"text":493,"config":494},"Duo Agent Platform",{"href":495,"dataGaLocation":415,"dataGaName":496},"/gitlab-duo-agent-platform/","duo agent platform",{"text":498,"config":499},"Orbit Context Graph",{"href":500,"dataGaLocation":415,"dataGaName":501},"/gitlab-orbit/","orbit",{"text":503,"config":504},"Visibility & Measurement",{"href":505,"dataGaLocation":415,"dataGaName":506},"/solutions/visibility-measurement/","visibility and measurement",{"config":508,"title":511,"text":512,"link":513},{"background":509,"textColor":510},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1790185102/o7dn32anqcb6z8dgd3as.jpg')","#000","GitLab Orbit is generally available","Higher agent accuracy, lower token cost",{"text":514,"config":515},"Read the announcement",{"href":516,"dataGaName":517,"dataGaLocation":415},"/blog/gitlab-orbit-ga/","orbit ga announcement",{"text":519,"left":398,"config":520,"menu":522},"Solutions",{"dataNavLevelOne":521},"solutions",{"type":437,"link":523,"columns":527,"feature":597},{"text":524,"config":525},"View all Solutions",{"href":526,"dataGaName":521,"dataGaLocation":415},"/solutions/",[528,546,564],{"title":529,"items":530},"Outcomes",[531,536,541],{"text":532,"config":533},"DevOps Modernization",{"href":534,"dataGaLocation":415,"dataGaName":535},"/assessments/devops-modernization-assessment/","devops modernization",{"text":537,"config":538},"Security Modernization",{"href":539,"dataGaLocation":415,"dataGaName":540},"/assessments/security-modernization-assessment/","security modernization",{"text":542,"config":543},"AI Modernization",{"href":544,"dataGaLocation":415,"dataGaName":545},"/assessments/ai-modernization-assessment/","ai modernization",{"title":547,"items":548},"By size",[549,554,559],{"text":550,"config":551},"Enterprise",{"href":552,"dataGaLocation":415,"dataGaName":553},"/enterprise/","enterprise",{"text":555,"config":556},"Small Business",{"href":557,"dataGaLocation":415,"dataGaName":558},"/small-business/","small business",{"text":560,"config":561},"Startups",{"href":562,"dataGaLocation":415,"dataGaName":563},"/solutions/startups/","startups",{"title":565,"items":566},"Industries",[567,572,577,582,587,592],{"text":568,"config":569},"Financial Services",{"href":570,"dataGaLocation":415,"dataGaName":571},"/solutions/finance/","financial services",{"text":573,"config":574},"Public Sector",{"href":575,"dataGaLocation":415,"dataGaName":576},"/solutions/public-sector/","public sector",{"text":578,"config":579},"Telecommunications",{"href":580,"dataGaLocation":415,"dataGaName":581},"/solutions/telecommunications/","telecommunications",{"text":583,"config":584},"Automotive",{"href":585,"dataGaLocation":415,"dataGaName":586},"/solutions/automotive/","automotive",{"text":588,"config":589},"Education",{"href":590,"dataGaLocation":415,"dataGaName":591},"/solutions/education/","education",{"text":593,"config":594},"Aerospace",{"href":595,"dataGaLocation":415,"dataGaName":596},"/solutions/aerospace/","aerospace",{"config":598,"title":599,"text":600,"link":601},{"background":509,"textColor":510},"GitLab Security Standard","Learn how to harden your software factory and let AI agents earn autonomy, without trading away security.",{"text":602,"config":603},"Explore the standard",{"href":604,"dataGaName":605,"dataGaLocation":415},"/gitlab-security-standard/","gitlab security standard",{"text":607,"config":608},"Pricing",{"href":609,"dataGaName":610,"dataGaLocation":415,"dataNavLevelOne":610},"/pricing/","pricing",{"text":612,"config":613,"menu":615},"Resources",{"dataNavLevelOne":614},"resources",{"type":437,"link":616,"columns":620,"feature":682},{"text":617,"config":618},"View all resources",{"href":619,"dataGaName":614,"dataGaLocation":415},"/resources/",[621,654],{"title":622,"items":623},"Discover",[624,629,634,639,644,649],{"text":625,"config":626},"Docs",{"href":627,"dataGaName":628,"dataGaLocation":415},"https://docs.gitlab.com/","docs",{"text":630,"config":631},"University",{"href":632,"dataGaName":633,"dataGaLocation":415},"https://university.gitlab.com/","university",{"text":635,"config":636},"Demo Series",{"href":637,"dataGaName":638,"dataGaLocation":415},"/gitlab-technical-demo-series/","demo series",{"text":640,"config":641},"Demo Hub",{"href":642,"dataGaName":643,"dataGaLocation":415},"/demo-hub/","demo hub",{"text":645,"config":646},"Services",{"href":647,"dataGaName":648,"dataGaLocation":415},"/services/","services",{"text":650,"config":651},"Well Architected",{"href":652,"dataGaName":653,"dataGaLocation":415},"/resources/well-architected/","well architected",{"title":655,"items":656},"Connect",[657,662,667,672,677],{"text":658,"config":659},"Blog",{"href":660,"dataGaName":661,"dataGaLocation":415},"/blog/","blog",{"text":663,"config":664},"Community",{"href":665,"dataGaName":666,"dataGaLocation":415},"/community/","community",{"text":668,"config":669},"Customers",{"href":670,"dataGaName":671,"dataGaLocation":415},"/customers/","customers",{"text":673,"config":674},"Partners",{"href":675,"dataGaName":676,"dataGaLocation":415},"/partners/","partners",{"text":678,"config":679},"Events",{"href":680,"dataGaName":681,"dataGaLocation":415},"/events/","events",{"config":683,"title":684,"text":685,"link":686},{"background":509,"textColor":510},"What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":687,"config":688},"Read the latest",{"href":689,"dataGaName":690,"dataGaLocation":415},"/whats-new/","whats new",{"text":692,"config":693,"menu":695},"Company",{"dataNavLevelOne":694},"company",{"type":437,"columns":696},[697],{"items":698},[699,704,710,715,720,725,730,735,740],{"text":700,"config":701},"About",{"href":702,"dataGaName":703,"dataGaLocation":415},"/company/","about",{"text":705,"config":706,"footerGa":709},"Jobs",{"href":707,"dataGaName":708,"dataGaLocation":415},"/jobs/","jobs",{"dataGaName":708},{"text":711,"config":712},"Press",{"href":713,"dataGaName":714,"dataGaLocation":415},"/press/","press",{"text":716,"config":717},"Handbook",{"href":718,"dataGaName":719,"dataGaLocation":415},"https://handbook.gitlab.com/","handbook",{"text":721,"config":722},"Leadership",{"href":723,"dataGaName":724,"dataGaLocation":415},"/company/team/e-group/","leadership",{"text":726,"config":727},"Investor relations",{"href":728,"dataGaName":729,"dataGaLocation":415},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":731,"config":732},"Trust Center",{"href":733,"dataGaName":734,"dataGaLocation":415},"/security/","trust center",{"text":736,"config":737},"AI Transparency Center",{"href":738,"dataGaName":739,"dataGaLocation":415},"/ai-transparency-center/","ai transparency center",{"text":741,"config":742},"Newsletter",{"href":743,"dataGaName":744,"dataGaLocation":415},"/company/contact/#contact-forms","newsletter",{"text":746,"config":747,"menu":748},"Contact us",{"dataNavLevelOne":694},{"type":437,"columns":749},[750],{"items":751},[752,757,762],{"text":753,"config":754},"Talk to sales",{"href":755,"dataGaName":756,"dataGaLocation":415},"/sales/","talk to sales",{"text":758,"config":759},"Support portal",{"href":760,"dataGaName":761,"dataGaLocation":415},"https://support.gitlab.com/hc/en-us","support portal",{"text":763,"config":764},"Customer portal",{"href":765,"dataGaName":766,"dataGaLocation":415},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":768,"login":769,"suggestions":776},"Close",{"text":770,"link":771},"To search repositories and projects, login to",{"text":772,"config":773},"gitlab.com",{"href":429,"dataGaName":774,"dataGaLocation":775},"search login","search",{"text":777,"default":778},"Suggestions",[779,782,785,787,791,795],{"text":780,"config":781},"GitLab Duo Agent Platform",{"href":495,"dataGaName":780,"dataGaLocation":775},{"text":783,"config":784},"Code Suggestions (AI)",{"href":495,"dataGaName":783,"dataGaLocation":775},{"text":447,"config":786},{"href":449,"dataGaName":447,"dataGaLocation":775},{"text":788,"config":789},"GitLab on AWS",{"href":790,"dataGaName":788,"dataGaLocation":775},"/partners/technology-partners/aws/",{"text":792,"config":793},"GitLab on Google Cloud",{"href":794,"dataGaName":792,"dataGaLocation":775},"/partners/technology-partners/google-cloud-platform/",{"text":796,"config":797},"Why GitLab?",{"href":798,"dataGaName":796,"dataGaLocation":775},"/why-gitlab/",{"freeTrial":800,"mobileIcon":805,"desktopIcon":810,"secondaryButton":813},{"text":801,"config":802},"Start free trial",{"href":803,"dataGaName":420,"dataGaLocation":804},"https://gitlab.com/-/trials/new/","nav",{"altText":806,"config":807},"Gitlab Icon",{"src":808,"dataGaName":809,"dataGaLocation":804},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":806,"config":811},{"src":812,"dataGaName":809,"dataGaLocation":804},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":814,"config":815},"Get Started",{"href":816,"dataGaName":817,"dataGaLocation":804},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":819,"mobileIcon":823,"desktopIcon":825},{"text":820,"config":821},"Learn more about GitLab Duo",{"href":495,"dataGaName":822,"dataGaLocation":804},"gitlab duo",{"altText":806,"config":824},{"src":808,"dataGaName":809,"dataGaLocation":804},{"altText":806,"config":826},{"src":812,"dataGaName":809,"dataGaLocation":804},{"button":828,"mobileIcon":833,"desktopIcon":835},{"text":829,"config":830},"/switch",{"href":831,"dataGaName":832,"dataGaLocation":804},"#contact","switch",{"altText":806,"config":834},{"src":808,"dataGaName":809,"dataGaLocation":804},{"altText":806,"config":836},{"src":837,"dataGaName":809,"dataGaLocation":804},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":839,"mobileIcon":844,"desktopIcon":846},{"text":840,"config":841},"Back to pricing",{"href":609,"dataGaName":842,"dataGaLocation":804,"icon":843},"back to pricing","GoBack",{"altText":806,"config":845},{"src":808,"dataGaName":809,"dataGaLocation":804},{"altText":806,"config":847},{"src":812,"dataGaName":809,"dataGaLocation":804},{"title":849,"button":850,"config":854},"GitLab Orbit is generally available. Increase agent accuracy and lower token cost",{"text":514,"config":851},{"href":516,"dataGaName":852,"dataGaLocation":853},"orbit-ga-banner","global-banner",{"layout":855,"disabled":395},"release",{"data":857},{"text":858,"config":859,"items":864,"minimal":1092},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"twitter":860,"facebook":861,"youtube":862,"linkedin":863},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[865,913,972,1019,1060],{"title":607,"links":866,"subMenu":882},[867,872,877],{"text":868,"config":869},"View plans",{"href":609,"dataGaName":870,"dataGaLocation":871},"view plans","footer",{"text":873,"config":874},"Why Premium?",{"href":875,"dataGaName":876,"dataGaLocation":871},"/pricing/premium/","why premium",{"text":878,"config":879},"Why Ultimate?",{"href":880,"dataGaName":881,"dataGaLocation":871},"/pricing/ultimate/","why ultimate",[883],{"title":884,"links":885},"Contact Us",[886,889,891,893,898,903,908],{"text":887,"config":888},"Contact sales",{"href":755,"dataGaName":425,"dataGaLocation":871},{"text":758,"config":890},{"href":760,"dataGaName":761,"dataGaLocation":871},{"text":763,"config":892},{"href":765,"dataGaName":766,"dataGaLocation":871},{"text":894,"config":895},"Status",{"href":896,"dataGaName":897,"dataGaLocation":871},"https://status.gitlab.com/","status",{"text":899,"config":900},"Terms of use",{"href":901,"dataGaName":902,"dataGaLocation":871},"/terms/","terms of use",{"text":904,"config":905},"Privacy statement",{"href":906,"dataGaName":907,"dataGaLocation":871},"/privacy/","privacy statement",{"text":909,"config":910},"Cookie preferences",{"dataGaName":911,"dataGaLocation":871,"id":912,"isOneTrustButton":398},"cookie preferences","ot-sdk-btn",{"title":433,"links":914,"subMenu":930},[915,918,920,922,924,926],{"text":916,"config":917},"Explore the platform",{"href":441,"dataGaName":435,"dataGaLocation":871},{"text":493,"config":919},{"href":495,"dataGaName":496,"dataGaLocation":871},{"text":447,"config":921},{"href":449,"dataGaName":450,"dataGaLocation":871},{"text":498,"config":923},{"href":500,"dataGaName":501,"dataGaLocation":871},{"text":480,"config":925},{"href":482,"dataGaName":483,"dataGaLocation":871},{"text":927,"config":928},"Why GitLab",{"href":798,"dataGaName":929,"dataGaLocation":871},"why gitlab",[931],{"title":932,"links":933},"Topics",[934,938,943,947,952,957,962,967],{"text":447,"config":935},{"href":936,"dataGaName":937,"dataGaLocation":871},"/topics/ci-cd/","cicd",{"text":939,"config":940},"GitOps",{"href":941,"dataGaName":942,"dataGaLocation":871},"/topics/gitops/","gitops",{"text":444,"config":944},{"href":945,"dataGaName":946,"dataGaLocation":871},"/topics/devops/","devops",{"text":948,"config":949},"Version Control",{"href":950,"dataGaName":951,"dataGaLocation":871},"/topics/version-control/","version control",{"text":953,"config":954},"DevSecOps",{"href":955,"dataGaName":956,"dataGaLocation":871},"/topics/devsecops/","devsecops",{"text":958,"config":959},"Cloud Native",{"href":960,"dataGaName":961,"dataGaLocation":871},"/topics/cloud-native/","cloud native",{"text":963,"config":964},"AI for Coding",{"href":965,"dataGaName":966,"dataGaLocation":871},"/topics/devops/ai-for-coding/","ai for coding",{"text":968,"config":969},"Agentic AI",{"href":970,"dataGaName":971,"dataGaLocation":871},"/topics/agentic-ai/","agentic ai",{"title":519,"links":973},[974,977,980,983,986,989,991,993,995,997,999,1002,1004,1007,1011,1015],{"text":975,"config":976},"DevOps modernization",{"href":534,"dataGaName":535,"dataGaLocation":871},{"text":978,"config":979},"Security modernization",{"href":539,"dataGaName":540,"dataGaLocation":871},{"text":981,"config":982},"AI modernization",{"href":544,"dataGaName":545,"dataGaLocation":871},{"text":984,"config":985},"Financial services",{"href":570,"dataGaName":571,"dataGaLocation":871},{"text":987,"config":988},"Public sector",{"href":575,"dataGaName":576,"dataGaLocation":871},{"text":578,"config":990},{"href":580,"dataGaName":581,"dataGaLocation":871},{"text":583,"config":992},{"href":585,"dataGaName":586,"dataGaLocation":871},{"text":588,"config":994},{"href":590,"dataGaName":591,"dataGaLocation":871},{"text":593,"config":996},{"href":595,"dataGaName":596,"dataGaLocation":871},{"text":550,"config":998},{"href":552,"dataGaName":553,"dataGaLocation":871},{"text":1000,"config":1001},"Small business",{"href":557,"dataGaName":558,"dataGaLocation":871},{"text":560,"config":1003},{"href":562,"dataGaName":563,"dataGaLocation":871},{"text":939,"config":1005},{"href":1006,"dataGaName":942,"dataGaLocation":871},"/solutions/gitops/",{"text":1008,"config":1009},"Software composition analysis",{"href":487,"dataGaName":1010,"dataGaLocation":871},"software composition analysis",{"text":1012,"config":1013},"Value stream management",{"href":505,"dataGaName":1014,"dataGaLocation":871},"value stream management",{"text":1016,"config":1017},"All solutions",{"href":526,"dataGaName":1018,"dataGaLocation":871},"all solutions",{"title":612,"links":1020},[1021,1026,1031,1033,1035,1037,1039,1041,1043,1045,1047,1049,1051,1053,1056],{"text":1022,"config":1023},"Install",{"href":1024,"dataGaName":1025,"dataGaLocation":871},"/install/","install",{"text":1027,"config":1028},"Quick start guides",{"href":1029,"dataGaName":1030,"dataGaLocation":871},"/get-started/","quick setup checklists",{"text":625,"config":1032},{"href":627,"dataGaName":628,"dataGaLocation":871},{"text":630,"config":1034},{"href":632,"dataGaName":633,"dataGaLocation":871},{"text":635,"config":1036},{"href":637,"dataGaName":638,"dataGaLocation":871},{"text":640,"config":1038},{"href":642,"dataGaName":643,"dataGaLocation":871},{"text":645,"config":1040},{"href":647,"dataGaName":648,"dataGaLocation":871},{"text":658,"config":1042},{"href":660,"dataGaName":661,"dataGaLocation":871},{"text":663,"config":1044},{"href":665,"dataGaName":666,"dataGaLocation":871},{"text":668,"config":1046},{"href":670,"dataGaName":671,"dataGaLocation":871},{"text":673,"config":1048},{"href":675,"dataGaName":676,"dataGaLocation":871},{"text":678,"config":1050},{"href":680,"dataGaName":681,"dataGaLocation":871},{"text":741,"config":1052},{"href":743,"dataGaName":744,"dataGaLocation":871},{"text":1054,"config":1055},"What's new",{"href":689,"dataGaName":690,"dataGaLocation":871},{"text":1057,"config":1058},"All resources",{"href":619,"dataGaName":1059,"dataGaLocation":871},"all resources",{"title":692,"links":1061},[1062,1064,1066,1068,1070,1072,1074,1076,1078,1082,1087],{"text":700,"config":1063},{"href":702,"dataGaName":694,"dataGaLocation":871},{"text":705,"config":1065},{"href":707,"dataGaName":708,"dataGaLocation":871},{"text":711,"config":1067},{"href":713,"dataGaName":714,"dataGaLocation":871},{"text":716,"config":1069},{"href":718,"dataGaName":719,"dataGaLocation":871},{"text":721,"config":1071},{"href":723,"dataGaName":724,"dataGaLocation":871},{"text":726,"config":1073},{"href":728,"dataGaName":729,"dataGaLocation":871},{"text":731,"config":1075},{"href":733,"dataGaName":734,"dataGaLocation":871},{"text":736,"config":1077},{"href":738,"dataGaName":739,"dataGaLocation":871},{"text":1079,"config":1080},"Sustainability",{"href":1081,"dataGaName":1079,"dataGaLocation":871},"/sustainability/",{"text":1083,"config":1084},"Diversity, inclusion and belonging (DIB)",{"href":1085,"dataGaName":1086,"dataGaLocation":871},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":1088,"config":1089},"Modern Slavery Transparency Statement",{"href":1090,"dataGaName":1091,"dataGaLocation":871},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":1093},[1094,1097,1100],{"text":1095,"config":1096},"Terms",{"href":901,"dataGaName":902,"dataGaLocation":871},{"text":1098,"config":1099},"Cookies",{"dataGaName":911,"dataGaLocation":871,"id":912,"isOneTrustButton":398},{"text":1101,"config":1102},"Privacy",{"href":906,"dataGaName":907,"dataGaLocation":871},[1104],{"id":1105,"title":7,"body":394,"config":1106,"content":1108,"description":394,"extension":1111,"meta":1112,"navigation":398,"path":1113,"seo":1114,"stem":1115,"__hash__":1116},"blogAuthors/en-us/blog/authors/steve-azzopardi.yml",{"template":1107},"BlogAuthor",{"name":7,"config":1109},{"headshot":160,"ctfId":1110},"steveazz","yml",{},"/en-us/blog/authors/steve-azzopardi",{},"en-us/blog/authors/steve-azzopardi","qnb6MwMS78l86ZK9JZJ-5qc-NcHCq2bdzgXn2tHRN40",[1118,1126,1134],{"title":1119,"description":1120,"heroImage":1121,"category":390,"date":1122,"authors":1123,"slug":1125,"externalUrl":394},"Two front doors: Module-level access in a Django GRC app","A technical deep-dive on restructuring a Django internal tool to support two distinct GitLab teams: Security Compliance and Internal Audit.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1750097663/Blog/Hero%20Images/Blog/Hero%20Images/Workflow%201800x945_2gQoQIbY9NvjLFpXtsxtXy_1750097663612.png","2026-10-05",[1124],"Donovan Felton","module-level-access-in-a-django-grc-app",{"title":1127,"description":1128,"heroImage":1129,"category":390,"date":1130,"authors":1131,"slug":1133,"externalUrl":394},"How GitLab reduced code-per-agentic-flow ratio by 45%","Learn how Flow Registry, a component-based framework that compiles YAML configurations into fully functional LangGraph flows, helped improve scalability and more.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-09-22",[1132],"Mikolaj Wawrzyniak","how-gitlab-reduced-code-per-agentic-flow-ratio",{"title":1135,"description":1136,"heroImage":1137,"category":390,"date":1138,"authors":1139,"slug":1141,"externalUrl":394},"Co-Create: Building GitLab with our users","How GitLab users collaborated with us to build practical product improvements in the first half of 2026.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1750099203/Blog/Hero%20Images/Blog/Hero%20Images/blog-image-template-1800x945%20%2820%29_2bJGC5ZP3WheoqzlLT05C5_1750099203484.png","2026-09-10",[1140],"Isa Huerga","co-create-h1-2026",{"header":1143,"text":1144,"button":1148,"disclaimer":1151},"Start your free\n\n30-day GitLab trial\n",[1145,1146,1147],"GitLab Duo Agent Platform access","Try our most advanced features","Automate complex tasks with AI",{"text":417,"config":1149},{"href":1150,"dataGaName":420,"dataGaLocation":661},"https://gitlab.com/-/trials/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/blog/","No credit card required.",{"header":1153,"blurb":1154,"button":1155,"secondaryButton":1160},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1156,"config":1157},"Get your free trial",{"href":1158,"dataGaName":420,"dataGaLocation":1159},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":887,"config":1161},{"href":755,"dataGaName":425,"dataGaLocation":1159},1794911783307]