[{"data":1,"prerenderedAt":1270},["ShallowReactive",2],{"/blog/puma-nakayoshi-fork-and-compaction":3,"navigation-en-us":516,"banner-en-us":954,"footer-en-us":962,"blog-post-authors-en-us-Matthias Käppler":1209,"blog-related-posts-en-us-puma-nakayoshi-fork-and-compaction":1225,"blog-share-wrapper-en-us":1250,"next-steps-en-us":1260},{"id":4,"title":5,"authors":6,"body":8,"category":494,"date":495,"description":496,"extension":497,"externalUrl":498,"faq":498,"featured":499,"heroImage":500,"hidePromo":499,"meta":501,"navigation":502,"path":503,"seo":504,"slug":508,"stem":509,"tags":510,"template":514,"updatedDate":498,"__hash__":515},"blogPosts/en-us/blog/puma-nakayoshi-fork-and-compaction.md","Ruby 2.7: Understand and debug problems with heap compaction",[7],"Matthias Käppler",{"type":9,"value":10,"toc":485},"minimark",[11,33,36,45,61,97,100,121,125,128,131,143,149,152,156,165,194,198,201,302,333,351,357,365,387,393,397,404,411,414,439,443,446,449,478,481],[12,13,14,15,21,22,27,28,32],"p",{},"The GitLab Rails application runs on ",[16,17,20],"a",{"href":18,"rel":19},"https://puma.io/",[],"Puma",", a multi-threaded Rack application server written in the new Ruby.\nWe recently updated Puma to major version 5, which introduced ",[16,23,26],{"href":24,"rel":25},"https://github.com/puma/puma/blob/master/History.md#500--2020-09-17",[],"a number of important\nchanges",",\nincluding support for ",[29,30,31],"em",{},"compaction",", a technique to reduce memory fragmentation in the\nRuby heap.",[12,34,35],{},"In this post we will describe what Puma's \"nakayoshi fork\" does, what compaction is,\nand some of the challenges we faced when first deploying it.",[37,38,40,41],"h2",{"id":39},"nakayoshi-a-friendlier-fork","Nakayoshi: A friendlier ",[42,43,44],"code",{},"fork",[12,46,47,48,51,52,57,58,60],{},"Puma 5 added a new configuration switch: ",[42,49,50],{},"nakayoshi_fork",". This switch affects Puma's behavior when\nforking new workers from the primary process. It is largely based on a ",[16,53,56],{"href":54,"rel":55},"https://github.com/ko1/nakayoshi_fork",[],"Ruby gem of the same name","\nbut adds new functionality. More specifically, enabling ",[42,59,50],{}," in Puma will result in two additional\nsteps prior to forking into new workers:",[62,63,64,81],"ol",{},[65,66,67,71,72,74,75,80],"li",{},[68,69,70],"strong",{},"Tenuring objects."," By running several minor garbage collection cycles ahead of a ",[42,73,44],{},", Ruby can promote survivors\nfrom the young to the old generation (referred to as \"tenuring\"). These objects are often classes, modules, or long-lived\nconstants that are unlikely to change.\nThis process makes forking copy-on-write friendly because tagging an object as \"old\" implies a write\nto the underlying heap page. Doing this prior to forking means the OS won't have\nto copy this page from the parent to the worker process later. We won't be discussing copy-on-write in detail but\n",[16,76,79],{"href":77,"rel":78},"https://brandur.org/ruby-memory",[],"this blog post offers a good introduction to the topic and how it relates to Ruby and pre-fork servers",".",[65,82,83,86,87,90,91,93,94,96],{},[68,84,85],{},"Heap compaction."," Ruby 2.7 added a new method ",[42,88,89],{},"GC.compact",", which\nwill reorganize the Ruby heap to pack objects closer together when invoked. ",[42,92,89],{}," reduces Ruby heap fragmentation and\npotentially frees up Ruby heap pages so that the physical memory consumed can be reclaimed by the OS.\nThis step only happens when ",[42,95,89],{}," is available in the version of Ruby that is in use (for MRI, 2.7 or newer).",[12,98,99],{},"In the remainder of this post, we will look at:",[101,102,103,109,112,115],"ul",{},[65,104,105,106,108],{},"How ",[42,107,89],{}," works and its potential benefits.",[65,110,111],{},"Why using C-extensions can be problematic when using compaction.",[65,113,114],{},"How we resolved a production incident that crashed GitLab.",[65,116,117,118,120],{},"What to look out for before enabling compaction in your app, via ",[42,119,50],{}," or otherwise.",[37,122,124],{"id":123},"how-compacting-garbage-collection-works","How compacting garbage collection works",[12,126,127],{},"The primary goal of a compacting garbage collector (GC) is to use allocated memory more\neffectively, which increases the likelihood of the application using less memory over time.\nCompaction is especially important when processes can share memory, as is the case with Ruby pre-fork\nservers such as Puma or Unicorn. But how does Ruby accomplish this?",[12,129,130],{},"Ruby manages its own object heap by allocating chunks of memory from the operating system called pages\n(a confusing term since Ruby heap pages are distinct from the smaller memory pages managed by the OS itself).\nWhen an application asks to create a new object, Ruby will try to find a free object slot in one of these\npages and fill it. As objects are allocated and deallocated over the lifetime of the application,\nthis can lead to fragmentation, with pages being neither entirely full nor entirely empty. This is the\nprimary cause for Ruby's infamous runaway memory problem: Since the available space isn't optimally used,\npages will rarely be entirely empty and become \"tomb pages\" which means it is necessary for the pages to be empty for them to be deallocated.",[12,132,133,134,136,137,142],{},"Ruby 2.7 added a new method, ",[42,135,89],{},", which aims to address this problem by walking the entire\nRuby heap space and moving objects around to obtain tightly packed pages. This process will ideally make\nsome pages unused, and unused memory can be reclaimed by the OS. ",[16,138,141],{"href":139,"rel":140},"https://www.youtube.com/watch?v=H8iWLoarTZc",[],"Watch this video from RubyConf 2019"," where Aaron Patterson, the author of this feature, gave a good introduction to compacting GC.",[12,144,145,146,148],{},"Compaction is a fairly expensive task since Ruby needs to stop-the-world for a complete heap reorganization so\nits best to perform this task before forking a new worker process, which is why Puma 5 included this step when performing ",[42,147,50],{},". Moreover, running compaction before forking\ninto worker processes increases the chance of workers being able to share memory.",[12,150,151],{},"We were eager to enable this feature on GitLab to see if it would reduce memory consumption, but things didn't entirely go as planned.",[37,153,155],{"id":154},"inside-the-incident","Inside the incident",[12,157,158,159,164],{},"After extensive testing via our automated performance test suite and in preproduction\nenvironments, we felt ready to explore compaction on production nodes. We kept a\n",[16,160,163],{"href":161,"rel":162},"https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3370",[],"detailed, public record of what happened\nduring this production incident",", but the key details are summarized below:",[101,166,167,170,173,180,191],{},[65,168,169],{},"The deployment passed the canary stage, meaning workers who had their heaps compacted were serving traffic\nsuccessfully at this point.",[65,171,172],{},"Sometime during the full fleet rollout, problems emerged: Error rates started spiking but not\nacross the entire fleet. This phenomenon is odd because errors tend to spread across all workers due to load balancing.",[65,174,175,176,179],{},"The error messages surfacing in Sentry were mysterious at best:\n",[42,177,178],{},"ActionView::Template::Error uninitialized constant #\u003CClass:#GrapePathHelpers::DecoratedRoute:0x00007f95f10ea5b8>::UNDERSCORE",". Remember this error message for later.",[65,181,182,183,190],{},"We discovered the affected workers were segfaulting in ",[16,184,187],{"href":185,"rel":186},"https://github.com/k0kubun/hamlit",[],[42,188,189],{},"hamlit",",\na high-performance HAML compiler. Hamlit uses a C-extension to achieve better performance. The segfaulting and the fact\nthat we were rolling out an optimization that touches GC-internal structures was a tell-tale sign that\ncompaction was likely to be the cause.",[65,192,193],{},"We rolled back the change to quickly recover from the outage.",[37,195,197],{"id":196},"how-we-diagnosed-the-problem","How we diagnosed the problem",[12,199,200],{},"We were disappointed by this setback and wanted to understand why the outage occurred. Fortunately,\nRuby provides detailed stack traces when crashing in C-extensions. The most effective way\nto quickly analyze these is to look for transitions where a C-extension calls into the Ruby VM\nor vice versa. These lines therefore caught our attention:",[202,203,208],"pre",{"className":204,"code":205,"language":206,"meta":207,"style":207},"language-shell shiki shiki-themes github-light","...\n/opt/gitlab/embedded/lib/libruby.so.2.7(sigsegv+0x52) [0x7f9601adb932] signal.c:946\n/lib/x86_64-linux-gnu/libc.so.6(0x7f960154c4c0) [0x7f960154c4c0]\n/opt/gitlab/embedded/lib/libruby.so.2.7(rb_id_table_lookup+0x1) [0x7f9601b15e11] id_table.c:227\n/opt/gitlab/embedded/lib/libruby.so.2.7(rb_const_lookup+0x1e) [0x7f9601b4861e] variable.c:3357\n/opt/gitlab/embedded/lib/libruby.so.2.7(rb_const_get+0x39) [0x7f9601b4a049] variable.c:2339\n# ^--- Ruby VM functions\n/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/hamlit-2.11.0/lib/hamlit/hamlit.so(str_underscore+0x16) [0x7f95ee3518f8] hamlit.c:17\n/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/hamlit-2.11.0/lib/hamlit/hamlit.so(rb_hamlit_build_id) hamlit.c:100\n# ^-- hamlit C-extension\n...\n","shell","",[42,209,210,219,230,239,248,257,266,273,282,291,297],{"__ignoreMap":207},[211,212,215],"span",{"class":213,"line":214},"line",1,[211,216,218],{"class":217},"sYu0t","...\n",[211,220,222,226],{"class":213,"line":221},2,[211,223,225],{"class":224},"s7eDp","/opt/gitlab/embedded/lib/libruby.so.2.7(sigsegv+0x52",[211,227,229],{"class":228},"sgsFI",") [0x7f9601adb932] signal.c:946\n",[211,231,233,236],{"class":213,"line":232},3,[211,234,235],{"class":224},"/lib/x86_64-linux-gnu/libc.so.6(0x7f960154c4c0",[211,237,238],{"class":228},") [0x7f960154c4c0]\n",[211,240,242,245],{"class":213,"line":241},4,[211,243,244],{"class":224},"/opt/gitlab/embedded/lib/libruby.so.2.7(rb_id_table_lookup+0x1",[211,246,247],{"class":228},") [0x7f9601b15e11] id_table.c:227\n",[211,249,251,254],{"class":213,"line":250},5,[211,252,253],{"class":224},"/opt/gitlab/embedded/lib/libruby.so.2.7(rb_const_lookup+0x1e",[211,255,256],{"class":228},") [0x7f9601b4861e] variable.c:3357\n",[211,258,260,263],{"class":213,"line":259},6,[211,261,262],{"class":224},"/opt/gitlab/embedded/lib/libruby.so.2.7(rb_const_get+0x39",[211,264,265],{"class":228},") [0x7f9601b4a049] variable.c:2339\n",[211,267,269],{"class":213,"line":268},7,[211,270,272],{"class":271},"sAwPA","# ^--- Ruby VM functions\n",[211,274,276,279],{"class":213,"line":275},8,[211,277,278],{"class":224},"/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/hamlit-2.11.0/lib/hamlit/hamlit.so(str_underscore+0x16",[211,280,281],{"class":228},") [0x7f95ee3518f8] hamlit.c:17\n",[211,283,285,288],{"class":213,"line":284},9,[211,286,287],{"class":224},"/opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/hamlit-2.11.0/lib/hamlit/hamlit.so(rb_hamlit_build_id",[211,289,290],{"class":228},") hamlit.c:100\n",[211,292,294],{"class":213,"line":293},10,[211,295,296],{"class":271},"# ^-- hamlit C-extension\n",[211,298,300],{"class":213,"line":299},11,[211,301,218],{"class":217},[12,303,304,305,308,309,312,313,316,317,320,321,328,329,332],{},"The topmost stack frame reveals the preceeding calls led to a segmentation fault (",[42,306,307],{},"SIGSEGV",").\nWe highlighted the lines where Hamlit calls back into Ruby: In a function called ",[42,310,311],{},"str_underscore"," which\nwas called by ",[42,314,315],{},"rb_hamlit_build_id",". The ",[42,318,319],{},"rb_*"," prefix tells us that this is a C-function we can call from Ruby,\nand indeed it is used by ",[16,322,325],{"href":323,"rel":324},"https://github.com/k0kubun/hamlit/blob/master/lib/hamlit/attribute_builder.rb",[],[42,326,327],{},"Hamlit::AttributeBuilder"," to construct DOM ",[42,330,331],{},"id","s.",[12,334,335,336,338,339,342,343,346,347,350],{},"But we still don't know why it is crashing. Next, we need to inspect what happens in ",[42,337,311],{},".\nWe can see that this function performs a constant lookup on ",[42,340,341],{},"mAttributeBuilder"," – searching\nfor a constant called ",[42,344,345],{},"UNDERSCORE",". When following the breadcrumbs it turns out to simply be the string ",[42,348,349],{},"\"_\"",".\nIt is this lookup that failed.",[12,352,353,354,356],{},"Wait -- ",[42,355,345],{},"? That sounds familiar. Recall the top-level error messages:",[202,358,363],{"className":359,"code":361,"language":362,"meta":207},[360],"language-text","ActionView::Template::Error\nuninitialized constant #\u003CClass:#GrapePathHelpers::DecoratedRoute:0x00007f95f10ea5b8>::UNDERSCORE\n","text",[42,364,361],{"__ignoreMap":207},[12,366,367,368,371,372,374,375,382,383,386],{},"But ",[42,369,370],{},"GrapePathHelpers"," is clearly not a Hamlit class. Hamlit is trying to look up its own ",[42,373,345],{},"\nconstant on a class in the ",[16,376,379],{"href":377,"rel":378},"https://github.com/ruby-grape/grape",[],[42,380,381],{},"grape"," gem, an entirely different library\nthat is not involved in HTML rendering at all and there is no such constant defined on Grape's\n",[42,384,385],{},"DecoratedRoute"," class either.",[12,388,389,390,392],{},"Now the penny dropped – remember how compaction moves around objects in Ruby's heap space? Classes in\nRuby are objects too, so ",[42,391,89],{}," must have moved a Grape class into an object slot that was previously\noccupied by a Hamlit class object, but Hamlit's C-extension never saw it coming!",[37,394,396],{"id":395},"how-we-solved-the-problem","How we solved the problem",[12,398,399,400,403],{},"To be clear, what happened above should ",[29,401,402],{},"not"," happen with a well-behaved C-extension. Compaction\nwas developed carefully with support for C-extensions that predate Ruby 2.7, so all\nexisting Ruby gems would continue to operate normally.",[12,405,406,407,410],{},"So what went wrong? When a C-extension allocates Ruby objects, it must ",[29,408,409],{},"mark"," them for as long as\nthey are alive. A marked object will not be garbage collected and because the Ruby GC cannot reason about objects\noutside of its own purview (i.e., objects created from Ruby code), it needs to rely on C-extensions\nto correctly mark and unmark objects themselves.",[12,412,413],{},"Now comes the twist: Marked objects can be moved during compaction and existing C-extensions\ncan't cope with an object they hold pointers to suddenly move into a different slot.\nTherefore, Ruby 2.7 does something clever: It \"pins\" objects allocated with the mark function that existed prior\nto Ruby 2.7, meaning the pinned objects are not allowed to move during compaction. For new code, it introduces\na special mark-but-don't-pin function that will also allow an object to move, giving gem authors the\nopportunity to make their libraries compaction-aware.",[12,415,416,417,422,423,428,429,432,433,438],{},"Hamlit does not implement compaction support, so this could only mean one thing:\nHamlit wasn't even properly marking those objects, otherwise Ruby 2.7\nwould have automatically pinned them so they wouldn't move during compaction.\nAfter ",[16,418,421],{"href":419,"rel":420},"https://github.com/k0kubun/hamlit/pull/171",[],"discussing an attempted fix we submitted"," but without\na reliable way to reproduce the issue for everyone, the Hamlit author decided to sidestep the\nproblem by ",[16,424,427],{"href":425,"rel":426},"https://github.com/k0kubun/hamlit/pull/172",[],"resolving those constants statically instead","\nand marking each via ",[42,430,431],{},"rb_gc_register_mark_object",".\nThis change landed in ",[16,434,437],{"href":435,"rel":436},"https://github.com/k0kubun/hamlit/blob/master/CHANGELOG.md#2142---2021-01-21",[],"Hamlit 2.14.2","\nwhich we confirmed resolves the issue.",[37,440,442],{"id":441},"the-next-steps","The next steps",[12,444,445],{},"It is exciting to see that the Ruby community is making progress on making Ruby a more memory-efficient\nlanguage but we learned that we need to step carefully when introducing such wide-reaching changes to a large\napplication like GitLab. It is difficult to investigate and fix problems that crash the Ruby VM, which is more likely for\nany library that uses C-extensions.",[12,447,448],{},"Two particular action items we took away from this were:",[62,450,451,461],{},[65,452,453,456,457,460],{},[68,454,455],{},"More reliable detection of compaction-related issues in CI."," We're not going to sugar-coat this:\nWe detected the problem late. Our comprehensive test suite was passing, our QA and performance tests\non staging environments passed, and the problem didn't even show up in canary deployments. Ideally, we\nwould have caught this issue with our automated test suite. One way to test whether compaction causes problems\nis by using ",[42,458,459],{},"GC.verify_compaction_references"," – this is a rather crude tool because it requires\nkeeping two copies of the Ruby heap, which can be prohibitively expensive in terms of memory use. We\nhave therefore not yet decided how to approach this.",[65,462,463,466,467,472,473,80],{},[68,464,465],{},"Improve our ability to roll out system configuration gradually."," Puma is part of our core infrastructure,\nsince it sits in the path of every web request, which makes it especially risky to experiment with Puma\nconfiguration. GitLab already supports ",[16,468,471],{"href":469,"rel":470},"https://docs.gitlab.com/development/feature_flags/",[],"feature flags","\nto allow developers to roll out product changes gradually, but it presents us with a catch-22 when\nmaking changes at the infrastructure level, because to query the state of a feature flag, the infrastructure\nneeds to already be up and running. It would be ideal to have a similar mechanism for system configuration, ",[16,474,477],{"href":475,"rel":476},"https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/154",[],"which we are currently exploring",[12,479,480],{},"While performance is a major focus for us at the moment it must not compromise availability.\nWe will continue to monitor developments in the Ruby community around compaction support, but decided to\nnot use it in production at this point in time since the gains don't appear to outweigh the risks.",[482,483,484],"style",{},"html pre.shiki code .sYu0t, html code.shiki .sYu0t{--shiki-default:#005CC5}html pre.shiki code .s7eDp, html code.shiki .s7eDp{--shiki-default:#6F42C1}html pre.shiki code .sgsFI, html code.shiki .sgsFI{--shiki-default:#24292E}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}",{"title":207,"searchDepth":221,"depth":221,"links":486},[487,489,490,491,492,493],{"id":39,"depth":221,"text":488},"Nakayoshi: A friendlier fork",{"id":123,"depth":221,"text":124},{"id":154,"depth":221,"text":155},{"id":196,"depth":221,"text":197},{"id":395,"depth":221,"text":396},{"id":441,"depth":221,"text":442},"engineering","2021-04-28","An overview of Ruby 2.7 heap compaction and the risks it adds to production Rails applications.","md",null,false,"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749669673/Blog/Hero%20Images/engineering.png",{},true,"/en-us/blog/puma-nakayoshi-fork-and-compaction",{"title":5,"description":496,"ogTitle":5,"ogDescription":496,"noIndex":499,"ogImage":500,"ogUrl":505,"ogSiteName":506,"ogType":507,"canonicalUrls":505},"https://about.gitlab.com/blog/puma-nakayoshi-fork-and-compaction","https://about.gitlab.com","article","puma-nakayoshi-fork-and-compaction","en-us/blog/puma-nakayoshi-fork-and-compaction",[511,512,513],"production","performance","inside GitLab","BlogPost","OzW0_9DsUYwOecJAnygnlmO4oSF-bo7AAkDsyj2Kf-A",{"logo":517,"freeTrial":522,"sales":527,"login":532,"items":537,"search":873,"minimal":905,"duo":924,"switchNav":933,"pricingDeployment":944},{"config":518},{"href":519,"dataGaName":520,"dataGaLocation":521},"/","gitlab logo","header",{"text":523,"config":524},"Get free trial",{"href":525,"dataGaName":526,"dataGaLocation":521},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":528,"config":529},"Request a demo",{"href":530,"dataGaName":531,"dataGaLocation":521},"/request-a-demo/","sales",{"text":533,"config":534},"Sign in",{"href":535,"dataGaName":536,"dataGaLocation":521},"https://gitlab.com/users/sign_in/","sign in",[538,624,712,717,797,851],{"text":539,"left":502,"config":540,"menu":542},"Platform",{"dataNavLevelOne":541},"platform",{"type":543,"link":544,"columns":548,"feature":613},"lists",{"text":545,"config":546},"Explore the Platform",{"href":547,"dataGaName":541,"dataGaLocation":521},"/platform/",[549,572,595],{"title":550,"items":551},"DevOps",[552,557,562,567],{"text":553,"config":554},"CI/CD",{"href":555,"dataGaLocation":521,"dataGaName":556},"/solutions/continuous-integration/","ci/cd",{"text":558,"config":559},"Source Code Management",{"href":560,"dataGaLocation":521,"dataGaName":561},"/solutions/source-code-management/","source code management",{"text":563,"config":564},"Agile Delivery",{"href":565,"dataGaLocation":521,"dataGaName":566},"/solutions/agile-delivery/","agile delivery",{"text":568,"config":569},"Artifact Management",{"href":570,"dataGaLocation":521,"dataGaName":571},"/artifact-central/","artifact management",{"title":573,"items":574},"Security & Governance",[575,580,585,590],{"text":576,"config":577},"Application Security Testing",{"href":578,"dataGaLocation":521,"dataGaName":579},"/solutions/application-security-testing/","application security testing",{"text":581,"config":582},"Governance & Compliance",{"href":583,"dataGaLocation":521,"dataGaName":584},"/solutions/software-compliance/","governance and compliance",{"text":586,"config":587},"Secrets Management",{"href":588,"dataGaLocation":521,"dataGaName":589},"/gitlab-secrets-manager/","secrets management",{"text":591,"config":592},"Supply Chain Security",{"href":593,"dataGaLocation":521,"dataGaName":594},"/solutions/supply-chain/","supply chain security",{"title":596,"items":597},"Agentic AI & Context",[598,603,608],{"text":599,"config":600},"Duo Agent Platform",{"href":601,"dataGaLocation":521,"dataGaName":602},"/gitlab-duo-agent-platform/","duo agent platform",{"text":604,"config":605},"Orbit Context Graph",{"href":606,"dataGaLocation":521,"dataGaName":607},"/gitlab-orbit/","orbit",{"text":609,"config":610},"Visibility & Measurement",{"href":611,"dataGaLocation":521,"dataGaName":612},"/solutions/visibility-measurement/","visibility and measurement",{"config":614,"title":617,"text":618,"link":619},{"background":615,"textColor":616},"url('https://res.cloudinary.com/about-gitlab-com/image/upload/v1790185102/o7dn32anqcb6z8dgd3as.jpg')","#000","GitLab Orbit is generally available","Higher agent accuracy, lower token cost",{"text":620,"config":621},"Read the announcement",{"href":622,"dataGaName":623,"dataGaLocation":521},"/blog/gitlab-orbit-ga/","orbit ga announcement",{"text":625,"left":502,"config":626,"menu":628},"Solutions",{"dataNavLevelOne":627},"solutions",{"type":543,"link":629,"columns":633,"feature":703},{"text":630,"config":631},"View all Solutions",{"href":632,"dataGaName":627,"dataGaLocation":521},"/solutions/",[634,652,670],{"title":635,"items":636},"Outcomes",[637,642,647],{"text":638,"config":639},"DevOps Modernization",{"href":640,"dataGaLocation":521,"dataGaName":641},"/assessments/devops-modernization-assessment/","devops modernization",{"text":643,"config":644},"Security Modernization",{"href":645,"dataGaLocation":521,"dataGaName":646},"/assessments/security-modernization-assessment/","security modernization",{"text":648,"config":649},"AI Modernization",{"href":650,"dataGaLocation":521,"dataGaName":651},"/assessments/ai-modernization-assessment/","ai modernization",{"title":653,"items":654},"By size",[655,660,665],{"text":656,"config":657},"Enterprise",{"href":658,"dataGaLocation":521,"dataGaName":659},"/enterprise/","enterprise",{"text":661,"config":662},"Small Business",{"href":663,"dataGaLocation":521,"dataGaName":664},"/small-business/","small business",{"text":666,"config":667},"Startups",{"href":668,"dataGaLocation":521,"dataGaName":669},"/solutions/startups/","startups",{"title":671,"items":672},"Industries",[673,678,683,688,693,698],{"text":674,"config":675},"Financial Services",{"href":676,"dataGaLocation":521,"dataGaName":677},"/solutions/finance/","financial services",{"text":679,"config":680},"Public Sector",{"href":681,"dataGaLocation":521,"dataGaName":682},"/solutions/public-sector/","public sector",{"text":684,"config":685},"Telecommunications",{"href":686,"dataGaLocation":521,"dataGaName":687},"/solutions/telecommunications/","telecommunications",{"text":689,"config":690},"Automotive",{"href":691,"dataGaLocation":521,"dataGaName":692},"/solutions/automotive/","automotive",{"text":694,"config":695},"Education",{"href":696,"dataGaLocation":521,"dataGaName":697},"/solutions/education/","education",{"text":699,"config":700},"Aerospace",{"href":701,"dataGaLocation":521,"dataGaName":702},"/solutions/aerospace/","aerospace",{"config":704,"title":705,"text":706,"link":707},{"background":615,"textColor":616},"GitLab Security Standard","Learn how to harden your software factory and let AI agents earn autonomy, without trading away security.",{"text":708,"config":709},"Explore the standard",{"href":710,"dataGaName":711,"dataGaLocation":521},"/gitlab-security-standard/","gitlab security standard",{"text":713,"config":714},"Pricing",{"href":715,"dataGaName":716,"dataGaLocation":521,"dataNavLevelOne":716},"/pricing/","pricing",{"text":718,"config":719,"menu":721},"Resources",{"dataNavLevelOne":720},"resources",{"type":543,"link":722,"columns":726,"feature":788},{"text":723,"config":724},"View all resources",{"href":725,"dataGaName":720,"dataGaLocation":521},"/resources/",[727,760],{"title":728,"items":729},"Discover",[730,735,740,745,750,755],{"text":731,"config":732},"Docs",{"href":733,"dataGaName":734,"dataGaLocation":521},"https://docs.gitlab.com/","docs",{"text":736,"config":737},"University",{"href":738,"dataGaName":739,"dataGaLocation":521},"https://university.gitlab.com/","university",{"text":741,"config":742},"Demo Series",{"href":743,"dataGaName":744,"dataGaLocation":521},"/gitlab-technical-demo-series/","demo series",{"text":746,"config":747},"Demo Hub",{"href":748,"dataGaName":749,"dataGaLocation":521},"/demo-hub/","demo hub",{"text":751,"config":752},"Services",{"href":753,"dataGaName":754,"dataGaLocation":521},"/services/","services",{"text":756,"config":757},"Well Architected",{"href":758,"dataGaName":759,"dataGaLocation":521},"/resources/well-architected/","well architected",{"title":761,"items":762},"Connect",[763,768,773,778,783],{"text":764,"config":765},"Blog",{"href":766,"dataGaName":767,"dataGaLocation":521},"/blog/","blog",{"text":769,"config":770},"Community",{"href":771,"dataGaName":772,"dataGaLocation":521},"/community/","community",{"text":774,"config":775},"Customers",{"href":776,"dataGaName":777,"dataGaLocation":521},"/customers/","customers",{"text":779,"config":780},"Partners",{"href":781,"dataGaName":782,"dataGaLocation":521},"/partners/","partners",{"text":784,"config":785},"Events",{"href":786,"dataGaName":787,"dataGaLocation":521},"/events/","events",{"config":789,"title":790,"text":791,"link":792},{"background":615,"textColor":616},"What’s new in GitLab","Stay updated with our latest features and improvements.",{"text":793,"config":794},"Read the latest",{"href":795,"dataGaName":796,"dataGaLocation":521},"/whats-new/","whats new",{"text":798,"config":799,"menu":801},"Company",{"dataNavLevelOne":800},"company",{"type":543,"columns":802},[803],{"items":804},[805,810,816,821,826,831,836,841,846],{"text":806,"config":807},"About",{"href":808,"dataGaName":809,"dataGaLocation":521},"/company/","about",{"text":811,"config":812,"footerGa":815},"Jobs",{"href":813,"dataGaName":814,"dataGaLocation":521},"/jobs/","jobs",{"dataGaName":814},{"text":817,"config":818},"Press",{"href":819,"dataGaName":820,"dataGaLocation":521},"/press/","press",{"text":822,"config":823},"Handbook",{"href":824,"dataGaName":825,"dataGaLocation":521},"https://handbook.gitlab.com/","handbook",{"text":827,"config":828},"Leadership",{"href":829,"dataGaName":830,"dataGaLocation":521},"/company/team/e-group/","leadership",{"text":832,"config":833},"Investor relations",{"href":834,"dataGaName":835,"dataGaLocation":521},"https://ir.gitlab.com/overview/default.aspx","investor relations",{"text":837,"config":838},"Trust Center",{"href":839,"dataGaName":840,"dataGaLocation":521},"/security/","trust center",{"text":842,"config":843},"AI Transparency Center",{"href":844,"dataGaName":845,"dataGaLocation":521},"/ai-transparency-center/","ai transparency center",{"text":847,"config":848},"Newsletter",{"href":849,"dataGaName":850,"dataGaLocation":521},"/company/contact/#contact-forms","newsletter",{"text":852,"config":853,"menu":854},"Contact us",{"dataNavLevelOne":800},{"type":543,"columns":855},[856],{"items":857},[858,863,868],{"text":859,"config":860},"Talk to sales",{"href":861,"dataGaName":862,"dataGaLocation":521},"/sales/","talk to sales",{"text":864,"config":865},"Support portal",{"href":866,"dataGaName":867,"dataGaLocation":521},"https://support.gitlab.com/hc/en-us","support portal",{"text":869,"config":870},"Customer portal",{"href":871,"dataGaName":872,"dataGaLocation":521},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":874,"login":875,"suggestions":882},"Close",{"text":876,"link":877},"To search repositories and projects, login to",{"text":878,"config":879},"gitlab.com",{"href":535,"dataGaName":880,"dataGaLocation":881},"search login","search",{"text":883,"default":884},"Suggestions",[885,888,891,893,897,901],{"text":886,"config":887},"GitLab Duo Agent Platform",{"href":601,"dataGaName":886,"dataGaLocation":881},{"text":889,"config":890},"Code Suggestions (AI)",{"href":601,"dataGaName":889,"dataGaLocation":881},{"text":553,"config":892},{"href":555,"dataGaName":553,"dataGaLocation":881},{"text":894,"config":895},"GitLab on AWS",{"href":896,"dataGaName":894,"dataGaLocation":881},"/partners/technology-partners/aws/",{"text":898,"config":899},"GitLab on Google Cloud",{"href":900,"dataGaName":898,"dataGaLocation":881},"/partners/technology-partners/google-cloud-platform/",{"text":902,"config":903},"Why GitLab?",{"href":904,"dataGaName":902,"dataGaLocation":881},"/why-gitlab/",{"freeTrial":906,"mobileIcon":911,"desktopIcon":916,"secondaryButton":919},{"text":907,"config":908},"Start free trial",{"href":909,"dataGaName":526,"dataGaLocation":910},"https://gitlab.com/-/trials/new/","nav",{"altText":912,"config":913},"Gitlab Icon",{"src":914,"dataGaName":915,"dataGaLocation":910},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":912,"config":917},{"src":918,"dataGaName":915,"dataGaLocation":910},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":920,"config":921},"Get Started",{"href":922,"dataGaName":923,"dataGaLocation":910},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/get-started/","get started",{"freeTrial":925,"mobileIcon":929,"desktopIcon":931},{"text":926,"config":927},"Learn more about GitLab Duo",{"href":601,"dataGaName":928,"dataGaLocation":910},"gitlab duo",{"altText":912,"config":930},{"src":914,"dataGaName":915,"dataGaLocation":910},{"altText":912,"config":932},{"src":918,"dataGaName":915,"dataGaLocation":910},{"button":934,"mobileIcon":939,"desktopIcon":941},{"text":935,"config":936},"/switch",{"href":937,"dataGaName":938,"dataGaLocation":910},"#contact","switch",{"altText":912,"config":940},{"src":914,"dataGaName":915,"dataGaLocation":910},{"altText":912,"config":942},{"src":943,"dataGaName":915,"dataGaLocation":910},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1773335277/ohhpiuoxoldryzrnhfrh.png",{"freeTrial":945,"mobileIcon":950,"desktopIcon":952},{"text":946,"config":947},"Back to pricing",{"href":715,"dataGaName":948,"dataGaLocation":910,"icon":949},"back to pricing","GoBack",{"altText":912,"config":951},{"src":914,"dataGaName":915,"dataGaLocation":910},{"altText":912,"config":953},{"src":918,"dataGaName":915,"dataGaLocation":910},{"title":955,"button":956,"config":960},"GitLab Orbit is generally available. Increase agent accuracy and lower token cost",{"text":620,"config":957},{"href":622,"dataGaName":958,"dataGaLocation":959},"orbit-ga-banner","global-banner",{"layout":961,"disabled":499},"release",{"data":963},{"text":964,"config":965,"items":970,"minimal":1198},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"twitter":966,"facebook":967,"youtube":968,"linkedin":969},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[971,1019,1078,1125,1166],{"title":713,"links":972,"subMenu":988},[973,978,983],{"text":974,"config":975},"View plans",{"href":715,"dataGaName":976,"dataGaLocation":977},"view plans","footer",{"text":979,"config":980},"Why Premium?",{"href":981,"dataGaName":982,"dataGaLocation":977},"/pricing/premium/","why premium",{"text":984,"config":985},"Why Ultimate?",{"href":986,"dataGaName":987,"dataGaLocation":977},"/pricing/ultimate/","why ultimate",[989],{"title":990,"links":991},"Contact Us",[992,995,997,999,1004,1009,1014],{"text":993,"config":994},"Contact sales",{"href":861,"dataGaName":531,"dataGaLocation":977},{"text":864,"config":996},{"href":866,"dataGaName":867,"dataGaLocation":977},{"text":869,"config":998},{"href":871,"dataGaName":872,"dataGaLocation":977},{"text":1000,"config":1001},"Status",{"href":1002,"dataGaName":1003,"dataGaLocation":977},"https://status.gitlab.com/","status",{"text":1005,"config":1006},"Terms of use",{"href":1007,"dataGaName":1008,"dataGaLocation":977},"/terms/","terms of use",{"text":1010,"config":1011},"Privacy statement",{"href":1012,"dataGaName":1013,"dataGaLocation":977},"/privacy/","privacy statement",{"text":1015,"config":1016},"Cookie preferences",{"dataGaName":1017,"dataGaLocation":977,"id":1018,"isOneTrustButton":502},"cookie preferences","ot-sdk-btn",{"title":539,"links":1020,"subMenu":1036},[1021,1024,1026,1028,1030,1032],{"text":1022,"config":1023},"Explore the platform",{"href":547,"dataGaName":541,"dataGaLocation":977},{"text":599,"config":1025},{"href":601,"dataGaName":602,"dataGaLocation":977},{"text":553,"config":1027},{"href":555,"dataGaName":556,"dataGaLocation":977},{"text":604,"config":1029},{"href":606,"dataGaName":607,"dataGaLocation":977},{"text":586,"config":1031},{"href":588,"dataGaName":589,"dataGaLocation":977},{"text":1033,"config":1034},"Why GitLab",{"href":904,"dataGaName":1035,"dataGaLocation":977},"why gitlab",[1037],{"title":1038,"links":1039},"Topics",[1040,1044,1049,1053,1058,1063,1068,1073],{"text":553,"config":1041},{"href":1042,"dataGaName":1043,"dataGaLocation":977},"/topics/ci-cd/","cicd",{"text":1045,"config":1046},"GitOps",{"href":1047,"dataGaName":1048,"dataGaLocation":977},"/topics/gitops/","gitops",{"text":550,"config":1050},{"href":1051,"dataGaName":1052,"dataGaLocation":977},"/topics/devops/","devops",{"text":1054,"config":1055},"Version Control",{"href":1056,"dataGaName":1057,"dataGaLocation":977},"/topics/version-control/","version control",{"text":1059,"config":1060},"DevSecOps",{"href":1061,"dataGaName":1062,"dataGaLocation":977},"/topics/devsecops/","devsecops",{"text":1064,"config":1065},"Cloud Native",{"href":1066,"dataGaName":1067,"dataGaLocation":977},"/topics/cloud-native/","cloud native",{"text":1069,"config":1070},"AI for Coding",{"href":1071,"dataGaName":1072,"dataGaLocation":977},"/topics/devops/ai-for-coding/","ai for coding",{"text":1074,"config":1075},"Agentic AI",{"href":1076,"dataGaName":1077,"dataGaLocation":977},"/topics/agentic-ai/","agentic ai",{"title":625,"links":1079},[1080,1083,1086,1089,1092,1095,1097,1099,1101,1103,1105,1108,1110,1113,1117,1121],{"text":1081,"config":1082},"DevOps modernization",{"href":640,"dataGaName":641,"dataGaLocation":977},{"text":1084,"config":1085},"Security modernization",{"href":645,"dataGaName":646,"dataGaLocation":977},{"text":1087,"config":1088},"AI modernization",{"href":650,"dataGaName":651,"dataGaLocation":977},{"text":1090,"config":1091},"Financial services",{"href":676,"dataGaName":677,"dataGaLocation":977},{"text":1093,"config":1094},"Public sector",{"href":681,"dataGaName":682,"dataGaLocation":977},{"text":684,"config":1096},{"href":686,"dataGaName":687,"dataGaLocation":977},{"text":689,"config":1098},{"href":691,"dataGaName":692,"dataGaLocation":977},{"text":694,"config":1100},{"href":696,"dataGaName":697,"dataGaLocation":977},{"text":699,"config":1102},{"href":701,"dataGaName":702,"dataGaLocation":977},{"text":656,"config":1104},{"href":658,"dataGaName":659,"dataGaLocation":977},{"text":1106,"config":1107},"Small business",{"href":663,"dataGaName":664,"dataGaLocation":977},{"text":666,"config":1109},{"href":668,"dataGaName":669,"dataGaLocation":977},{"text":1045,"config":1111},{"href":1112,"dataGaName":1048,"dataGaLocation":977},"/solutions/gitops/",{"text":1114,"config":1115},"Software composition analysis",{"href":593,"dataGaName":1116,"dataGaLocation":977},"software composition analysis",{"text":1118,"config":1119},"Value stream management",{"href":611,"dataGaName":1120,"dataGaLocation":977},"value stream management",{"text":1122,"config":1123},"All solutions",{"href":632,"dataGaName":1124,"dataGaLocation":977},"all solutions",{"title":718,"links":1126},[1127,1132,1137,1139,1141,1143,1145,1147,1149,1151,1153,1155,1157,1159,1162],{"text":1128,"config":1129},"Install",{"href":1130,"dataGaName":1131,"dataGaLocation":977},"/install/","install",{"text":1133,"config":1134},"Quick start guides",{"href":1135,"dataGaName":1136,"dataGaLocation":977},"/get-started/","quick setup checklists",{"text":731,"config":1138},{"href":733,"dataGaName":734,"dataGaLocation":977},{"text":736,"config":1140},{"href":738,"dataGaName":739,"dataGaLocation":977},{"text":741,"config":1142},{"href":743,"dataGaName":744,"dataGaLocation":977},{"text":746,"config":1144},{"href":748,"dataGaName":749,"dataGaLocation":977},{"text":751,"config":1146},{"href":753,"dataGaName":754,"dataGaLocation":977},{"text":764,"config":1148},{"href":766,"dataGaName":767,"dataGaLocation":977},{"text":769,"config":1150},{"href":771,"dataGaName":772,"dataGaLocation":977},{"text":774,"config":1152},{"href":776,"dataGaName":777,"dataGaLocation":977},{"text":779,"config":1154},{"href":781,"dataGaName":782,"dataGaLocation":977},{"text":784,"config":1156},{"href":786,"dataGaName":787,"dataGaLocation":977},{"text":847,"config":1158},{"href":849,"dataGaName":850,"dataGaLocation":977},{"text":1160,"config":1161},"What's new",{"href":795,"dataGaName":796,"dataGaLocation":977},{"text":1163,"config":1164},"All resources",{"href":725,"dataGaName":1165,"dataGaLocation":977},"all resources",{"title":798,"links":1167},[1168,1170,1172,1174,1176,1178,1180,1182,1184,1188,1193],{"text":806,"config":1169},{"href":808,"dataGaName":800,"dataGaLocation":977},{"text":811,"config":1171},{"href":813,"dataGaName":814,"dataGaLocation":977},{"text":817,"config":1173},{"href":819,"dataGaName":820,"dataGaLocation":977},{"text":822,"config":1175},{"href":824,"dataGaName":825,"dataGaLocation":977},{"text":827,"config":1177},{"href":829,"dataGaName":830,"dataGaLocation":977},{"text":832,"config":1179},{"href":834,"dataGaName":835,"dataGaLocation":977},{"text":837,"config":1181},{"href":839,"dataGaName":840,"dataGaLocation":977},{"text":842,"config":1183},{"href":844,"dataGaName":845,"dataGaLocation":977},{"text":1185,"config":1186},"Sustainability",{"href":1187,"dataGaName":1185,"dataGaLocation":977},"/sustainability/",{"text":1189,"config":1190},"Diversity, inclusion and belonging (DIB)",{"href":1191,"dataGaName":1192,"dataGaLocation":977},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":1194,"config":1195},"Modern Slavery Transparency Statement",{"href":1196,"dataGaName":1197,"dataGaLocation":977},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"items":1199},[1200,1203,1206],{"text":1201,"config":1202},"Terms",{"href":1007,"dataGaName":1008,"dataGaLocation":977},{"text":1204,"config":1205},"Cookies",{"dataGaName":1017,"dataGaLocation":977,"id":1018,"isOneTrustButton":502},{"text":1207,"config":1208},"Privacy",{"href":1012,"dataGaName":1013,"dataGaLocation":977},[1210],{"id":1211,"title":1212,"body":498,"config":1213,"content":1215,"description":498,"extension":1219,"meta":1220,"navigation":502,"path":1221,"seo":1222,"stem":1223,"__hash__":1224},"blogAuthors/en-us/blog/authors/matthias-kppler.yml","Matthias Kppler",{"template":1214},"BlogAuthor",{"name":7,"config":1216},{"headshot":1217,"ctfId":1218},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749670351/Blog/Author%20Headshots/mkaeppler-headshot.jpg","mkaeppler","yml",{},"/en-us/blog/authors/matthias-kppler",{},"en-us/blog/authors/matthias-kppler","ifYoCrN_DXBz-4NxWtEPrkhCLzL27iwXIaXEy_ogYFc",[1226,1234,1242],{"title":1227,"description":1228,"heroImage":1229,"category":494,"date":1230,"authors":1231,"slug":1233,"externalUrl":498},"Two front doors: Module-level access in a Django GRC app","A technical deep-dive on restructuring a Django internal tool to support two distinct GitLab teams: Security Compliance and Internal Audit.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1750097663/Blog/Hero%20Images/Blog/Hero%20Images/Workflow%201800x945_2gQoQIbY9NvjLFpXtsxtXy_1750097663612.png","2026-10-05",[1232],"Donovan Felton","module-level-access-in-a-django-grc-app",{"title":1235,"description":1236,"heroImage":1237,"category":494,"date":1238,"authors":1239,"slug":1241,"externalUrl":498},"How GitLab reduced code-per-agentic-flow ratio by 45%","Learn how Flow Registry, a component-based framework that compiles YAML configurations into fully functional LangGraph flows, helped improve scalability and more.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1774465167/n5hlvrsrheadeccyr1oz.png","2026-09-22",[1240],"Mikolaj Wawrzyniak","how-gitlab-reduced-code-per-agentic-flow-ratio",{"title":1243,"description":1244,"heroImage":1245,"category":494,"date":1246,"authors":1247,"slug":1249,"externalUrl":498},"Co-Create: Building GitLab with our users","How GitLab users collaborated with us to build practical product improvements in the first half of 2026.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1750099203/Blog/Hero%20Images/Blog/Hero%20Images/blog-image-template-1800x945%20%2820%29_2bJGC5ZP3WheoqzlLT05C5_1750099203484.png","2026-09-10",[1248],"Isa Huerga","co-create-h1-2026",{"header":1251,"text":1252,"button":1256,"disclaimer":1259},"Start your free\n\n30-day GitLab trial\n",[1253,1254,1255],"GitLab Duo Agent Platform access","Try our most advanced features","Automate complex tasks with AI",{"text":523,"config":1257},{"href":1258,"dataGaName":526,"dataGaLocation":767},"https://gitlab.com/-/trials/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/blog/","No credit card required.",{"header":1261,"blurb":1262,"button":1263,"secondaryButton":1268},"Start building faster today","See what your team can do with the intelligent orchestration platform for DevSecOps.\n",{"text":1264,"config":1265},"Get your free trial",{"href":1266,"dataGaName":526,"dataGaLocation":1267},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":993,"config":1269},{"href":861,"dataGaName":531,"dataGaLocation":1267},1794911796560]