In the project environment of bahn.de and DB Navigator, we have migrated our pipeline engine from (several) Jenkins instances and hundreds of pipeline jobs to GitLab CI over the past few months. Due to the large number of instances, pipeline variants, and special features, this was a mammoth task that took us a lot of time and effort. Nevertheless, the migration was naturally associated with high expectations: standardization and unification of the development infrastructure, reduction of build infrastructure costs, reduction of cognitive load in the teams, and less exposure to security vulnerabilities.

Conceptually, there are some differences between the two platforms (is one really “better” than the other?), which we want to discuss. In addition to the reasons for our decision to migrate and our lessons learned, we want to provide insights into how to find a viable build infrastructure with GitLab CI (especially GitLab CI Components) in a larger organizational environment (20+ teams with 120+ developers). Last but not least, we also want to address the things we miss from Jenkins and find annoying about GitLab CI, and finally answer the question of whether we regret the switch or would do it again with the knowledge we have today.

Pipeline Engines

In modern software development, your pipeline engine is a critical component in your tool chain because manual builds and deploys have essentially ceased to exist, and more often than not have become virtually impossible due to the complexity and size of typical current pipelines. Therefore, their most important requirements are availability, reliability, convenience, and versatility.

In our case, we had begun building our new platform around 2017, a time when Jenkins was in high demand in the Java ecosystem in particular, and GitLab CI was still in its very early days and only had beta status. Because we wanted to get our development teams running as quickly as possible, we provided them with a basic Ansible setup to quickly spin up their own Jenkins instances, and for the most part, let them customize and configure them as they saw fit.

This approach worked well for us in the early stages, but as we got closer to shipping services towards production, we had to start bullet-proofing our Jenkins instances. Suddenly, our teams had to guarantee that their build servers were always available in case of required hot-fixes; they needed to make sure that they could be recovered within just a few hours in case of catastrophic infrastructure disaster events. Furthermore, they had to continuously monitor them for recent security vulnerabilities, i.e., frequently update and patch them, which could occasionally be very time-consuming. We noticed that the cognitive load of our teams increased considerably as they had to invest a growing portion of their capacity into running their build servers.

Lastly, the diversity in instances which were highly adapted and optimized for the specific needs and habits of a single team turned into a sizeable obstacle whenever we needed to rotate engineers into different teams, as they had to dig into the specific ins and outs every time.

Choosing a migration approach

Mainly due to the size of our organization and our requirements, we quickly agreed that a single scalable managed build server would be best suited for our needs. In our case, we have more than 20 teams maintaining over 100 microservices, each of which requires multiple daily deployments, so we required a build server that could sustain 100+ concurrent builds during peak hours. Naturally, this could have been achieved with a managed Jenkins instance set up adequately as well, or with the help of e.g. CloudBees.

However, in our case, we had been using (and paying for) GitLab as our SCM from the start. We were hoping that by simply using it more intensely, we would also be able to cut our infrastructure expenses a bit, because GitLab runners allow for very precise dynamic scaling. In contrast, with Jenkins dynamic agents, this can be quite tricky. The cloud instances for our various Jenkins masters had been on-demand due to our hotfix readiness requirement, and therefore, were rather costly. The same requirement served as a credible justification for having a platform team with the sole and clear purpose of ensuring a stable SCM and pipeline system for all of our development teams.

While the service pipelines themselves were very homogeneous among services within the same domain or team, there were vast differences between pipelines from different domains and teams. To address this, each team had developed and maintained a shared Jenkins pipeline library^(1). In declarative Jenkins pipelines, these allow you to reuse steps or entire stages of a pipeline without having to copy and paste pipeline code everywhere, which makes things a nightmare to maintain and update. They can also be versioned and tested, which makes them a very powerful tool for maintaining your pipelines.

GitLab CI provides a similar mechanism with CI/CD components^(2). These are basically reusable pipeline building blocks that can also be configured via input arguments and versioned to ensure stability and continuity in the development of your pipelines. Furthermore, components can be published into the CI/CD catalog^(3), which allows other teams to use pipeline components as well. We found this to be exceptionally useful in cases where pipeline steps are truly universal (e.g., static code analysis via Sonar or static code analysis for security vulnerability with managed tools), so that it suffices to implement them once for all teams rather than every team having to reproduce them in their respective components.

Since all our Jenkins pipelines were declarative and built using shared pipeline libraries and we saw no need to change their overall structure, the actual migration was very straight forward: after taking inventory of the building blocks in the shared pipeline library that we decided to keep, it came down to just rewriting them into GitLab CI components (so basically migrating from Groovy scripts to shell scripts ran in suitable docker containers). Therefore, the migration could be broken down into very small, mostly independent tasks that could be performed in parallel, which made estimation and planning very easy after we had gathered some initial experience.

Since GitLab CI components are built and published into the catalog with a GitLab pipeline themselves, we had the bonus of being able to properly and thoroughly test our components by executing them in this very pipeline which actually helped us identify a few bugs along the way as well and improved overall quality of our pipeline code which had previously been treated somewhat sloppily because it was not actual application code - a fallacy many software engineers fall for, but can prove fatal. A painful lesson everyone learns once their faulty pipeline code ships code to production that should have broken your pipeline.

Viable build infrastructure in large organizations

In our case, the migration has certainly paid off. We were able to shut off most of our infrastructure related to Jenkins by now, which has resulted in considerable savings even after you factor in the increased demand in GitLab runner infrastructure. Our development teams have been unchained from the burden of having to maintain and run their build infrastructure and have considerably improved their feature output. The dedicated platform team is focused on their single task, and they are very good at it - unlike before, we have not had any build server downtimes worth mentioning beyond scheduled maintenance windows.

We found that with the way our pipelines work, execution times have actually reduced with the shift, and success rate has improved as well, which was unexpected considering that we are seeing almost 1 million pipeline runs a month. As it turns out, most of our previous Jenkins instances suffered occasional “hiccups” that needed manual fixing every once in a while.

While Jenkins allowed for a near limitless degree of customization, our teams agreed unanimously that they are glad to be rid of the plugin version compatibility issues that plague Jenkins to this day. When we found bugs or lacked certain features in Jenkins or one of its plugins, we would occasionally contribute changes, which was convenient as Jenkins is entirely open source. However, to a large extent, the same holds true for GitLab, which is “open core” and allows for community contributions. When we were unable to solve issues on our own, enterprise support can certainly be a worthwhile investment if your organization is big enough.

Due to the way our pipelines work, we frequently use the “restart from stage” action in Jenkins to pick up on pipelines that were failed or suspended for various reasons. This action was always flaky in Jenkins (in particular if nodes were shut off or restarted in between pipeline runs), whereas the GitLab CI appears to do a much better job at properly persisting and restoring workspaces from single stages. More generally, in direct comparison, we found that Jenkins appears to be way more stateful, with all its implications, whereas GitLab CI stages are truly isolated and independent, built on a very simple foundation of Docker containers and shell scripts within them.

While plugins can be one of the most annoying traits of Jenkins (especially when they are poorly maintained or documented), they are occasionally also among the most useful. To this day, we dearly miss the plugins that provide clear graphic reports from test results, for example, of Cucumber integration tests or Gatling performance tests - a feature that GitLab sadly lacks to this day.

In general, however, you will find overwhelming amounts of publicly available troubleshooting help for Jenkins-related issues in the usual channels, whereas GitLab CI still has some catching up to do.

Conclusion

In summary, we have concluded that for large-scale software development organizations, a managed pipeline engine instance is highly advisable to optimize infrastructure costs, standardize among development teams, account for personnel shifts, and keep cognitive load focused on the feature tasks at hand. While neither Jenkins nor GitLab CI is inherently better, both are more or less equally capable of scaling and enabling the demands of enterprises, in particular if you can opt for enterprise support from either GitLab or CloudBees (or their likes).

In hindsight, we have not regretted our expensive and taxing migration effort as our development has seen the boost we were hoping for, while reducing our costs and making our build server more robust.

As with all major migration tasks, the decision of swapping your pipeline engine should not be made lightly. Carefully consider your priorities, requirements, and external circumstances, and weigh your options before arriving at your decision. While both systems have their pros and cons, no universal recommendation can be made, as the specifics of any software organization are unique and must be taken into account. Even so, there are cases in which the overall outcome and improvements outweigh the considerable effort and investment.

References

This article is part of the JAVAPRO special magazine issue:

Java in the Age of AI

Explore how AI is transforming the way we build, secure, and operate software with Java.
From AI agents and new architectural patterns to security, data, and team dynamics—this edition brings together real-world insights for building intelligent, production-ready systems.

**Discover the edition **→