For any software development team is critical to delivering value as quickly as possible, safely and reliably. It is proven that speed of delivery is directly correlated with the organisation performance (see, e.g. State of DevOps report. So the delivery process influence company valuation and is critical for scaling the engineering effort withholding the desired quality of the product. How to achieve this is one of the cornerstones of DevOps and SRE. To get an idea of how the modern software delivery works in a successful company, see how delivery pipeline works in AWS.
This blog post is by no means a replacement for deep-dive specialised or best practice literature, e.g. Continous Delivery, Continous Integration but rather an evaluation of the current tooling landscape which can help in achieving project goals. No need to mention that tool alone won’t make the magic happen without correct delivery pipeline design. But in the post, we will focus solely on the tooling.

CI/CD key terms clarification

Continuous integration

Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project.

Continuous Delivery

Continuous Delivery (CD) is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments— into production, or into the hands of users, safely and quickly in a sustainable way.

Continuous deployment

Continuous deployment (CD) is a strategy for software releases wherein any code commit that passes the automated testing phase is automatically released into the production environment. It is paramount to software delivery processes. 

The delivery process is critical in any software company. From my perspective and experience current state is far from being “solved”, and the number of tools appearing every year confirms that. The amount of money spend by VCs is just confirming that. The majority of tools are imperative, while the next big trend seems to be a “declarative” CI/CD tooling. Curious about what the future will bring.

CI/CD tools available

Wide variety of tools available (by no means list is extensive):

• Jenkins
• TeamCity
• CircleCI
• Concourse
• Harness
• GoCD
• Spinnaker
• ArgoCD
• Google Cloud Build
• GitLab
• GitHub actions
• … and sure there are others and new will arise …

Selection criteria

Our selection and evaluation criteria base on our current and future needs:

• cost-effective (auto-scaling workers, etc.)
• cost of maintenance
• speed of development/ability to contribute
• manual approval stage
• ability to pass certification (audit-ability, permission and roles, etc.)
• multi-cloud support
• support VMs + kubernetes deployments + potentially serverless
• ability to integrate Infrastructure as Code to delivery pipeline
• do not scratch all our development infra (keep in mind cost/benefit ratio)
• majority of our workloads are running in GCP
• deals with mono-repo
• support for long term support (LTS) branches

CI/CD tools shortlist

Following tools made it into shortlist for evaluation and deep dive. See dedicated post for each of those:

• Jenkins evaluation
• GitLab evaluation
• Cloud Build evaluation
• Spinnaker evaluation

Summary

Our ideal solution would be tooling provided by our primary cloud provider, which meets our current and near feature needs and is fully managed. We partially matched that with a combination of Cloud Build and Spinnaker for GCP based on tutorial provided by GCP.
Generally, my impression from the study and evaluation of tools listed is that claim of “full CI/CD” support are neither great in CI nor CD and lay somewhere in the middle. They provide a platform a let you code the rest. Another pain point is to tackle the monorepo and provide a means to be efficient. Platforms seem to be somewhat pricy, and the amount of infra work needed is not that low to justify it when providing all necessary features. Curious about what the Harness will provide in this space.
Not promoting the combination with end up with but was clear win moving away from Concourse CI. Where missing resource management for stages was a total killer, insufficient authorisation and role management and absence of manual steps was clear do not continue this journey. For a fresh new project, a GitLab would be a brainer to start with. It provides all needed for development, but when the project grows significantly, it can become pricy, and you are motivated even by GitLab to move partially to your infrastructure. Needless to say, that setup requires some amount of work, especially proxying and create network waypoints.
If you have some experiences with tools evaluated or disagree with the points, please use the comment section to share your view and don’t forget to like and follow me on Twitter!