Using Pipelines for CI/CD
Continuous integration and continuous deployment have become very important in the daily workflow of building a product to production. In medium to large organizations, developer operations are often done by a person other than a developer, but it is still necessary to have a good idea of the stages and processes. We can divide the stages of the flow into four stages: the build stage, the test stage, the release stage, and the deploy stage. It is necessary to follow the stages and in the correct order. Continuous integration involves the first three stages, and the last stage is continuous deployment. We know the need for pipelines—from stopping errors and some bugs from going to production to easy versioning and an automated deployment process.
The build stage compiles the uploaded code and on completion goes to the test stage where the code is tested with test scripts to ensure expected behaviors. After this, the build can be merged to the main repository from the developer’s branch, which is the release stage. Next comes the deployment stage; and if everything goes, it moves to the deployment stage where you can deploy the main build to any platform or service. Another minor step is the validation where a check for compliance and vulnerabilities is done.
Like many things in IT, it is easy to start and hard to build something of even moderate complexity. It is now very popular to build pipelines. We have many tools we can use for CI/CD—among them are Jenkins, Circle CI, Github actions, and Azure DevOps pipelines. Each has its own unique features and use cases, but I’ll pick Azure DevOps because of its integration to GitHub, easy user interface when setting up pipelines, and the high number of concurrent jobs and long minutes of build that are possible.
So let’s look at a workflow that uses an external API in Node.js using a GitHub repository and the GCP app engine for deployment and uses the Azure DevOps pipelines for build and deployment orchestration.
Azure DevOps works well with GitHub, so you can get your repositories from Azure DevOps and make changes to configure a CI/CD build. After the continuous integration, you can also add a deploy stage to deploy the GCP app engine, which should definitely include using a service account certificate to allow for deploy access to the GCP resource.
All of these are easy to find but work mostly for tightly coupled systems and loosely coupled products using a lot of microservices and serverless systems. This brings up a worthy mention of serverless CI/CD for those managing their services with serverless, but it doesn’t invalidate the use of other tools for CI/CD even in applications with microservice architecture.
There is a great need for CI/CD pipeline tools as it has become the norm and status quo when setting a product build to production workflow, but it still requires more central management for multiple workflows or pipelines, especially in microservices architectures. Sometimes it would be overkill to build and configure pipelines for test projects or personal tools as there isn’t a lot of versioning and updates going on when it’s just you—except when you’re learning to build pipelines of course.
