API, Microservices
Progressive deployment of microservices with IDP
Infosys is currently involved in a digital transformation program in the retirement domain catering to five million participants and 1,500 sponsors. This initiative necessitates a comprehensive deployment of microservices on a public cloud platform.
Insights
- As we construct the microservices architecture, the continuous integration (CI)/continuous deployment (CD) pipeline emerges as a vital and critical component of the modern cloud platform.
- In our endeavor to ensure that the CI/CD pipeline aligns with diverse needs, including technology stacks, business models, compliance, security, and seamless integration with collaboration tools and platforms, we have created a successful model.
- This model is adaptable and can be implemented by any large-scale program.
This document covers CI/ CD, GitOps, and progressive delivery in a large microservices architecture for a leading retirement benefits client. It details a reference architecture that fulfills all main requirements using modern components from the cloud native computing foundation (CNCF) landscape. Infosys DevSecOps Platform (IDP) serves as the core tool to create pipelines.
For deployment and rollouts, we have selected Argo Rollouts, providing a short comparison to tools like Flagger for teams considering the adoption of progressive delivery principles. Finally, we cover integration with Service Now (SNOW) to ensure that all deployments to higher environments, typically production and DR, undergo testing and approval. This approach guarantees that software versions are tested and approved, ensuring traceability in the software supply chain.
This paper addresses decision makers overseeing the foundational elements of CI/CD and deployment tools. We expect the following takeaways:
- A recognition of the necessity for automated DevSecOps across the lifecycle of a cloud native architecture.
- A best-in-class stack with projects from the CNCF landscape deployed around IDP as the central tool for building pipelines.
Navigating microservices architecture challenges
Many organizations have piloted or migrated applications to a microservices architecture, reaping benefits such as enhanced agility and increased throughput. Microservices architecture are mostly container-based with Kubernetes as the orchestration platform. Kubernetes offers enterprises with built-in features like health checks, resilience, and high availability, making container- and microservices-based architectures robust and fault-tolerant.
However, this architecture presents certain implementation challenges:
- By design, a microservice architecture fragments monoliths into smaller, independently deployable services on the container platform. This independent deployment of small, orthogonal services enhances the architecture’s agility. However, the decomposition of a monolith into microservices results in a proliferation of components that require effective management and version control.
- In contrast to monoliths, a container-based architecture requires additional layers. Monoliths run typically on virtual machines with a hypervisor running on bare metal. Conversely, containers require an additional runtime on top of a hypervisor, and orchestration requires an extensive set of components provided by Kubernetes.
These factors complicate versioning and configuration of a microservice architecture. Other challenges involve ensuring service resilience, even in the face of network failures.
Legacy tools and approaches are ill-equipped for this new world. A typical team that maintains and versions around a dozen components in a monolithic architecture, including the operating system version and application version, may now find themselves tasked with overseeing and versioning hundreds of components. This expanded scope includes the host OS version, Kubernetes version, system component versions, and the microservices themselves.
The rest of this paper performs a deep dive into the technical concerns of progressive, automated deployment of the entire microservices stack using Argo and the Infosys Cobalt IDP tool.
A layered cloud native architecture with Kubernetes
A cloud native architecture with Kubernetes as the platform consists of various layers.
- Kubernetes foundation: This fundamental Kubernetes element controls and manages the entire cluster, orchestrating containers and managing the system’s overall state. The Kubernetes layer consists of components within the control plane and a set of machines called nodes. In a cloud vendor scenario, a vendor manages and provides the control plane.
- Networking and storage: Enable communication among containers, manage networking resources, and provide storage solutions for applications running in a Kubernetes cluster. Networking layer consists of pod networking, service networking, and container networking interface (CNI) like Calico and Flannel & Weave. Storage layer comprises persistent volume (PV), persistent volume claim (PVC), and static/dynamic volume provisioning.
- Logging and monitoring: Collect, store, and analyze logs and metrics from applications and components in a Kubernetes cluster. These components are crucial to ensure observability and performance of the cluster and its applications.
- Security: Focuses on implementing measures to secure the Kubernetes cluster, its components, and applications running within it. Key aspects of the security layer are authentication, authorization, network policies, secrets management, pod security policies, service account isolation, image security, pod security contexts, and network segmentation.
- Operation: Refers to practices, tools, and processes used to manage and maintain the Kubernetes cluster throughout its lifecycle. It involves various operational tasks to ensure the cluster's reliability, scalability, performance, and overall health.
Enter DevSecOps, a key tenet of a cloud native architecture that automates the versioning and rollout of the various components of a microservice architecture starting with the platform.
Successful DevSecOps teams embrace the following concepts:
- Shift-left: Involves pushing concerns such as security and operational ease to the earliest stages of the application lifecycle. For example, using code scanning tools to ensure secure coding practices from the outset.
- Observability: Ensures continuous monitoring of the entire technology stack, from hardware to end-user applications, with automated alerts for issue detection.
- Full life cycle automation: Confirms operational concerns such as deployments are automated from the development environment onward in a zero-touch fashion.
- GitOps: Ensures the current state of the entire technology stack is always available in a version-controlled system as code.
In this client engagement, we used IDP as the central tool for this capability. Deployments and changes for all layers of the architecture, from the underlying Kubernetes clusters to applications, were done using IDP pipelines. While initially applied in the financial services sector, this architecture and implementation are industry-agnostic, offering versatility across our entire client base.
Continuous integration/continuous delivery
In modern technology environments, the norm is to incrementally promote and integrated new functionality in higher environments daily or even multiple times a day. Modern agile teams comprise members from requirements, development, testing, and site reliability engineering (SRE) teams. Tasks such as development, integration, and testing progress within the team such that completed functionality is available at the end of a sprint. Automated CI/ CD is imperative in such an approach to reduce friction and achieve optimal velocity. We successfully delivered end-to-end automation in CI/ CD for this client, using pipelines built with IDP (Figure x).
The containerized application pipeline is pivotal in our program, orchestrating the creation and deployment of docker images on Kubernetes (EKS on AWS) via application-specific manifests. Argo Rollouts is a Kubernetes controller and toolset for managing rolling updates and canary deployments of applications running in Kubernetes clusters. SAST (#2, #3) and image scanning (#6) are seamlessly integrated into the pipeline as mandatory stages.
The security team performs DAST on the running application (#10), as per guidelines in the Web-Inspect DAST implementation (CD Flow) section of this document.
Though CI/ CD originally rose to prominence for application deployments, it is now used for all layers of the technology stack. For example, the Cluster API project - https://github.com/kubernetes-sigs/cluster-api from CNCF allows for entire Kubernetes clusters to be declared as code and rolled out using GitOps tooling. In our client's context, Terraform serves as the technology for infrastructure as code (IaC), complemented by minimal usage of Ansible as needed.
For this engagement, we have extended the pipeline framework to encompass other layers of the architecture, starting from the platform layer (Kubernetes clusters, networking and other system components) to application deployments.
GitOps and progressive delivery
Progressive delivery is a DevSecOps approach that focuses on incrementally and securely rolling out changes to software applications, services, and platform. The goal is to minimize risks and instill confidence in the deployment process. Traditional software delivery methods often involve big-bang deployments, where all changes are released simultaneously. This leads to risks and challenges, such as service disruptions, increased complexity, and rollback difficulties in the event of issues.
For this approach, we require a strong GitOps setup, where changes to any part of the technology stack are performed using a version-controlled method such as Git PRs. This necessitates complete automation of all changes, including rollbacks, directly from source control.
This zero-touch approach maintains the entire state of the technology stack, including components and their versions in code. This ensures constant awareness of the complete state and versions of each component.
Progressive delivery in Kubernetes is an approach to deploying and releasing applications that emphasizes gradual and controlled updates to minimize risks, improve stability, and enhance user experience. It ensures continuous delivery with feature flags and deployment strategies (B/G, Canary, A/B testing and Rolling update). A feature flag can be used to deploy features to a subset of users, then evaluates key metrics before rolling out to more users or rolling back if there are issues.
The concept of release progression aligns with the principles of progressive delivery, which emphasizes a controlled, incremental rollout of changes to minimize risk and gather feedback before reaching a wider audience. It enables teams to catch potential issues early, gain confidence in the release's stability, and minimize the impact of any problems on end users by identifying and mitigating risks at each stage.
Progressive delegation:
- Control of a feature is delegated to the owner most closely responsible for the outcome.
- As a feature transitions from development, through testing, and into production, ownership shifts from engineering to a role closer to the end user (usually the product manager).
Open-source platforms for progressive delivery
| Argo rollout | Flagger | |
|---|---|---|
| Features |
|
|
| Rollback occurs | Easily integrated with ArgoCD. The rollout progress can be seen from ArgoCD UI |
Not integrated with ArgoCD as seamlessly as Argo rollouts; several resources have been created and are visible in the ArgoCD UI where there is no feedback. |
| Canary (Pause) |
|
|
| Advantages |
|
|
| Disadvantages |
|
|
With this approach, we comprehend the current version and running state of the entire technology stack, including microservices themselves, without any runtime inspection. Instead, this information is codified in source control. Similarly, any changes to the technology stack are available as git commit logs, providing a thorough understanding of when modifications occurred and by whom. This offers perfect traceability and auditability for all changes made.
Finally, having the entire state in source control offers the capability to destroy and recreate any part of our technology landscape as required. In real life, this translates to development and operations teams confidently destroying entire environments and recreating them from scratch as required.
Some of the Argo components are:
- Argo workflows: A Kubernetes-native workflow engine supporting DAG and step-based workflows.
- ArgoCD: Declarative continuous delivery with a fully loaded UI, based on GitOps principles.
- Argo rollouts: Facilitates advanced Kubernetes deployment strategies such as Canary and Blue-Green.
- Argo events: Event-based dependency management for Kubernetes.
An example of progressive delivery implemented for Kubernetes/EKS
- A user accesses the service running in EKS pods.
- User requests are forwarded to the ingress controller via a network load balancer.
- All microservices in EKS use Istio as service Mesh. Incoming requests are delegated via the ingress gateway.
- The ingress gateway distributes requests to one of the virtual services of the service pod (stable or canary) based on weights specified in rollout. yaml.
- The corresponding Kubernetes service object forwards to stable/canary pods.
- AnalyisTemplate.yaml defines the evaluation of requests based on specified metrics for success/error outcomes.
- Metrics like count of success/error response of request are measured in the metric server.
- If requests within a specified time period match the result as per AnalysisTemplate from the metric server, the canary is adjusted (20%-40%) after a wait period or rolled out to a stable pod.
IDP with progressive delivery (Argo rollout)/without Argo rollout
- Application team triggers IDP for app deployment.
- All deployment yamls are managed via Helm charts in Jfrog antifactory and hem charts are developed and maintained by the DevOps team.
- Helm value files, used by the helm charts , are hosted in each app level bitbucket repo and are customized by the application team based on their requirements.
- During pipeline execution, the Helm-Chart is pulled from Jfrog, and Helm-Values files are retrieved from the app level bitbucket.
- Values are rendered in the Helm-Chart using the “helm install” command, deploying application in the EKS cluster.
- Based on replicas count specific in Helm Values file, specific number of applications as pods spined up in EKS cluster in specific namespace.
- By default, deployment strategy for app is Rolling Update, the new incremental version of the app is deployed as pods in EKS cluster.
Securing pipelines through seamless integration with ServiceNow
IDP can be easily integrated with ServiceNow using an out-of-the-box IDP ServiceNow plugin. This integration strengthens production elevation controls.
Below are some additional checks that can be added to fortify production rollouts pipelines and deployment can be aborted in case of a mismatch.
- Tagging of “Pipelines” in ServiceNow tickets
- To ensure that approved/valid pipeline are executed for production rollout.
- To ensure linked CR approved state and pipeline are executed in valid change window.
- Tagging of “Release Artifact Version” in ServiceNow tickets
- To ensure that only approved release artifact version/tags are deployed in production.
- To ensure that only approved release artifact version/tags are deployed in production.
Key points
Below is a reference architecture for DevSecOps and GitOps in a microservices environment that offers the following capabilities:
- Comprehensive packaging of all components using helm charts.
- Standard deployment using custom Argo rollout-based helm charts.
- Automated Canary analysis and rollback in case of failure.
- All pipelines implemented using IDP.
- Full automation of infrastructure and applications, starting with the platform, using infrastructure as a code (Terraform, some Ansible).
- Security checks, integrated with Service Now, ensure the promotion of only approved versions of application images.
- More lightweight than image signing, yet delivering a significant portion of the benefits of a secure pipeline.
This reference architecture and model are not limited to Retirement Hub applications. It is a highly automated, secure, and configurable platform offering that will be just as efficient if implemented for other industry verticals like retail, consumer, logistics, insurance, and core banking.
Subscribe
To keep yourself updated on the latest technology and industry trends subscribe to the Infosys Knowledge Institute's publications
Count me in!