Practicing Continuous Integration and Continuous Delivery on AWS (2017)

AWS offerings:

  • AWS CodeStar
  • AWS CodeCommit
  • AWS CopePipeline
  • AWS CodeBuild
  • AWS CodeDeploy

What is CI/CD?

Continuous Integration

  • Developers regularly merge their code to a central repository
  • Refers to the build/integration stage of the software release lifecycle
  • Smaller commits and smaller code changes to integrate
  • Developer commits to repo minimum once a day
  • Goals: find and address bugs faster, improve software quality, reduce the time to validate and relase new changes
  • Automating builds, testing
  • Similar testing environments to production

Continuous Delivery

  • Code changes are automatically built, tested, and prepared for production release
  • Expands CI on deploying changes to test and production env after build stage
  • As a result, developers have deployment-ready build artifact that has passed a test process
  • Before deploying to production you can implement a decision process to ensure that the production deployment is authorized and audited
  • Decision to go live becomes a business decision and not a technical one

Continuous Deployment

  • Revisions are deployed to production automatically without explicit approval from a developer
  • Automating the entire software release process

Benefits of Continuous Delivery

  • Automate the software release process
  • Improve developer productivity
  • Improve code quality
  • Deliver updates faster

Implementing CI/CD

CI/CD Pipeline Stages

  • Source
    • Developers commit changes
  • Build
    • Changes are built
    • Quality checks, such as unit test coverage, style check, static analysis
  • Staging
    • Code is deployed and tested
    • Staging env could be env pre-made for testing
    • Staging env can be also created dynamically using IaaC
  • Production
    • Code is deployed to public servers

Maturity and Beyond (Improvements to CI/CD model)

  • More staging environments for specific performance, compliance, security and user interface (UI) tests
  • Unit tests of infrastructure and configuration code
  • Integration with other systems and processes, such as code review, issue tracking and event notification
  • Integration with database schema migration (if applicable)
  • Additional step for auditing and business approval

Teams

Teams should be no larger than 10-12 people (two-pizzas team)

  • Application team
    • Backlog, stories, and unit tests
    • Develop features based on a specified application target
  • Infrastructure team
    • Code the infrastructure needed to run the application
    • AWS CloudFormation / HashiCorp Terraform, Chef, Puppet, Ansible, or Salt
  • Tools team
    • Manages CI/CD pipeline
    • Building and integrating all parts of the CI/CD pipeline
    • Building source control repositories, workflow engines, build environments, testing frameworks, and artifact repositories
    • AWS CodeStar, AWS CodePipeline, AWS CodeCommit, AWS CodeDeploy, and AWS CodeBuild, along with Jenkins, GitHub, Artifactory, TeamCenter, and other similar tools

Testing Stages in CI/CD

Mike Cohn, Succeeding with Agile: CI/CD Testing Pyramid

  • Setting Up and Executing Builds
    • Unit Testing
      • Performed during development stage
      • Tests specific section of the code to ensure the expected behavior
    • Static Code Analysis
      • Helps find coding errors and security holes
      • Ensures conformance to coding guidelines
  • Staging
    • Integration Testing
      • Verifies interfaces between components against software design
    • Component Testing
      • Tests message passing between various components and their outcomes
      • Idempotency in component testing
    • System Testing
      • End-to-end tests, including Ui, API, backend logic, and end state
    • Performance Testing
      • Stability under a particular workload
      • Load tests, stress tests, and spike tests
      • Benchmarking against predefined criteria
    • Compliance Testing
      • Code change compliance with the requirements of a nonfunctional specification and/or regulations
    • User Acceptance Testing
      • Validates end-to-end business flow
      • Conducted by an end user in staging environment
  • Production
    • Canary Test
      • New code deployed on a small subset of servers before deploying to entire production env

Building the Pipeline

  • Minimum Viable Pipeline for Continuous Integration
    • Use AWS CodeStar
      • Uses AWS CodePipeline, AWS CodeBuild, AWS CodeCommit, and AWS CodeDeploy
      • Integrated setup process, tools, templates, and dashboard
  • AWS CodePipeline
    • CI/CD service for fast and reliable application and infrastructure updates
    • Source stage accepts inputs from GitHub, AWS CodeCommit, and Amazon S3
    • Works seamlessly with AWS CodeBuild
    • Integrates with build servers, e.g. Jenkins, Solano CI and TeamCity
    • Certain CodeBuild stages can run in parallel

Continuous Delivery Pipeline

  • Presence of Staging and Production steps
  • Production step is performed after a manual approval
  • AWS CodePipeline directly integrates with AWS CodeDeploy
  • AWS CodePipeline directly integrates with AWS OpsWorks and AWS Elastic Beanstalk
  • Deployment stages can be expanded with various tests
    • Out-of-the-box integrations with services like Ghost Inspector, Runscope, Apica, and others

Adding Lambda Actions

  • Supported by AWS CodeStar and AWS CodePipeline
  • Lambda use cases
    • Roll out changes to your environment by applying or updating an AWS CloudFormation template
    • Create resources on demand in one stage of a pipeline using AWS CloudFormation and delete them in another stage
    • Deploy application versions with zero downtime in AWS Elastic Beanstalk with a Lambda function that swaps CNAME values
    • Deploy to Amazon EC2 Container Service (ECS) Docker instances
    • Back up resources before building or deploying by creating an AMI snapshot
      • Add integration with third-party products to your pipeline, such as posting messages to an IRC client

Deploying Infrastructure Code Changes in a CI/CD Pipeline

  • AWS CodePipeline supports AWS CloudFormation as a deployment action
  • Choose create/delete stack or create/execute change sets

CI/CD for Serverless Applications

  • Use AWS CodeStar, AWS CodePipeline, AWS CodeBuild, and AWS CloudFormation
  • Automate the building, testing, and deployment of serverless applications
    • Expressed in templates built with the AWS Serverless Application Model

Pipelines for Multiple Teams, Branches, and Regions

  • Create specific pipelines for the team branches and another release pipeline for the final product delivery

AWS CodeBuild

  • Tight integration with AWS CodeCommit, AWS CodePipeline, and AWS CodeDeploy, as well as Git and CodePipeline Lambda
  • Quickstart environments for a number of popular languages
  • Ability to run any Docker container

Deployment Methods

Deployment Methods

  • All at Once (In-Place Deployment)
    • Deploy new application code to an existing fleet of servers
  • Rolling Deployment
    • Fleet is divided into portions so that all of the fleet isn’t upgraded at once
    • Two software versions, new and old, are running on the same fleet
    • Canary release is a variation of a rolling deployment
  • Immutable
    • Deployment of application code by starting an entirely new set of servers with a new configuration or version of application code
  • Blue/Green Deployment
    • Requires creation of another environment
    • When ready, traffic is shifted to the new deployment

Summary of Best Practices

  • Treat your infrastructure as code
    • Use version control for your infrastructure code
    • Make use of bug tracking/ticketing systems
    • Have peers review changes before applying them
    • Establish infrastructure code patterns/designs
    • Test infrastructure changes like code changes
  • Put developers into integrated teams of no more than 12 self-sustaining members
  • Have all developers commit code to the main trunk frequently, with no long-running feature branches
  • Consistently adopt a build system such as Maven or Gradle across your organization and standardize builds
  • Have developers build unit tests toward 100% coverage of the code base.
  • Ensure that unit tests are 70% of the overall testing in duration, number, and scope.
  • Ensure that unit tests are up-to-date and not neglected. Unit test failures should be fixed, not bypassed
  • Treat your continuous delivery configuration as code
  • Establish role-based security controls (that is, who can do what and when)
    • Monitor/track every resource possible
  • Alert on services, availability, and response times
  • Capture, learn, and improve
  • Share access with everyone on the team
  • Plan metrics and monitoring into the lifecycle
  • Keep and track standard metrics.
    • Number of builds.
    • Number of deployments
    • Average time for changes to reach production
    • Average time from first pipeline stage to each stage
    • Number of changes reaching production
    • Average build time
  • Use multiple distinct pipelines for each branch and team