Module 3: Deploying 1
Module Overview
Learn the fundamentals of deploying applications and managing deployment environments. Understand the process of moving applications from development to production.
Learning Objectives
- Understand the deployment lifecycle and environments
- Learn about AWS Elastic Beanstalk for application deployment
- Configure deployment settings for Java applications
- Manage environment variables and configuration
- Troubleshoot common deployment issues
- Classify the purpose of each part of a given Continuous Deployment pipeline into source, build, test, and deploy
- Explain how having different deployment stages separates test data from production data
- Examine where in a provided pipeline a given test should be executed
- Compare and contrast integration tests and unit tests
- Explain why tests should be run and catch bugs in the earliest stage possible
- Explain why separating test data from production data is valuable when testing
- Recall that an approval workflow only runs after a successful promotion to that stage
Deployment Pipeline Overview
Deploying software is the process by which code developed on your computer gets shared with the world. The deployment pipeline typically consists of several stages that ensure quality and reliability.
Stages of Deployment
- Source Stage: Local development, testing, code reviews, and pushing to a code repository
- Build Stage: Automated build processes, unit tests, and static code analysis
- Test Stage: Integration testing in isolated environments (often called Beta/QA)
- Production Stage: Deployment to live servers for end-user access
Each stage serves as a quality gate, ensuring that only properly functioning code moves to the next level. This staged approach minimizes the risk of introducing bugs into production systems.
Deploying Software at Amazon
The act of deploying software is the process by which the software you develop on your computer gets shared with the world! At a high-level view, this involves you writing code on your development machine, then checking that code into the source code repository. From there the deployment system builds your change and runs unit and integration tests verifying your change before finally delivering your code bug free (hopefully) to end users. We'll be walking through each stage of this process in this reading.
Challenges at Amazon
Before we dig into the nuts and bolts of the deployment process, let's take a look at why we've put so much effort into standardizing our process. Any company deploying code to their customers faces challenges, and those only multiply when you have many teams developing many products across many code bases deploying to many servers around the world! That is a tremendous number of moving parts, so Amazon has dedicated time and money to tools that ensure we can manage all those moving parts and deploy our changes with consistency and quality.
Deployment pipeline overview
Figure 1: An overview of the development pipeline showing the flow from the Source stage - where we develop the code, complete local testing, get a code review, and finally push to the remote code repository - to the Build stage - where an automated build runs our unit tests and static code analysis to verify our code change - to the Beta stage - where new changes are tested in an experimental stage - to the Gamma stage - where new changes are run against production data without facing actual customers - and finally to the Production stage - where our code goes live and actual customers see the change.
The Amazon deployment pipeline is broken into four major stages, culminating in the final deployment to the live users in production. Amazon's Pipelines system automates the deployment of our code changes through the different stages. Each stage is distinct from the rest, either interacting with another service or running on its own set of hosts separate to the other stages.
The Source stage covers local development, local testing, code reviews and finally pushing code to the remote code repository. The git system we use allows developers to work together on the same projects without stepping on each other's toes. As such, it is one of the most important tools that Amazon developers use every day.
Once we push to the remote repository at the end of the Source stage, Pipelines moves our change into the Build stage, where all of the packages in our version set (the dependencies of our package, our package, and packages that depend on our package) are built together, unit tests are run, and code coverage tests are run.
If our build succeeds and our test and verifications pass, the pipeline will promote the package to the Beta stage. Here, tests are run against the deployed service to ensure all the changes work together, along with other services our service may interact with. These tests are called integration tests.
If our integration tests all pass in Beta, our changes move to the Gamma stage. In the Gamma stage, our code will be tested again with production data. Integration tests are run on the Gamma stage as well. And while we strive for fully-automated testing, the Gamma stage is where any required manual testing usually occurs. Depending on the size of the team, that manual testing might be completed by the software developers on the team, or there might be dedicated Quality Assurance Engineers (QAEs) whose job it is to develop and execute testing plans to validate that changes are working correctly.
Once our changes have been approved in Gamma, they are promoted to Production, where they are available to customers! We don't run more tests on Production, but we do have monitors set up tracking the status of our system. If we see anything shortly after deployment indicating a problem with our service (for example, a lot of NullPointerExceptions suddenly appear in our service logs), Pipelines can rollback our deployment, re-deploying the previous, working version of our service. This protects both us and our customers from a service outage while we investigate the issue.
Big Projects -- Big Impact -- Big Challenges
You may have seen testing mentioned over and over in the previous section. This underlines how important testing is to the deployment process. Quality and velocity are the primary goals of our deployment process. When we say quality, we mean bug-free code that delivers value to our customers. By velocity, we mean how fast we go from an idea, to implementation, to release of a new feature, improvement, or bug fix. Consistent testing across all stages is key to delivering quality software to our customers while working quickly. Thorough testing throughout our deployment process helps us catch bugs early, where they are easier and quicker to fix. This lets us regularly deliver new products and features to our customers without glaring bugs.
In addition to quality and velocity, our deployment process enables coordination within groups and across organizations as software is developed by many teams and many individuals. Pipelines apply a consistent quality bar and ensure that changes from various people or teams work together before those changes reach customers.
Continuous Deployment
The concept of Continuous Deployment isn't only a nice way to describe how we are continuously deploying new features and new fixes, but it's also a recognized Software Development process. Continuous Deployment is a strategy for developing and releasing software that automates the testing and deployment process in order to reduce the amount of repetitive work done by humans. Automating the tasks that nobody likes to do manually reduces the potential for human error.
Continuous deployment supports our underlying principles as well. It helps to ensure quality through rigorous automated testing that takes place at every stage for every piece of code written. Coordination is improved by giving every code package its own deployment pipeline to ensure that code for that package is tested and deployed to the servers or devices that are configured correctly for each stage. With new code being constantly tested and deployed bottlenecks in the system are reduced and velocity is improved. Notifications of code failure are sent out quickly, and fixes will automatically start a new deployment.
It's important to note that it is still possible (and likely) that a bug in someone's code could block your code's deployment or vice versa. This situation does impact velocity, but it also ensures quality. There is always a tradeoff. As a developer, it's important to realize that there may be times when the process can feel slow, but Amazon has tried to find the best balance between timely deployment and code quality.
The Stages of Continuous Deployment
At this point, we would like to walk through all the steps involved in each stage of the continuous deployment process. We will specifically be looking at the Actions that are taken in each stage, the Approvals required to move past the stage, and any Blockers that could come up in that stage.
Source Stage
The source stage is the first stage and the stage where you, as a developer, have the most responsibilities.
Actions
In this stage, a task has been given to a developer, and code will be written. The developer will test their changes in a local environment. Unit tests and integration tests are written and verified here as part of the development process to verify their code works as expected. Once all development is complete, the developer will commit their code locally and publish a code review.
The code review is when other developers review the code, providing guidance or suggestions and potentially catching bugs. This is a time for teammates to ensure code is being developed following the team's best practices and coding style. Those who did not write the code become familiar with the new code they will also be responsible for.
Approvals
The approval needed in the Source stage is an approved Code Review.
Blockers
The code review is designed to ensure that code functionality, quality, and style adhere to the project specifications and team's standards. If the developer's peers feel there are inconsistencies or improvements, the team can work together during the review to make adjustments and revisions. At this point, the code has NOT yet entered the deployment pipeline, but this is an important part of the development process.
Final Action
Once the Code Review is approved, the developer pushes the code to the remote repository.
Build Stage
The Build stage marks the beginning of the Continuous Deployment pipeline.
Automated Actions
A pipeline triggers a new build whenever changes are pushed to a package that the pipeline is configured to auto-build. The build system pulls down all the code, builds it (which includes running any unit tests), and prepares to pass the built code to the next pipeline stage. After the build completes, Static Code Analysis is run against the changes to approve them.
Approval
The Code Analysis that is done after the code is built marks the next approval point. Static code analysis can include the following:
- Verification that Code Reviews were done and approved.
- Verifying code coverage on newly added code and/or the package(s) that was changed meets a configured level.
Blockers
If the build fails (either because it could not compile of unit tests failed) or any of the Static Code Analysis verifications fail, then the deployment to the next stage is stopped. Notifications will be automatically generated to alert the pipeline owner of the failure. Depending on how the pipeline is configured, a new ticket may be generated to track the issue.
Final Action
If the code builds successfully and all Static Code Analysis passes, then the build output is promoted to the next stage of the pipeline.
Test & Deploy
After the build stage is approved, the built code will go through a series of stages that can be simplified down to "Test & Deploy". As it moves along, tests are run against environments more and more similar to Production.
Beta Stage
The term "beta testing" has been used for decades in software development to refer to the process of testing something before it's done. New features and new products come together in their first shared environment to begin testing how all the pieces of the project fit together. The "environment" here is a server (or AWS account) that is configured to run the code in a dedicated space for testing, referred to as the beta stage. Beta is the first place where a code change in one service can interact with another service. We often wire our beta stage to the beta stages of any other services our service uses. We describe the beta stage as "shared" because the code being tested here will have been written by many developers and potentially multiple development teams. Beta is an environment where the data is often in flux as new features are quickly deployed.
Actions
Code is deployed to the Beta stage after the build succeeds and is approved. Testing here is usually automated, but developers might manually test features on Beta if they need to test inter-service interactions that their local version of the service cannot replicate.
Approval
Approval on beta is based on the integration tests that are run against the build here. These are automated tests that go beyond the code coverage and unit testing of the build stage in order to ensure our system as a whole is functioning as expected. The developer will have written integration tests for their code and submitted them along with the rest of the code change in the Source stage. The testing here is closer to production and will be tested against real services, no longer utilizing mocks. Instead of setting up mocks, integration tests will usually have preliminary steps that set up starter data in the service for the tests to use. As new code is developed and deployed to beta, the integration tests continue to be run to catch any bugs the new code might introduce in existing functionality.
Blockers
If any of the automated integration tests fail, the deployment process is stopped and the system will send out automated notifications describing what failed. Depending on how the pipeline is configured, a new ticket may be generated and assigned to the pipeline owner, at which point the team's oncall can determine who was responsible for the breaking change and inform them that they need to investigate and fix the bug.
Final Action
Once all integration tests pass, the code change is promoted to the Gamma stage of the deployment pipeline.
Gamma Stage
For Amazon the Gamma stage (or pre-prod, short for pre-production) represents the last testing environment before Production. Gamma should mimic the production environment as closely as possible in order to provide the most realistic tests for the code. Data here is often a snapshot of real production data. If a team has Quality Assurance Engineers involved in testing their code, they are usually testing code once it has reached the Gamma stage.
Actions
More automated testing, as well as manual validation, can take place in the Gamma stage.
Approval
Final acceptance testing is done here. Our goal is to have all final acceptance testing automated. This includes running our integration tests, and can include other automated steps, like load testing our service, which simulates high traffic (like we see on Prime Day or around holidays) hitting our service to see if it can handle all the requests. Our approvals here may also include manual tests if we are unable to automate the actions necessary to test some aspects of our service.
Blockers
As with other stages, any tests that fail will trigger the deployment process to be halted. Often the issues found here are more complicated or far-reaching, but like everything else, they will trigger notifications so that they can be addressed by a team of QAEs and SDEs and move through this entire process again when they are fixed to ensure the best experience for our customers.
The tests performed in Gamma are generally more thorough and look at more than pure technical functionality compared to previous stages. It is also painful to be blocked this late in the process, but it happens. If the deployment pipeline has done its job, then the number of issues that come up in the Gamma stage should be low.
Final Action
Once the Gamma build is approved, the code is promoted to Production in the pipeline.
Production
This is it. The big time. Production is what this whole process is about. Production is live to our customers and all the code there has gone through this entire process in order to ensure it is of the highest quality with the fewest bugs.
Actions
Once all the tests on Gamma have been successfully completed, the code can be pushed out to production. For the most part, deployments from Gamma to Production are automated, but on some teams, they remain manual. At Amazon, we often deploy our code to regions all over the world over the period of a week. In this case, "Production" might actually be a dozen stages as we deploy in waves spanning some of our smallest regions (like Frankfurt, Germany), to our largest (Virginia, USA).
Approval
While we do not generally run more tests against our production service, we do track that our deployment succeeds. We can set up monitors that track statistics about our service (such as traffic logs, service logs, or resource access logs) and alarm if it looks like something has gone wrong. This could include lots of exceptions suddenly appearing in our logs, or our traffic suddenly dropping a significant amount, or our DynamoDB tables reporting many more read requests than there were before the deployment. If one of these alarms goes off within a set time after our deployment, our pipeline can rollback that deployment, re-deploying the previous version of our service blocking deployments.
A production rollback will certainly generate a new ticket to the team, and the oncall will be responsible for figuring out what has gone wrong and coordinating a fix.
Conclusion
As you can see, there are a lot of moving parts that all work together to deploy software at Amazon. We hope you're also beginning to see why there's so much going on. In order to provide our customers with the highest quality software products we need to ensure that we follow a well-defined process that validates our deliverables at every step of the process. In the next reading we're going to look at a larger example of how all these pieces work together as we follow some code through the pipeline from creation to production.
Deployment Environments
Modern deployment practices utilize multiple environments to safely test and validate changes before they reach customers.
Common Deployment Environments
- Development: Where initial coding and testing occur
- Beta/QA: Where integration testing across services happens
- Staging/Gamma: A pre-production environment that closely mimics production configuration
- Production: The live environment that serves end users
The Beta stage is where tests run to ensure all components work together correctly. The Gamma stage uses production-like data and configuration for final validation before deployment.
Benefits of Multiple Environments
- Safely test and verify changes without affecting users
- Separate test data from production data
- Identify integration issues early
- Validate performance in realistic conditions
- Enable gradual rollout strategies
Testing in the Deployment Pipeline
Different types of tests should be executed at appropriate stages of the deployment pipeline to catch issues early and prevent costly bugs in production.
Types of Tests
- Unit Tests: Test individual components in isolation (Source/Build stage)
- Integration Tests: Test interactions between components (Beta/QA stage)
- System Tests: Test the application as a whole (Beta/QA stage)
- Performance Tests: Test application speed and reliability (Staging/Gamma stage)
- Acceptance Tests: Validate that requirements are met (Staging/Gamma stage)
Testing Best Practices
- Run tests as early as possible in the pipeline
- Automate tests whenever possible
- Use isolated test environments with appropriate test data
- Ensure test coverage across critical functionality
- Incorporate security testing in the pipeline
AWS Elastic Beanstalk
AWS Elastic Beanstalk is a Platform as a Service (PaaS) offering that simplifies deploying and managing applications. It handles infrastructure provisioning, load balancing, scaling, and monitoring, while you maintain control over the underlying resources.
Key Features
- Supports multiple programming languages and application servers
- Automatic scaling based on application needs
- Integrated with other AWS services
- Environment management tools
- Customizable deployment options
Deployment Process
- Create an application in Elastic Beanstalk
- Upload your application code (typically as a JAR or WAR file)
- Configure environment settings (instance type, scaling options, etc.)
- Deploy the application
- Monitor and manage the deployed application
Guided Project
Mastery Task 6: Create an API Gateway
Right now we have 5 different Lambda functions that are responsible for the REST API methods we'd like our users to have access to. That's great but not very useable.
What we'd like now is to build a Gateway on AWS that maps HTTP requests to lambda functions. Refer to the Design Document for specific end points.
We will build this together during our final guided project.