CI/CD with a MEAN app and AWS CodePipeline

The following is a walkthrough of the steps taken to release a MEAN app through AWS CodePipeline. The objective is to create a simple pipeline on AWS infrastructure that compiles, installs and runs the source code. As this is the first time I’ve done a deployment in this way I’d welcome any feedback to improve. I much preferred using this compared to AWS OpsWorks. It has a relatively shallow learning curve, and you only need to understand a few Linux commands to be able to make it work.

The project

As the title suggests, this project has been built with an Angular front end, an Express server running on NodeJs and a MongoDB for data storage. An Express route and public files are used to serve the Angular app. The express server has been written in Typescript, and the MongoDB instance is hosted on cloud.mongodb.com. Throughout this guide, ‘StudyCards’ refers to the project name, so you will want to replace any occurrences of it with your project name.

Limitations/Omissions

• I’ve only used one environment to keep it simple, there are no testing/staging environments etc..
• Haven’t written any tests yet, but they could be easily integrated into the build process.
• Haven’t made any considerations for scalability/availability, again just to keep it simple.

Pre-requisites

• You can launch EC2 instances running Linux and have made SSH connections to them before.
• You’ve already created a MEAN app and have it running locally.

Outline

• Prepare an EC2 instance to be used as a server hosting an express app, and run the CodeDeploy agent
• Initialise git locally and link it to AWS CodeCommit
• Create a build project in CodeBuild to test and compile the application
• Setting environment variables
• Set up an application and deployment group in CodeDeploy to install and run the application on the EC2 instance
• Create a pipeline to combine and automate all of these steps
• conclusion

Setting up and launching the EC2 instance

Prepare an instance to be launched into a region/subnet of your choice as long as it has an accessible public IP. I have used Ubuntu, if you use something different, remember to check any changes needed for shell scripts/commands. If you’ll be doing this frequently, or intending to use auto-scaling features, create a launch template.

The instance will need ReadOnly access to S3. Under IAM roles, create a new role and attache the S3ReadOnlyAccess policy to it. Unless you already have an appropriate role created with this policy attached.

As this is just for learning, I’m not going to allow public traffic, or set up https yet. So for the security group, create a rule for SSH from your IP, and a rule for port 8080 from your IP. I say 8080, as I believe permissions on Ubuntu prevent non-root users serving anything under port 1000.

Add a tag to the instance that is unique within your account. It is used to identify the instance for code deployment so any instances that match the tag query will be part of a group that will have code deployed to. I’ve used Key: PROJECT, Value: StudyCards. No other instance in the region or my account has this tag.

The instance needs to have nodeJs and npm installed. This can be done by adding the commands to the user data field when preparing the instance, or, once launched you can create an SSH connection and run the commands. Either way, they need to be successful for the deployment to work. If you are creating a launch template as you plan on repeating the process, enter the commands as user data in the template form.

The following commands will set up an EC2 instance appropriately for this project:

#! /bin/bash

sudo apt-get update

sudo apt install nodejs -y # required to run express server

sudo apt install npm -y # required to install express app dependencies

Once the instance is running check that node and npm have been installed. Depending on your project you may need to change the versions.

node -v

npm -v

Initialising Git and connecting to CodeCommit

CodeBuild can pull source code from a few different locations. I chose CodeCommit as it reduced the number of authorization steps needed and meant everything could be handled through the AWS console developer tools section.

Your IAM user needs the AWSCodeCommitPowerUser policy attached to it. You’ll also need to add an SSH key in the Security credentials for your IAM user, and set up your local ~/.ssh/config. Step 1,2 & 3 of this doc will explain that all in detail. Once this connection is ready, we can prepare the code to be pushed.

The method I found easiest was having both the express and angular source code all in one repository. I had built the Angular app with the CLI, which automatically initializes git by default. To remove this, first, delete the .git folder. It is hidden so you may have to do it from the command line. Doing this removes all previous commits, so if you‘ve been managing versions and changes make a backup first. Creating project links with nested repositories is another way of handling this type of project.

Angular’s .gitignore needs to be moved to the root of the project where the commits will be made from, and combined with the .gitignore from the express app. The paths need to be adjusted to include the folder the app is in. For example, the line node_modules had to be changed to ng-StudyCards/node_modules.

On AWS CodeCommit, create a repository, and copy the SSH link to it, from connection steps > SSH or the clone URL dropdown. Now we can link the local project to CodeCommit. In the terminal run git init at the project root if you haven’t already.

Connect the local project to the CodeCommit repo with

git remote add origin <your-ssh-url>

You can run git remote -v to check the remote repository is connected. Make a commit, then run git push -u origin master

Create a build project in AWS CodeBuild

Before setting up the build project, go to S3 and create a bucket where build artefacts can be stored.

Create project at CodeBuild > Build Projects > Create Project

1. Choose a name, and add tags if you need to
2. Select CodeCommit as the source provider, select your repository and select the branch you want to deploy
3. All the environment settings are really up to you and your project. Let it create a new service role.
4. Leave buildspec to be “use a build spec file”. If you click “insert build commands” then click “switch to editor” below it, you can see a very handy template for the buildspec.yaml file. Click back to the other option unless you want to create your buildspec file this way. It’s not the most reusable method.
5. Change artefact type to S3. Select the bucket you want the build artefacts to be placed into. Set artefacts packaging to Zip. Everything else after this is dependent on your project

To be able to build, there needs to be a buildspec.yaml file in the top level of the repository. This file contains all the commands the build environment needs to process the source code into a deployable project. Pay attention to the indentation in .yaml files. AWS Docs has a good resource on how to create this file.

This .yaml file references some files/folders that haven’t been made yet as they are required for the deploy stage. To ensure the build succeeds, for now, create a blank file called appspec.yml and an empty folder called shScripts in the root of the repository.

Once this is ready, make a commit then go back to your build project and click start build. The build history will be populated. Click on the build run, and watch the build log as it works its way through the commands. From here you will be able to debug anything that causes a failure. Once the UPLOAD_ARTIFACTS phase has completed, you’ll be able to go to S3 and check out your zipped project distribution ready to go. At this point, it’s best to turn on versioning for the bucket to make your revisions easy to manage, and use previous versions if needed.

Setting Express and Angular Environment Variables

Whilst developing the angular app, I was running express locally. Angular needs to know where to make API calls to, so in the environment.prod.ts file of the angular app, ensure that the IP address of the instance is available in that file. The environment.ts file can have the local address. When the angular app is built in CodeBuild, it is told to run ng build — prod, which will make it use the .prod.ts environment.

EC2 instances lose their IP’s when stopped unless you assign an Elastic IP. Just remember you are charged for elastic IP’s that are not assigned to an instance. If you are not using elastic IP’s you’ll need to remember to update your source code if the instance reboots or stops.

You may also want to set up environment variables for the MongoDB password.

Prepare and run CodeDeploy

The appspec.yml file is the key part to getting CodeDeploy to run. Before writing this file, you’ll need to make an IAM role for CodeDeploy, and get the deployment set up in the AWS console.

Go to IAM > Roles > Create role

1. Under “select a service to view it’s use cases” click CodeDeploy. Below the list of services, you’ll now be able to choose CodeDeploy. We don’t need the one for ECS or lambda. Click Next
2. The required permissions have automatically been added. Click next. Add Tags if you need to.
3. Chose a role name and click create

Got to CodeDeploy > Applications > Create Application

1. Choose a name, and EC2/On-premises as the compute platform to create an application
2. Create a deployment group within this application, choose a name, and select the service role previously created. Leave deployment type as in-place
3. Select Amazon EC2 instances, and enter the key-value tag that matches the EC2 instance created to use for this project. There should be 1 unique match now.
4. Leave everything else, except Load balancer. Deselect that.

Now we can prepare the appspec.yml and associated files, commit and then deploy. There are several files that need to be created alongside the appspec.

A .service file for Linux’s systemctl to run node in the background, a script to remove any previous versions of the app, one to install any dependencies the Express app needs, and another to start the express app.

appspec.yml file

This file must be at the root of the project. It tells the CodeDeploy agent where to move the project to, and what scripts to run after moving the files.

There are many other hooks available, and the AWS documentation has plenty of reference for this.

I created a shScripts folder, which is placed at the root of the project along with the appspec file then added cleanup.sh, dependencies.sh and runApp.sh files to it.

cleanup.sh

#!/bin/bash

sudo rm -rf /var/www/StudyCards

dependencies.sh

#!/bin/bash

cd /var/www/StudyCards

sudo npm install

runApp.sh

This script moves the .service file so it is available to systemctl. It also gives execution permissions to the express server.js file so it can run the app. Make sure that your server.js file’s 1st line is #!/usr/bin/env node so that it can be executed.

#!/bin/bash

cp /var/www/StudyCards/studycards.service /etc/systemd/system

chmod +x /var/www/StudyCards/server.js

sudo systemctl daemon-reload

sudo systemctl start studycards

sudo systemctl enable studycards

.service file

Systemctl uses this to launch the express app, so it needs to point to wherever the server.js express file is. Be aware there may be some slight changes required if not using Ubuntu.

With all of these files in place, commit and push to the online repository. From there you are ready to deploy. Because we’ve committed a new version of the source files, you’ll need to run the build again which will update S3 with the new version.

Under your application, select the create deployment group. From here you can select the revision location. If you turned on versioning in S3, you’ll see just one item for your app, which will be the most recent. Leave the system manager to be installed and managed by CodeDeploy, change any other settings needed, and deploy.

If you go to the deployment groups deployment history, you’ll be able to click on the id of the deployment you just made, and view progress. From here you can debug any errors. Scroll to the bottom to see deployment lifecycle events. To view the log for a lifecycle event, click view events. Scroll down and you’ll see a list of events, and potentially ‘ScriptFailed’ click it to see the error log. If you have success you should be able to navigate to your app from a web browser.

Creating a pipeline

Before you begin, ensure you have committed and pushed any changes to CodeCommit.

Go to CodePipeline > Create Pipeline

1. Give the pipeline a name, let it create a new service role
2. Under advanced, you can change where it stores the build artefacts if you want.
3. Choose CodeCommit, your repo, and branch. Leave CloudWatch as the detection option
4. Choose CodeBuild, your region, and the build project created previously. Leave it as a single build.
5. Choose CodeDeploy as the provider, your region, and the application created previously. You’ll then be able to select that applications associated deployment groups
6. Review and create pipeline

The pipeline will now run and you will be able to watch the stages progress. It provides links to the details of each phase, which simply redirect you to that tools page, such as build history in CodeBuild.

Conclusion

There are quite a few steps involved in getting a pipeline up and running, but once set-up it is clearly a valuable workflow tool. Hopefully, this walkthrough has been able to help you achieve a functioning pipeline, and I would welcome any feedback on how to improve it!