GitHub is one of the most popular Git-based Source Code Management and DevOps tools. GitHub has over 73m developer community (1) contributing to over 170m pull requests and 61m+ new repositories created in 2020. It’s a code hosting platform that lets developers work together on projects. With GitHub Actions – a set of commands that help to automate software development tasks – GitHub added CI/CD capabilities to its SCM system. Now developers can build and test their code in their repositories.
A runner is a server that has the GitHub Actions runner application installed. You can use a runner hosted by GitHub, or you can host your own. A runner listens for available jobs, runs one job at a time, and reports the progress, logs, and results back to GitHub. CI/CD (Continuous Integration, Continuous Delivery) is key towards a successful DevOps practice. With GitHub Actions, developers can run CI and build, test and deploy applications.
GitHub introduced support for self-hosted Arm Neoverse-based GitHub Action Runners for developers to natively add AWS Graviton2-based Amazon EC2 instances to their CI/CD workflows.
AWS Graviton processors are custom built by Amazon Web Services using 64-bit Arm Neoverse cores to deliver the best price performance for your cloud workloads running in Amazon EC2. They power Amazon EC2 general purpose (M6g, M6gd, T4g), compute optimized (C6g, C6gd, C6gn), and memory optimized (R6g, R6gd, X2gd) instances, that provide up to 40% better price performance over comparable current generation x86-based instances for a wide variety of workloads. These include application servers, micro-services, high-performance computing, CPU-based machine learning inference, video encoding, Electronic Design Automation, gaming, open-source databases, and in-memory caches.
GitHub Actions lets you execute and automate their CI/CD workflows with the help of self-hosted runners. Developers can set up their code pipelines and build/test applications on Arm-based GitHub Action Runners. This empowers the developers with multi-arch pipelines, however, when the runners are not in use and sitting idle, they incur huge costs. With Arm based AWS Graviton2 instances customers can benefit from 20% less cost than comparable x86 based instances. The savings are greatly increased when the runners are dynamically managed with the execution of a code pipeline.
In the diagram below, we showcase how developers can scale their CI/CD practices leveraging Arm-based GitHub Action runners (GHA) on AWS Graviton2 instances:
Take a look at the following video showing the DevOps workflow of this use case:
Create the actions runner's image by following the steps below:
Navigate to the home directory on Arm-based EC2 instance (M6g.large). It is typically the following directory -
/home/ec2-user
Install expect and jq with the following command:
sudo yum update -y sudo yum install jq, expect -y
Install docker and provide ec2-user with needed permissions.
Use the following command to create your runner folder and change the current directory to it.
mkdir actions-runner; cd actions-runner
Download a new runner package using the following command:
curl -O -L https://github.com/actions/runner/releases/download/v2.273.2/actions-runner-linux-arm64-2.280.1.tar.gz
Unpack the runner's binaries to the folder you created with the next command:
tar xzf ./actions-runner-linux-arm64-2.280.1.tar.gz
Navigate to /home/ec2-user/actions-runner. Using the text editor create file script.exp and paste the contents of this file. Verify the contents with the cat command:
After script and binaries are edited and added, exit the ssh connection and navigate back to EC2 Dashboard. Create an AMI of the EC2 instance. The runner image is now created. We use it with the Lambda function as mentioned below
In AWS console, navigate to AWS Lambda and on the dashboard select the Create function. Provide a Name for the function, select Python 2.7 runtime and select an IAM Role
In the function, scroll down to the Basic settings and select Edit. Set the Handler to "lambda_function.lambda_to_ec2" and save it.
In the function code, please add this script and update with relevant environment details.
Scroll up and save the changes.
On the same screen, scroll down to Triggers. Select Add Trigger.
Select the API Gateway. Create an API based on HTTP API type with the security type and default deployment stage.
After Trigger is created, we should be able to see the details in the Designer section. Here we can find the API Endpoint.
Now, let’s create a GitHub Webhook to trigger Lambda function.
In our GitHub repo navigate to Settings, then select Webhooks
Select Add Webhook and add the API Gateway endpoint into Payload URL section. After the webhook is created, the lambda function will trigger on every push to the repository.
On the image builder instance, ensure that the pre-requisites are met by installing docker, awscli, and kubectl.
Use a text editor of your choice to edit the script.exp file by adding a Pod-runner tag. On the image builder VM, please navigate to the actions-runner folder.
Now use the text editor to create selfconfig.sh script. This script will install the needed tools on the future runner containers. Now we add a script that is our entry point for the docker container. Use the text editor of your choice and create runner.sh script.
Now let’s create a Dockerfile with this content and build the docker image. Tag the image as shown below
docker build ./ -t arm-runner
Create an AWS ECR repository for the runner image. In the AWS management console, navigate to the Elastic Container Registry page and create a repository for the docker image.
After the repository is created, go to the image builder instance and login to ECR and push the docker image.
Now, let’s create a master-runner. It is an AWS Graviton2-based actions runner that deploys pod runners to EKS. We can use the Lambda function to deploy the Pod runners. Let’s create the master-builder by cloning the existing image-builder machine by using AMI.
Register the master-runner with GitHub and add the Master-Runner tag using the following command.
TOKEN=$(/usr/bin/curl --user "<GitHub username>:<GitHub PAT>" -X POST -H "Accept: application/vnd.github.v3+json" https://api. github.com/repos/<username>/<repository name>/actions/runners/registration-token | /usr/bin/jq '.token' | /usr/bin/awk -F\\" '{print $2}') echo $TOKEN > /home/ec2-user/actions-runner/token.file chmod 755 /home/ec2-user/actions-runner/token.file ./config.sh --token $TOKEN
Use the same page to add the Pod-runner tag to the Runner-Placeholder.
Execute the following command:
sudo ./svc.sh install sudo ./svc.sh start
The master runner is set up. Now let's create Kubernetes job manifest to deploy runners. Create job-runner.yaml file with this content.
In our repository, let’s create a CI YAML file wf-ci.yaml under:
<Repo Name>/.github/workflows/.
In this yaml file, we separate the workflow into two jobs. The masterbuild job runs on the master-runner, that deploys a pod runner to the EKS cluster. The runnerbuild job runs on Pod-runner, so it is queued until the container with the runner is deployed to EKS. It runs a simple sleep command for 20sec for demo purposes. To run some specific tasks with the specific instruments, those instruments should be added to a Dockerfile of the runner.
To start the workflow, commit a change to the repository by adding a new file. After committing the change, navigate to the Actions tab of the GitHub repository page.
We will see the masterbuild job run first, and then the runnerbuild job is waiting for the runner container. When the runner container is deployed, the runnerbuild job starts.
We can also see the newly deployed pod runner in the list of runners on the GitHub page. After both jobs are completed, the pod runner will remove itself from GitHub automatically.
As shown above, we can use self-hosted GitHub Actions runner to build our CI/CD system on AWS Graviton2 based EC2 instances. It can help in bringing down the overall cost of the build components by 20%. By adding auto-scaling and on-demand capabilities to our existing CI/CD infrastructure, we can mitigate the need to have static runners configured that incur additional costs. Please feel free reach out to us with any additional comments, questions, or concerns around GitHub Actions runner on Arm.
[CTAToken URL = "https://developer.arm.com/solutions/infrastructure" target="_blank" text="Explore Developer Resources" class ="green"]