AWS CDK GitLab Runner autoscaling on EC2

This project provides a CDK construct to execute jobs on auto-scaled EC2 instances using the Docker Machine executor.

Running out of Runner minutes, using Docker-in-Docker (dind), speed up jobs with shared S3 Cache, cross compiling/building environment multiarch, cost effective autoscaling on EC2, deploy directly from AWS accounts (without AWS Access Key), running on Spot instances, having a bigger build log size

Install

TypeScript

1npm install @pepperize/cdk-autoscaling-gitlab-runner

or

1yarn add @pepperize/cdk-autoscaling-gitlab-runner

Python

1pip install pepperize.cdk-autoscaling-gitlab-runner

C# / .Net

dotnet add package Pepperize.CDK.AutoscalingGitlabRunner

Java

1<dependency>
2  <groupId>com.pepperize</groupId>
3  <artifactId>cdk-autoscaling-gitlab-runner</artifactId>
4  <version>${cdkAutoscalingGitlabRunner.version}</version>
5</dependency>

Quickstart

1. Create a new AWS CDK App in TypeScript with projen

   1mkdir gitlab-runner
   2cd gitlab-runner
   3git init
   4npx projen new awscdk-app-ts

2. Configure your project in .projenrc.js

   • Add deps: ["@pepperize/cdk-autoscaling-gitlab-runner"],

3. Update project files and install dependencies

   1npx projen

4. Register a new runner

   Registering runners:

   • For a shared runner, go to the GitLab Admin Area and click Overview > Runners
   • For a group runner, go to Settings > CI/CD and expand the Runners section
   • For a project runner, go to Settings > CI/CD and expand the Runners section

   Optionally enable: Run untagged jobs [x] Indicates whether this runner can pick jobs without tags

   See also Registration token vs. Authentication token

5. Retrieve a new runner authentication token

   Register a new runner

   1curl --request POST "https://gitlab.com/api/v4/runners" --form "token=<your register token>" --form "description=gitlab-runner" --form "tag_list=pepperize,docker,production"

6. Store runner authentication token in SSM ParameterStore

   Create a String parameter

   1aws ssm put-parameter --name "/gitlab-runner/token" --value "<your runner authentication token>" --type "String"

7. Add to your main.ts

   1import { Vpc } from "@aws-cdk/aws-ec2";
   2import { App, Stack } from "@aws-cdk/core";
   3import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";
   4
   5const app = new App();
   6const stack = new Stack(app, "GitLabRunnerStack");
   7const vpc = Vpc.fromLookup(app, "ExistingVpc", {
   8  vpcId: "<your vpc id>",
   9});
   10const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
   11  parameterName: "/gitlab-runner/token",
   12});
   13new GitlabRunnerAutoscaling(stack, "GitlabRunner", {
   14  network: {
   15    vpc: vpc,
   16  },
   17  runners: [
   18    {
   19      token: token,
   20      configuration: {
   21        // optionally configure your runner
   22      },
   23    },
   24  ],
   25});

8. Create service linked role

   (If requesting spot instances, default: true)

   1aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

9. Configure the AWS CLI

   • AWSume
   • Configuring the AWS CLI
   • AWS Single Sign-On

10. Deploy the GitLab Runner

    1npm run deploy

Example

Custom cache bucket

By default, an AWS S3 Bucket is created as GitLab Runner's distributed cache. It's encrypted and public access is blocked. A custom S3 Bucket can be configured:

1const cache = new Bucket(this, "Cache", {
2  // Your custom bucket
3});
4const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
5  parameterName: "/gitlab-runner/token",
6});
7
8new GitlabRunnerAutoscaling(this, "Runner", {
9  runners: [
10    {
11      token: token,
12    },
13  ],
14  cache: { bucket: cache },
15});

See example, GitlabRunnerAutoscalingCacheProps

Custom EC2 key pair

By default, the amazonec2 driver will create an EC2 key pair for each runner. To use custom ssh credentials provide a SecretsManager Secret with the private and public key file:

1. Create a key pair, download the private key file and remember the created key pair name

2. Generate the public key file

   ssh-keygen -f <the downloaded private key file> -y
   

3. Create an AWS SecretsManager Secret from the key pair

   1aws secretsmanager create-secret --name <the secret name> --secret-string "{\"<the key pair name>\":\"<the private key>\",\"<the key pair name>.pub\":\"<the public key>\"}"

4. Configure the job runner

   1const keyPair = Secret.fromSecretNameV2(stack, "Secret", "CustomEC2KeyPair");
   2
   3new GitlabRunnerAutoscaling(this, "Runner", {
   4  runners: [
   5    {
   6      keyPair: keyPair,
   7      configuration: {
   8        machine: {
   9          machineOptions: {
   10            keypairName: "<the key pair name>",
   11          },
   12        },
   13      },
   14    },
   15  ],
   16  cache: { bucket: cache },
   17});

Configure Docker Machine

By default, docker machine is configured to run privileged with CAP_SYS_ADMIN to support Docker-in-Docker using the OverlayFS driver and cross compiling/building with multiarch.

See runners.docker section in Advanced configuration

1import { GitlabRunnerAutoscaling } from "@pepperize/cdk-autoscaling-gitlab-runner";
2import { StringParameter } from "aws-cdk-lib/aws-ssm";
3
4const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
5  parameterName: "/gitlab-runner/token",
6});
7
8new GitlabRunnerAutoscaling(this, "Runner", {
9  runners: [
10    {
11      token: token,
12      configuration: {
13        environment: [], // Reset the OverlayFS driver for every project
14        docker: {
15          capAdd: [], // Remove the CAP_SYS_ADMIN
16          privileged: false, // Run unprivileged
17        },
18        machine: {
19          idleCount: 2, // Number of idle machine
20          idleTime: 3000, // Waiting time in idle state
21          maxBuilds: 1, // Max builds before instance is removed
22        },
23      },
24    },
25  ],
26});

See example, DockerConfiguration

Bigger instance type

By default, t3.nano is used for the manager/coordinator and t3.micro instances will be spawned. For bigger projects, for example with webpack, this won't be enough memory.

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  manager: {
7    instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.SMALL),
8  },
9  runners: [
10    {
11      instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.LARGE),
12      token: token,
13      configuration: {
14        // optionally configure your runner
15      },
16    },
17  ],
18});

You may have to disable or configure Spot instances

See example, GitlabRunnerAutoscalingManagerProps, GitlabRunnerAutoscalingJobRunnerProps

Different machine image

By default, the latest Amazon 2 Linux will be used for the manager/coordinator. The manager/coordinator instance's cloud init scripts requires yum is installed, any RHEL flavor should work. The requested runner instances by default using Ubuntu 20.04, any OS implemented by the Docker Machine provisioner should work.

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  manager: {
7    machineImage: MachineImage.genericLinux(managerAmiMap),
8  },
9  runners: [
10    {
11      machineImage: MachineImage.genericLinux(runnerAmiMap),
12      token: token,
13      configuration: {
14        // optionally configure your runner
15      },
16    },
17  ],
18});

See example, GitlabRunnerAutoscalingManagerProps, GitlabRunnerAutoscalingJobRunnerProps

Multiple runners configuration

Each runner defines one [[runners]] section in the configuration file. Use Specific runners when you want to use runners for specific projects.

1const privilegedRole = new Role(this, "PrivilegedRunnersRole", {
2  // role 1
3});
4
5const restrictedRole = new Role(this, "RestrictedRunnersRole", {
6  // role 2
7});
8
9const token1 = StringParameter.fromStringParameterAttributes(stack, "Token1", {
10  parameterName: "/gitlab-runner/token1",
11});
12
13const token2 = StringParameter.fromStringParameterAttributes(stack, "Token2", {
14  parameterName: "/gitlab-runner/token2",
15});
16
17new GitlabRunnerAutoscaling(this, "Runner", {
18  runners: [
19    {
20      token: token1,
21      configuration: {
22        name: "privileged-runner",
23      },
24      role: privilegedRole,
25    },
26    {
27      token: token2,
28      configuration: {
29        name: "restricted-runner",
30        docker: {
31          privileged: false, // Run unprivileged
32        },
33      },
34      role: restrictedRole,
35    },
36  ],
37});

See example, GitlabRunnerAutoscalingProps

Spot instances

By default, EC2 Spot Instances are requested.

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  runners: [
7    {
8      token: token,
9      configuration: {
10        machine: {
11          machineOptions: {
12            requestSpotInstance: false,
13            spotPrice: 0.5,
14          },
15        },
16      },
17    },
18  ],
19});

See example, EC2 spot price, MachineConfiguration, MachineOptions, Advanced configuration - runners.machine.autoscaling

Cross-Compile with Multiarch

To build binaries of different architectures can also use Multiarch

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  runners: [
7    {
8      token: token,
9      configuration: {
10        docker: {
11          privileged: true,
12        },
13      },
14    },
15  ],
16});

Configure your .gitlab-ci.yml file

1build:
2  image: multiarch/debian-debootstrap:armhf-buster
3  services:
4    - docker:stable-dind
5    - name: multiarch/qemu-user-static:register
6      command:
7        - "--reset"
8  script:
9    - make build

See multiarch/qemu-user-static

Running on AWS Graviton

To run your jobs on AWS Graviton you have to provide an AMI for arm64 architecture.

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  runners: [
7    {
8      token: token,
9      configuration: {
10        instanceType: InstanceType.of(InstanceClass.M6G, InstanceSize.LARGE),
11        machineImage: MachineImage.genericLinux({
12          [this.region]: new LookupMachineImage({
13            name: "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-*-server-*",
14            owners: ["099720109477"],
15            filters: {
16              architecture: [InstanceArchitecture.ARM_64],
17              "image-type": ["machine"],
18              state: ["available"],
19              "root-device-type": ["ebs"],
20              "virtualization-type": ["hvm"],
21            },
22          }).getImage(this).imageId,
23        }),
24      },
25    },
26  ],
27});

See Ubuntu Amazon EC2 AMI Locator

Custom runner's role

To deploy from within your GitLab Runner Instances, you may pass a Role with the IAM Policies attached.

1const role = new Role(this, "RunnersRole", {
2  assumedBy: new ServicePrincipal("ec2.amazonaws.com", {}),
3  inlinePolicies: {},
4});
5const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
6  parameterName: "/gitlab-runner/token",
7});
8
9new GitlabRunnerAutoscaling(this, "Runner", {
10  runners: [
11    {
12      role: role,
13      token: token,
14      configuration: {
15        // optionally configure your runner
16      },
17    },
18  ],
19});

See example, GitlabRunnerAutoscalingProps

Vpc

If no existing Vpc is passed, a cheap VPC with a NatInstance (t3.nano) and a single AZ will be created.

1const natInstanceProvider = aws_ec2.NatProvider.instance({
2  instanceType: aws_ec2.InstanceType.of(InstanceClass.T3, InstanceSize.NANO), // using a cheaper gateway (not scalable)
3});
4const vpc = new Vpc(this, "Vpc", {
5  // Your custom vpc, i.e.:
6  natGatewayProvider: natInstanceProvider,
7  maxAzs: 1,
8});
9
10const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
11  parameterName: "/gitlab-runner/token",
12});
13
14new GitlabRunnerAutoscaling(this, "Runner", {
15  runners: [
16    {
17      token: token,
18      configuration: {
19        // optionally configure your runner
20      },
21    },
22  ],
23  network: { vpc: vpc },
24});

See example, GitlabRunnerAutoscalingProps

Zero config

Deploys the Autoscaling GitLab Runner on AWS EC2 with the default settings mentioned above.

Happy with the presets?

1const token = StringParameter.fromStringParameterAttributes(stack, "Token", {
2  parameterName: "/gitlab-runner/token",
3});
4
5new GitlabRunnerAutoscaling(this, "Runner", {
6  runners: [
7    {
8      token: token,
9      configuration: {
10        // optionally configure your runner
11      },
12    },
13  ],
14});

See example, GitlabRunnerAutoscalingProps

ECR Credentials Helper

By default, the GitLab amzonec2 driver will be configured to install the amazon-ecr-credential-helper on the runner's instances.

To configure, override the default job runners environment:

1new GitlabRunnerAutoscaling(this, "Runner", {
2  runners: [
3    {
4      // ...
5      environment: [
6        "DOCKER_DRIVER=overlay2",
7        "DOCKER_TLS_CERTDIR=/certs",
8        'DOCKER_AUTH_CONFIG={"credHelpers": { "public.ecr.aws": "ecr-login", "<aws_account_id>.dkr.ecr.<region>.amazonaws.com": "ecr-login" } }',
9      ],
10    },
11  ],
12});

Projen

This project uses projen to maintain project configuration through code. Thus, the synthesized files with projen should never be manually edited (in fact, projen enforces that).

To modify the project setup, you should interact with rich strongly-typed class AwsCdkTypeScriptApp and execute npx projen to update project configuration files.

In simple words, developers can only modify .projenrc.js file for configuration/maintenance and files under /src directory for development.

See also Create and Publish CDK Constructs Using projen and jsii.