Introducing the AWS GitLab Helper

What and why?

The AWS GitLab Helper (AGH) is a tool that can be implemented in GitLab Pipelines to automatically fetch temporary credentials, secrets and parameters from AWS using ABAC. I wasn’t happy with the suggested way of authenticating with AWS. Due to limitations of AWS OIDC provider implementation, you can only assert on the ID Token’s subject claim { "sub": "project_path:my-group/my-project:ref_type:branch:ref:feature-branch-1" }, which means you’re closely coupling security and your policies to Git References (tags or branches), which in my mind seems semantically wrong. Security should be coupled with environments and GitLab has Protected Environments - so why couldn’t I use that. 🤷‍♂️

Well it transpires that you can! 🚀

Say hello to AWS Cognito

AWS OIDC when used on it’s own can only use limited claims (see Available keys for AWS OIDC federation). After pouring over the docs I found Passing session tags using AssumeRoleWithWebIdentity, but alas it would only work if GitLab changed their ID Token to match AWS’s schema… I eventually found Using attributes for access control in the AWS Cognito Identity Pools documentation and got very excited.

Unfortunately you cannot hack with it using the aws-cli, so I had to use the SDK for JavaScript. And it was incredibly hacky and ugly code but it worked. 🎉

/insert "candy girl gif" # I would embed - but you'd probably leave [link]

The magic is in Cognito mappings:

“Attributes for access control” is 50% of the magic sauce. This takes the claims from the GitLab ID Token (JWT), and maps them to the AghCognito role. However they’re a bit janky to use directly. I don’t want my AWS Resources having to have tags like NamespacePath and ProjectPath. I describe my infrastructure using Domain, System, Service and Environment tags. I thought, if I assume a 2nd role I could rename them… assuming a 2nd role opens up the environment for abuse (changing the tags to a claim you’re not entitled to). However after too much coffee and some creative thinking I worked out a safe Trust Policy that would achieve what I wanted (it also has some extra features too).

Introducing the wild Trust Policy for AghAbac - the other 50%:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>:role/agh/AghCognito"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalTag/Environment": "${aws:RequestTag/Environment}",
                    "aws:PrincipalTag/ProjectPath": "${aws:PrincipalTag/NamespacePath}/${aws:RequestTag/Service}",
                    "aws:PrincipalTag/UserAccessLevel": [
                        "developer",
                        "maintainer",
                        "owner"
                    ],
                    "aws:PrincipalTag/EnvironmentProtected": "${aws:RequestTag/EnvironmentProtected}"
                },
                "Null": {
                    "aws:TagKeys": "false"
                },
                "StringLike": {
                    "aws:PrincipalTag/NamespacePath": "*/${aws:RequestTag/Domain}/${aws:RequestTag/System}"
                },
                "ForAllValues:StringEquals": {
                    "aws:TagKeys": [
                        "Domain",
                        "System",
                        "Service",
                        "Environment",
                        "EnvironmentProtected",
                        "UserAccessLevel"
                    ]
                }
            }
        }
    ]
}

Let me explain it:

So now I have a well configured ABAC role, what can I do? In my .gitlab-ci.yml file I can add the following code:

include: # @todo publish GitLab CI/CD component
  - project: msyea-sa/aws-gitlab-helper
    file: /templates/AwsGitLabHelper.gitlab-ci.yml

variables:
	AGH_CREDENTIAL_DEFAULT: "arn:aws:iam::123456789012:role/service-role/shared-system-role"
	AGH_CREDENTIAL_PROFILE1: "arn:aws:iam::123456789012:role/service-role/service-role"
	AGH_SECRET_SLACK_TOKEN: "arn:aws:secretsmanager:us-east-1:123456789012:secret:slack-token-V6d7a8"
	AGH_SECRET_TEST_DB: "arn:aws:secretsmanager:us-west-2:123456789012:secret:db-connection-string-F1r2t3"
	AGH_PARAMETER_MAX_RETRY: "arn:aws:ssm:us-west-2:123456789012:parameter/max-retry-attempts"

test:
  extends: [.aws-gitlab-helper]
  script:
    - aws --profile profile1 sts get-caller-identity
    - echo "Max retry: ${MAX_RETRY}" # **do not echo secrets as they will not be masked!**

The template executes in the GitLab Helper Container (hook:pre_get_sources_script). This is hugely advantageous as it doesn’t interfere with job scripts (like before_script might) and ${GITLAB_ENV} plays nicely (see below). As it’s in the GitLab Helper Container it knows curl is available and with uname -s it can download an execute the correct version from the GitLab Package Registry. The app is written in javascript/node and statically complied for Linux and macOS. This makes the download quick with no dependencies.

The AWS GitLab Helper parses the environment variables prefixed with ARG_, and, when necessary executes AssumeRole, GetSecretValue and GetParameter commands and saves the responses. To make them available to the job it does something like:

export AWS_SHARED_CREDENTIALS_FILE="${RUNNER_TEMP_PROJECT_DIR}/aws-credentials"
echo "credentials" >> ${AWS_SHARED_CREDENTIALS_FILE}
echo "secrets" >> ${GITLAB_ENV}

If you haven’t used ${GITLAB_ENV} before see Pass an environment variable from the script section to another section in the same job. (It’s much less janky than creating than doing: export $(xargs < .env)).

Your AWS credentials file is populated outside the ${CI_PROJECT_DIR}, so that it cannot be accidentally saved as an artifact [sic] or committed. The secrets and parameters are also made available as environment variables.

Click AGH Sequence Diagram button below to take a look at the complete sequence diagram.

AGH Sequence Diagram

sequenceDiagram
    participant Job as GitLab Job
    box transparent GitLab Helper Container hook
        participant AGH as AWS GitLab Helper
    end
    box transparent AWS
        participant Cognito
        participant STS
        participant SM as Secrets Manager
        participant PS as Parameter Store
    end
    Job->>+AGH: Implements AGH Template
    AGH->>AGH: Configures ID Token
    AGH->>AGH: Downloads AGH Binary<br/>and executes it
    Note right of AGH: Downloads AGH Binary<br/>and executes it
    AGH->>+Cognito: Authenticates
    Cognito->>Cognito: Maps claims to Principal Tags
    Cognito->>-AGH: Returns temporary credentials (AghCognito Role)
    AGH->>+STS: Assume AghAbac Role with AGH Tags
    STS->>-AGH: Returns temporary credentials (AghAbac Role)
    loop Fetch credentials for additional roles
        AGH->>+STS: Assume other Role with AGH Tags
        STS->>-AGH: Returns temporary credentials
        AGH->>Job: Write credentials to AWS_SHARED_CREDENTIALS_FILE
    end
    loop Fetch secrets
        AGH->>+SM: GetSecret
        SM->>-AGH: Returns secret
        AGH->>Job: Writes secret to GITLAB_ENV
    end
    loop Fetch parameters
        AGH->>+PS: GetParameter
        PS->>-AGH: Returns parameter
        AGH->>Job: Write parameter to GITLAB_ENV
    end
    AGH-xJob: AGH Binary exits<br/> and Helper Container completes
    %% destroy AGH
    alt GitLab Build Container
        Job->>Job: Runs your Job scripts
    end

And that is it. AGH is now available for you to use. I’ve been using it in production for a few months but I will improve the documentation and automated tests before I make it Generally Available.

Note that the AWS GitLab Helper is available under BSL -> MIT license. If you have 25 or fewer employees or less than £1 million in revenue you can use it in production for free. If you’re larger than that then you will need a Commercial License or wait 4 years.

Links