What and why?
The AWS GitLab Helper (AGH) is a tool that can be implemented in GitLab Pipelines to automatically fetch temporary credentials, secrets and parameters from AWS using ABAC.
I wasn’t happy with the suggested way of authenticating with AWS. Due to limitations of AWS OIDC provider implementation, you can only assert on the ID Token’s subject claim { "sub": "project_path:my-group/my-project:ref_type:branch:ref:feature-branch-1" }
, which means you’re closely coupling security and your policies to Git References (tags or branches), which in my mind seems semantically wrong. Security should be coupled with environments and GitLab has Protected Environments - so why couldn’t I use that. 🤷♂️
Well it transpires that you can! 🚀
Say hello to AWS Cognito
AWS OIDC when used on it’s own can only use limited claims (see Available keys for AWS OIDC federation). After pouring over the docs I found Passing session tags using AssumeRoleWithWebIdentity, but alas it would only work if GitLab changed their ID Token to match AWS’s schema… I eventually found Using attributes for access control in the AWS Cognito Identity Pools documentation and got very excited.
Unfortunately you cannot hack with it using the aws-cli
, so I had to use the SDK for JavaScript. And it was incredibly hacky and ugly code but it worked. 🎉
/insert "candy girl gif" # I would embed - but you'd probably leave
[link]
The magic is in Cognito mappings:
“Attributes for access control” is 50% of the magic sauce. This takes the claims from the GitLab ID Token (JWT), and maps them to the AghCognito
role. However they’re a bit janky to use directly. I don’t want my AWS Resources having to have tags like NamespacePath
and ProjectPath
. I describe my infrastructure using Domain
, System
, Service
and Environment
tags. I thought, if I assume a 2nd role I could rename them… assuming a 2nd role opens up the environment for abuse (changing the tags to a claim you’re not entitled to). However after too much coffee and some creative thinking I worked out a safe Trust Policy that would achieve what I wanted (it also has some extra features too).
Introducing the wild Trust Policy for AghAbac
- the other 50%:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account-id>:role/agh/AghCognito"
},
"Action": [
"sts:AssumeRole",
"sts:TagSession"
],
"Condition": {
"StringEquals": {
"aws:PrincipalTag/Environment": "${aws:RequestTag/Environment}",
"aws:PrincipalTag/ProjectPath": "${aws:PrincipalTag/NamespacePath}/${aws:RequestTag/Service}",
"aws:PrincipalTag/UserAccessLevel": [
"developer",
"maintainer",
"owner"
],
"aws:PrincipalTag/EnvironmentProtected": "${aws:RequestTag/EnvironmentProtected}"
},
"Null": {
"aws:TagKeys": "false"
},
"StringLike": {
"aws:PrincipalTag/NamespacePath": "*/${aws:RequestTag/Domain}/${aws:RequestTag/System}"
},
"ForAllValues:StringEquals": {
"aws:TagKeys": [
"Domain",
"System",
"Service",
"Environment",
"EnvironmentProtected",
"UserAccessLevel"
]
}
}
}
]
}
Let me explain it:
Desired Tag | Assertion | Explanation |
---|---|---|
Domain and System |
"${aws:PrincipalTag/NamespacePath}" like "*/${aws:RequestTag/Domain}/${aws:RequestTag/System}" |
The NamespacePath contains root-group/subgroup1/subgroup2 and my GitLab group hierarchy is root-group/domain/system . |
Service |
"${aws:PrincipalTag/NamespacePath}/${aws:RequestTag/Service}" == "${aws:PrincipalTag/ProjectPath}" |
The ProjectPath contains root-group/subgroup1/subgroup2/service and my GitLab service hierarchy is root-group/domain/system/service . |
Environment |
"${aws:RequestTag/Environment}" == "${aws:PrincipalTag/Environment}" |
|
EnvironmentProtected |
"${aws:RequestTag/EnvironmentProtected}" == "${aws:PrincipalTag/EnvironmentProtected}" |
I’ve added this as in my other Policies I assert that "Environment" = "production" also has "EnvironmentProtected" = "true" . |
UserAccessLevel |
"${aws:PrincipalTag/UserAccessLevel}" in ["owner", "maintainer", "owner"] |
This gives me the option to block developers from ever interacting with some resources. |
So now I have a well configured ABAC role, what can I do?
In my .gitlab-ci.yml
file I can add the following code:
include: # @todo publish GitLab CI/CD component
- project: msyea-sa/aws-gitlab-helper
file: /templates/AwsGitLabHelper.gitlab-ci.yml
variables:
AGH_CREDENTIAL_DEFAULT: "arn:aws:iam::123456789012:role/service-role/shared-system-role"
AGH_CREDENTIAL_PROFILE1: "arn:aws:iam::123456789012:role/service-role/service-role"
AGH_SECRET_SLACK_TOKEN: "arn:aws:secretsmanager:us-east-1:123456789012:secret:slack-token-V6d7a8"
AGH_SECRET_TEST_DB: "arn:aws:secretsmanager:us-west-2:123456789012:secret:db-connection-string-F1r2t3"
AGH_PARAMETER_MAX_RETRY: "arn:aws:ssm:us-west-2:123456789012:parameter/max-retry-attempts"
test:
extends: [.aws-gitlab-helper]
script:
- aws --profile profile1 sts get-caller-identity
- echo "Max retry: ${MAX_RETRY}" # **do not echo secrets as they will not be masked!**
The template executes in the GitLab Helper Container (hook:pre_get_sources_script
). This is hugely advantageous as it doesn’t interfere with job scripts (like before_script
might) and ${GITLAB_ENV}
plays nicely (see below). As it’s in the GitLab Helper Container it knows curl
is available and with uname -s
it can download an execute the correct version from the GitLab Package Registry. The app is written in javascript/node and statically complied for Linux and macOS. This makes the download quick with no dependencies.
The AWS GitLab Helper parses the environment variables prefixed with ARG_
, and, when necessary executes AssumeRole
, GetSecretValue
and GetParameter
commands and saves the responses. To make them available to the job it does something like:
export AWS_SHARED_CREDENTIALS_FILE="${RUNNER_TEMP_PROJECT_DIR}/aws-credentials"
echo "credentials" >> ${AWS_SHARED_CREDENTIALS_FILE}
echo "secrets" >> ${GITLAB_ENV}
If you haven’t used ${GITLAB_ENV}
before see Pass an environment variable from the script
section to another section in the same job. (It’s much less janky than creating than doing: export $(xargs < .env)
).
Your AWS credentials file is populated outside the ${CI_PROJECT_DIR}
, so that it cannot be accidentally saved as an artifact [sic] or committed. The secrets and parameters are also made available as environment variables.
Click AGH Sequence Diagram button below to take a look at the complete sequence diagram.
sequenceDiagram
participant Job as GitLab Job
box transparent GitLab Helper Container hook
participant AGH as AWS GitLab Helper
end
box transparent AWS
participant Cognito
participant STS
participant SM as Secrets Manager
participant PS as Parameter Store
end
Job->>+AGH: Implements AGH Template
AGH->>AGH: Configures ID Token
AGH->>AGH: Downloads AGH Binary<br/>and executes it
Note right of AGH: Downloads AGH Binary<br/>and executes it
AGH->>+Cognito: Authenticates
Cognito->>Cognito: Maps claims to Principal Tags
Cognito->>-AGH: Returns temporary credentials (AghCognito Role)
AGH->>+STS: Assume AghAbac Role with AGH Tags
STS->>-AGH: Returns temporary credentials (AghAbac Role)
loop Fetch credentials for additional roles
AGH->>+STS: Assume other Role with AGH Tags
STS->>-AGH: Returns temporary credentials
AGH->>Job: Write credentials to AWS_SHARED_CREDENTIALS_FILE
end
loop Fetch secrets
AGH->>+SM: GetSecret
SM->>-AGH: Returns secret
AGH->>Job: Writes secret to GITLAB_ENV
end
loop Fetch parameters
AGH->>+PS: GetParameter
PS->>-AGH: Returns parameter
AGH->>Job: Write parameter to GITLAB_ENV
end
AGH-xJob: AGH Binary exits<br/> and Helper Container completes
%% destroy AGH
alt GitLab Build Container
Job->>Job: Runs your Job scripts
end
And that is it. AGH is now available for you to use. I’ve been using it in production for a few months but I will improve the documentation and automated tests before I make it Generally Available.
Note that the AWS GitLab Helper is available under BSL -> MIT license. If you have 25 or fewer employees or less than £1 million in revenue you can use it in production for free. If you’re larger than that then you will need a Commercial License or wait 4 years.
Links
Component | Location |
---|---|
Template | https://gitlab.com/msyea-sa/aws-gitlab-helper/-/tree/main/templates |
Application | https://gitlab.com/msyea-sa/aws-gitlab-helper/-/tree/main/app |
Terraform Module | https://gitlab.com/msyea-sa/aws-gitlab-helper/-/tree/main/terraform |