Quickstart with AWS KMS XKS (External Key Store)

Want to try AWS KMS XKS but don’t have an expensive Key Management solution on-premise already? Here’s how…

Background

Released during reInvent 2022, AWS KMS XKS allows AWS KMS to forward API calls to securely communicate with your HSM. Your key material never leaves your HSM. This solution allows you to encrypt data with external keys for the vast majority of AWS services that support AWS KMS customer managed keys, such as Amazon EBS, AWS Lambda, Amazon S3, Amazon DynamoDB, and over 100 more services.

With some companies wanting to directly manage their key material this is a great new feature providing you can provide the reliability of access required for such critical infrastructure.

Pre-Requisites

• Suitable domain/FQDN to direct towards on-prem
• Ability to make provide SSL termination (letsencrypt for example) for that FQDN
• Necessary services to forward external internet requests to a docker container
• Docker and git installed

In my lab I use pfsense as an internet gateway, firewall, and use the haproxy functionality for SSL termination, with a wildcard letsencrypt certificate. I registered an additional CNAME in my route53 domain, and ran the docker containers within one of my Ubuntu 22 VM’s. You can host it in AWS as EC2 if preferred of course.

Setting up On-Premise

1. git clone https://github.com/aws-samples/aws-kms-xks-proxy
2. Adapt the xks-proxy settings. To do this customise this file: aws-kms-xks-proxy/xks-axum/configuration/settings_docker.toml
3. The changes required can be limited to those below. Note these are your preferences, they only need to meet the requirements. Some URI prefix is required and I kept it to two. Key ID can be left at ‘foo’ or change it, but if changed must be reflected in the SoftHSM config too.

[[external_key_stores]]
uri_path_prefix = "/davecloud/secure"
# Access key ID must have between 20 and 30 characters. Valid characters are uppercase A-Z and 2-7
sigv4_access_key_id = "DAVECLOUDxxxxxxxxxxxxxxx"
# Secret access key must have between 43 and 64 characters. Valid characters are a-z, A-Z, 0-9, /, +, and =
sigv4_secret_access_key = "rekjhrkjerkhekhrkjehrkjretZ6NSD4K1aTRMUPjOxxxxxxDE3ritZ6"
xks_key_id_set = ["foo"]

Also note if you are using a region other than us-east-1 the file must also be updated near the top here:

# This is a sample xks-proxy configuration file of using SoftHSMv2 with pkcs11-logger disabled in a Docker environment.
[server]
region = "eu-west-1"

4. Adapt the xks-proxy Dockerfile. To do this customise this file: aws-kms-xks-proxy/Dockerfile . If you left the key_id external name as ‘foo’ you can skip this step, if you changed it, you must reflect the change here too. You can also update pins but again this must be reflected in the xks-proxy file too.

RUN softhsm2-util --init-token --slot 0 --label "xks-proxy" --so-pin 1234 --pin 1234
RUN pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
                --token-label xks-proxy --login --login-type user \
                --keygen --id F0 --label foo --key-type aes:32 \
                --pin 1234

5. Build the docker container: docker build -t xks-proxy:latest .

6. Run the docker container:

docker run — rm — name xks-proxy -d -p 0.0.0.0:80:80 xks-proxy:latest

7. Update your DNS, web forwarding and SSL termination rules accordingly.

8. Test you have working connectivity from the internet

~$ curl https://kms.davecloud.org
{"errorName":"InvalidUriPathException"}

Note: The Docker container will recreate its key each time it’s rebuilt (though, not each time it’s started). If you think you might need/want to re-run config multiple times afterwards it might be a good idea to persist the configuration through mounts etc.

Setting up AWS

1. Login to AWS , go to the KMS console, and ensure you are within the region that you would like to place the KMS registration.
2. Select Custom Key Stores on the left, External Key Stores, and select Create External Key Store button on the right
3. Fill out all prescribed information:

4. Create the key store, and then click to connect it

5. Now create a new key within the store

6. Fill out all prescribed fields, including external key id. This is ‘foo’ unless you changed it in earlier steps:

7. Define users/administrators as usual

8. You’re Done!

Testing it Out!

Heres the key i’ve registered in my on-premise (home!) KMS:

creating a new VM with this new key backing the root volume ebs encryption:

and here it is running:

and finally, here’s an object uploaded to S3 with the related KMS telemetry coming from docker. If you want to see what the kms proxy is up to type:

docker logs xks-proxy --follow

Monitoring

I thought i’d call out that the status page on the external key store page gives a number of interesting metrics. Note that AWS insist on a <250ms latency for KMS requests for productive use. The spike you see here is me intentionally breaking things :)

Worth noting that AWS KMS is health checking the External Store at 3min intervals to measure latency and verify health.

I hope you enjoyed the blog!