Vault AWS Lambda extension

18min
|
Terraform
Vault

AWS Lambda lets you run code without provisioning and managing servers. Lambda functions perform processing when you need it, and now can leverage Vault secrets through the Vault Lambda Extension.

Challenge

A database within your AWS infrastructure maintains sensitive information. A periodic function is required to audit and update the data stored. This function must authenticate with Vault and receive database credentials.

Solution

Lambda Extensions are Lambda layers used to augment Lambda functions, and you can use them to give the Lambda function access to the Vault cluster. During execution, the Lambda layer packages the extension code with the main Lambda function code.

In this lab you will create an AWS Lambda function that uses the Vault Lambda Extension to authenticate with Vault, receive the database credentials, and perform the necessary queries.

Prerequisites

This tutorial requires an AWS account, Terraform, go, AWS CLI, Docker, and additional configuration to create a demonstration environment.

Create an AWS account with AWS credentials and a EC2 key pair
Install Terraform
Install go
Install AWS CLI
Install Docker

Scenario introduction

You learn about the Vault Lambda Extension by creating a local scenario environment, using Terraform configuration to deploy supporting AWS-based infrastructure which you will then use to configure Vault for use with the AWS auth method.

Finally, you will build an example Lambda function that uses the Vault Lambda Extension to read secrets from Vault, and use Terraform to deploy it.

Create environment

You can retrieve the lambda function and terraform configuration by cloning the hashicorp-education/learn-vault-lambda-extension repository from GitHub.

$ git clone https://github.com/hashicorp-education/learn-vault-lambda-extension

This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.

Change into the learn-vault-lambda-extension directory.

$ cd learn-vault-lambda-extension

Working directory

This tutorial assumes that you execute all following commands from within this directory.

The demonstration environment sets up a Vault server, a PostgreSQL database, and configures the database secrets engine to generate dynamic credentials. The infrastructure is described by the Terraform configuration files in the directory.

Create an environment variable named AWS_ACCESS_KEY_ID with your AWS access key ID.

$ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"

Create an environment variable named AWS_SECRET_ACCESS_KEY with your AWS secret access key.

$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"

These environment variables are used by Terraform and the AWS CLI to authenticate and create resources.

Tip

The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.

Copy terraform.tfvars.example and rename to terraform.tfvars.

$ cp terraform.tfvars.example terraform.tfvars

Edit terraform.tfvars to override the default settings that describe your environment.

# AWS region and AZs in which to deploy
# default: 'us-east-1'
aws_region = "us-east-1"

# All resources will be tagged with this
# default: 'vault-lambda-extension-demo'
environment_name = "vault-lambda-extension-demo"

# URL for Vault binary
# default: Vault v1.10.0
vault_zip_file = "https://releases.hashicorp.com/vault/1.10.0/vault_1.10.0_linux_amd64.zip"

# Instance size
# default: 't2.micro'
instance_type = "t2.micro"

# DB instance size
# default: 'db.t2.micro'
db_instance_type = "db.t2.micro"

Initialize Terraform.

$ terraform init

The initialization retrieves the additional modules required to apply the resources defined within the configuration.

Apply Terraform to review the planned actions.

$ terraform apply
...

The terminal output displays the plan that it found and the resources it creates.

Enter yes to confirm and resume.

Note

Keep in mind that answering yes at this time will create actual resources with associated costs.

When the terraform apply command completes, the Terraform output displays the IP addresses of the Vault server and other services created.

PostgreSQL database address
    terraform-20201118041217579300000001.ci12bl452q7n.us-east-1.rds.amazonaws.com

Elastic Container Registry (ECR) repository address
    906192817044.dkr.ecr.us-east-1.amazonaws.com/demo-function

Vault Server IP (public)
    3.82.241.157

Vault UI URL
    http://3.82.241.157:8200/ui

You can SSH into the Vault EC2 instance using private.key:
    ssh -i private.key ubuntu@3.82.241.157

The output displays the database address, the container repository address, and connection information for the Vault server. The terraform configuration also generated a private key and enabled SSH login on the Vault server. An example SSH command is provided in the output.

In a new terminal session, change into the scenario directory.

$ cd learn-vault-lambda-extension

Then SSH into the Vault Server.

$ ssh -i private.key ubuntu@3.82.241.157

A single Vault server is running with the filesystem storage backend. The Terraform configuration automatically initialized and unsealed the server.

Verify that Vault is ready, initialized, and unsealed.

$ vault status
Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    5
Threshold                3
Version                  1.10.0
Storage Type             file
Cluster Name             vault-cluster-07d67d93
Cluster ID               04353a37-6208-775d-4655-b5e3bfb53d5f
HA Enabled               false

If the value of Initialized is true and the value of Sealed is false, then the Vault server is ready for use.

Note

The Vault server was configured with the template found in ./templates/userdata-vault-server.tpl.

The database secrets engine is already enabled and configured to communicate with the database through credentials generated for the role lambda-function.

Read credentials from the lambda-function database role.

$ vault read database/creds/lambda-function
Key                Value
---                -----
lease_id           database/creds/lambda-function/V2KfrawfpkJqGYr2N5cFaSH0
lease_duration     1h
lease_renewable    true
password           A1a-9ihozcMd4RGqmgfA
username           v-root-lambda-f-ilX2242kMoLeWnY7ZqKS-1605567098

The PostgreSQL credentials are displayed as username and password.

Read the Dynamic Secrets: Database Secrets Engine tutorial to learn about configuring the database secrets engine.

Set the value of environment variables DB_USER and DB_PASSWORD to another set of credentials read from the role.

$ read -d "\n" DB_USER DB_PASSWORD <<<$(vault read database/creds/lambda-function -format=json | jq -r ".data.username,.data.password")

The same read operation is performed. The results are returned in JSON format, and parsed for the username and password fields. These two values are then read into the values of variables DB_USER and DB_PASSWORD.

The PostgreSQL client tool requires this username, password, and host address to connect with its command-line tool. The database's address is stored in an environment variable named DB_HOST.

Display the database address.

$ echo $DB_HOST

Connect to the PostgreSQL database via the CLI with the credentials and host.

$ PGPASSWORD=$DB_PASSWORD psql -h $DB_HOST -d postgres -U $DB_USER

Your system prompt is replaced with a new prompt root=#. Commands issued at this prompt are executed against the PostgreSQL database running within the container.

List all the database users.

$ SELECT usename, valuntil FROM pg_user;
                       usename                       |        valuntil
-----------------------------------------------------+------------------------
 vaultadmin                                          | infinity
 rdsadmin                                            | infinity
 v-root-lambda-f-0BtQGCRTzGm2QgWHgv09-1647293547     | 2022-03-14 22:32:32+00
 v-root-lambda-f-DBN5PYDVkZ7u7yYymt47-1647293577     | 2022-03-14 22:33:02+00
(4 rows)

The output displays a table of all the database credentials generated. The credentials that were recently generated appear in this list.

Disconnect from the PostgreSQL database.

$ \q

Vault issued database credentials, successfully connected to the database and queried the users within the database.

Enable AWS authentication

The AWS Lambda function created in this step performs the same operation of requesting credentials and performing a query. The function needs its own set of Vault credentials that it requests through the AWS authentication engine.

Enable the AWS authentication engine.

$ vault auth enable aws

Configure the AWS client to use the default options.

$ vault write -force auth/aws/config/client

The lambda function needs credentials to query the database credentials at the configured path.

Create a policy named lambda-function.

$ vault policy write lambda-function - <<EOF
path "database/creds/lambda-function" {
    capabilities = ["read"]
}
EOF

The policy grants the read capability to the database/creds/lambda-function. A policy is assigned to a token during the authentication through a role.

Read the Vault Policies tutorial to learn about defining Vault policies.

Create a role prefixed with the AWS environment name.

$ vault write auth/aws/role/$AWS_IAM_ROLE_LAMBDA_NAME \
    auth_type=iam \
    bound_iam_principal_arn="arn:aws:iam::$AWS_ACCOUNT_ID:role/$AWS_IAM_ROLE_LAMBDA_NAME" \
    policies=lambda-function \
    ttl=5m

The Vault role connects the AWS IAM role, created by the Terraform configuration in ./iam.tf and the Vault policy, lambda-function. The token returned after authentication is valid for 5 minutes.

Enable Audit Device Log

You can observe the requests and responses associated with deploying the Lambda in a Vault audit device log. If you choose to try the caching example later in the scenario, the audit log is helpful for observing the lack of requests as well.

Enable a filesystem audit device.

$ vault audit enable file file_path=/tmp/vault_audit.log

The Vault Server is configured and ready for the AWS Lambda function to authenticate.

Tail the audit device log.

$ sudo tail -f /tmp/vault_audit.log

Successful output:

{"time":"2022-03-21T15:57:12.602098086Z","type":"request","auth":{"token_type":"default"},"request":{"id":"77f6f837-5b68-6097-5b02-5c37ccc4e124","operation":"update","namespace":{"id":"root"},"path":"sys/audit/test"}}
{"time":"2022-03-21T15:57:12.60420102Z","type":"response","auth":{"client_token":"hmac-sha256:2bfa837462c4c3997ca6b82aaf2047bd20a3c886198d2db81b240a798a591642","accessor":"hmac-sha256:6a8a815bc0554a66b8b5ebfc82641b03297cda2acfe285efe1ec6de7cf7a1f25","display_name":"root","policies":["root"],"token_policies":["root"],"token_type":"service","token_issue_time":"2022-03-21T15:27:59Z"},"request":{"id":"aa7dc060-69ab-f436-7fca-0fa67c16ca7b","operation":"update","mount_type":"system","client_token":"hmac-sha256:2bfa837462c4c3997ca6b82aaf2047bd20a3c886198d2db81b240a798a591642","client_token_accessor":"hmac-sha256:6a8a815bc0554a66b8b5ebfc82641b03297cda2acfe285efe1ec6de7cf7a1f25","namespace":{"id":"root"},"path":"sys/audit/file","data":{"description":"hmac-sha256:155ca16daa7e61da1d2e81b5bf7681dabe180d00d063784fdbd2ad1ab8309413","local":false,"options":{"file_path":"hmac-sha256:b91e77649a56647b56e76d65267801b02211b3f6ef79e85ddc6e462b90a6dc5e"},"type":"hmac-sha256:e4b3338f90bf149a84f11ea34df4b00667ddcff000f288f41fe90153beaa2f65"},"remote_address":"172.31.25.24","remote_port":58854},"response":{"mount_type":"system"}}

There is a single request and response pair present in the log.

Build and package lambda function

Return to your original terminal session for the next steps.

An AWS lambda function is code that is packaged and delivered to a specific environment where it is executed. The package is an archive or a container image.

The lambda function for this tutorial is written in Go. It reads in the location of the Vault server and credentials provided to it. It requests dynamic credentials to gain access to the PostgreSQL database and performs a query to display the current list of database users.

Clean, build, and archive the lambda function.

$ ./build.sh

The demo function archive is found the path demo-function/demo-function.zip.

Display the archived lambda function.

$ ls demo-function/demo-function.zip
demo-function/demo-function.zip

The archived function can now become an AWS lambda function.

The container image requires the Vault Lambda Extension executable and an executable version of the code.

Retrieve the Vault Lambda Extension layer.

$ curl --silent $(aws lambda get-layer-version-by-arn --arn arn:aws:lambda:us-east-1:634166935893:layer:vault-lambda-extension:13 --query 'Content.Location' --output text --region us-east-1) \
  --output vault-lambda-extension.zip

This request returns an archive that contains the vault-lambda-extension executable.

Unzip the Vault lambda extension.

$ unzip vault-lambda-extension.zip

The extension is found the path extensions/vault-lambda-extension.

Display the vault-lambda-extension executable.

$ ls extensions/vault-lambda-extension

This executable and the source are packaged together in a container image with definition found in the Dockerfile.

Create the Docker image named it demo-function tagged latest.

$ docker build -t demo-function:latest  .

The Docker image exists locally, for the AWS Lambda function the image must be pushed to the ECR repository.

Generate a Docker login for the ECR repository.

$ aws ecr get-login
docker login -u AWS \
  -p eyJwY...5NDh9 \
  -e none \
  https://906192817044.dkr.ecr.us-east-1.amazonaws.com

The results display a command to execute to login to the ECR repository.

Example:

$ docker login -u AWS \
  -p eyJwY...5NDh9 \
  -e none \
  https://906192817044.dkr.ecr.us-east-1.amazonaws.com

Note

If you receive the an error similar to Error "unknown shorthand flag: 'e' in -e", the generated command may include a flag that is not supported. Re-run the command with the flag and value removed.

The address of the container repository is present in the terraform output and in the login command.

Create a variable named ECR_REPO_ADDR that stores the Amazon ECR repository address.

$ ECR_REPO_ADDR=<Elastic Container Registry (ECR) repository address>

Tag the Docker image in preparation for publishing to the repo.

$ docker tag demo-function:latest ${ECR_REPO_ADDR}/demo-function:latest

Publish the Docker image to the ECR repository.

$ docker push ${ECR_REPO_ADDR}/demo-function:latest

The container image can now become an AWS lambda function.

Create the lambda function

The lambda function is packaged and is ready for the creation of an AWS Lambda resource. Terraform provides the aws_lambda_function resource that enables you to create the lambda function with an archive or container image.

Rename the file lambda-as_an_archive.tf.disabled to lambda.tf.

$ cp lambda-as_an_archive.tf.disabled lambda.tf

Display the contents of the lambda.tf file.

$ cat lambda.tf

Successful output:

resource "aws_lambda_function" "function" {
  function_name = "${var.environment_name}-function"
  description   = "Demo Vault AWS Lambda extension"
  role          = aws_iam_role.lambda.arn
  filename      = "./demo-function/demo-function.zip"
  handler       = "main"
  runtime       = "provided.al2"
  layers        = ["arn:aws:lambda:${var.aws_region}:634166935893:layer:vault-lambda-extension:13"]

  environment {
    variables = {
      VAULT_ADDR           = "http://${aws_instance.vault-server.public_ip}:8200",
      VAULT_AUTH_ROLE      = aws_iam_role.lambda.name,
      VAULT_AUTH_PROVIDER  = "aws",
      VAULT_SECRET_PATH_DB = "database/creds/lambda-function",
      VAULT_SECRET_FILE_DB = "/tmp/vault_secret.json",
      DATABASE_URL         = aws_db_instance.main.address
    }
  }
}

output "lambda" {
  value = <<EOF

The lambda function is ready to run.

    aws lambda invoke --function-name ${aws_lambda_function.function.function_name} /dev/null \
        --log-type Tail \
        --region ${var.aws_region} \
        | jq -r '.LogResult' \
        | base64 --decode

EOF

This file contains two resources. The resource to create the lambda and output to display how the lambda is executed via AWS CLI.

The aws_lambda_function loads the packaged function found at the path defined at filename. To connect and authenticate with Vault several environment variables are defined:

Rename the file lambda-as_a_container.tf.disabled to lambda.tf.

$ cp lambda-as_a_container.tf.disabled lambda.tf

Display the contents of the lambda.tf file.

$ cat lambda.tf

Successful output:

resource "aws_lambda_function" "function" {
  function_name = "${var.environment_name}-function"
  description   = "Demo Vault AWS Lambda extension in container"
  role          = aws_iam_role.lambda.arn
  image_uri     = "${aws_ecr_repository.demo-function.repository_url}/demo-function:latest"
  package_type  = "Image"

  environment {
    variables = {
      VAULT_ADDR           = "http://${aws_instance.vault-server.public_ip}:8200",
      VAULT_AUTH_ROLE      = aws_iam_role.lambda.name,
      VAULT_AUTH_PROVIDER  = "aws",
      VAULT_SECRET_PATH_DB = "database/creds/lambda-function",
      VAULT_SECRET_FILE_DB = "/tmp/vault_secret.json",
      DATABASE_URL         = aws_db_instance.main.address
    }
  }
}

output "lambda" {
  value = <<EOF

The lambda function is ready to run.

    aws lambda invoke --function-name ${aws_lambda_function.function.function_name} /dev/null \
        --log-type Tail \
        --region ${var.aws_region} \
        | jq -r '.LogResult' \
        | base64 --decode

EOF
}

This file contains two resources. The resource to create the lambda and output to display how the lambda is executed via AWS CLI.

The aws_lambda_function loads the image found at the path defined at image_uri. To connect and authenticate with Vault several environment variables are defined:

VAULT_ADDR is the address of the Vault server. An AWS instance resource named `aws_instance.vault-server contains that address.
VAULT_AUTH_ROLE is the IAM role used to perform the function.
VAULT_AUTH_PROVIDER is the authentication method to use.
VAULT_SECRET_PATH_DB is the Vault path to read the database credentials
VAULT_SECRET_FILE_DB is the path that the credentials are saved. This path /tmp/vault_secret.json is defined in the custom function. DATABASE_URL is an environment variable defined in the custom function.

Run terraform apply and review the planned actions. Your terminal output indicates the plan that it found and what resources will be created.

$ terraform apply

Enter yes to confirm and resume.

When the terraform apply command completes, the lambda function is created and the additional output displays how to invoke it.

lambda =
The lambda function is ready to run.

    aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

Invoke the lambda function.

$ aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

The output displays the logging and the current accounts found within the database.

START RequestId: 0999f92b-1a57-4e77-a47a-2e17682038a2 Version: $LATEST
c487ab6c-e834-4d1f-a333-8fcae8e1c0e1[vault-lambda-extension] 2022/03/14 21:50:55 Initialising
[vault-lambda-extension] 2022/03/14 21:50:55 attemping Vault login...
[runtime] handler in bootstrap: main
[runtime] Initializing...
[runtime] Waiting for invocation...
[vault-lambda-extension] 2022/03/14 21:50:55 Initialised
[vault-lambda-extension] 2022/03/14 21:50:55 Waiting for event...
EXTENSION   Name: vault-lambda-extension    State: Ready    Events: [INVOKE,SHUTDOWN]
[vault-lambda-extension] 2022/03/14 21:50:55 Received event
[vault-lambda-extension] 2022/03/14 21:50:55 Waiting for event...
[runtime] Received invocation: {}
[runtime] Executing function: main
[demo-function] Received:
[demo-function] Reading file /tmp/vault_secret.json
[demo-function] users:
[demo-function]      vaultadmin
[demo-function]      rdsadmin
[demo-function]      v-root-lambda-f-0BtQGCRTzGm2QgWHgv09-1647293547
[demo-function]      v-root-lambda-f-DBN5PYDVkZ7u7yYymt47-1647293577
[demo-function]      v-aws-vaul-lambda-f-P0akiqTACqHLWDCssTyj-1647294655
END RequestId: 0999f92b-1a57-4e77-a47a-2e17682038a2
REPORT RequestId: 0999f92b-1a57-4e77-a47a-2e17682038a2  Duration: 1076.90 ms    Billed Duration: 1185 ms    Memory Size: 128 MBMax Memory Used: 46 MB   Init Duration: 107.88 ms

The lambda function successfully executed.

Check the other terminal session, and you will observe a pair of new requests and responses corresponding to the Lambda execution.

Example output:

{"time":"2022-03-21T16:04:47.651293247Z","type":"request","auth":{"token_type":"default"},"request":{"id":"96877994-df04-1053-2fe5-d139cf9159e2","operation":"update","mount_type":"aws","namespace":{"id":"root"},"path":"auth/aws/login","data":{"iam_http_request_method":"hmac-sha256:05864ff6246edd8910fd4f6c4e04b75c7b1dcad3c0333940b87bf5fbf46f3bea","iam_request_body":"hmac-sha256:a6b996161178463d6e4cda2e9176c129c21bd3e8105a8053fba90fdbd1069117","iam_request_headers":"hmac-sha256:4d22c5076f549af3e00ed95f943481f166cf5a484dacda60653abf64d688a750","iam_request_url":"hmac-sha256:de80ceb22a3568566f33a623d7bcf03cc0933c5439e9ded75013ba07236f6451","role":"hmac-sha256:0796f22fa9fecbc94c1ab6c60c15e88d6eb61cbd552600a4a6373b5268cc8228"},"remote_address":"3.81.200.22","remote_port":58258}}
{"time":"2022-03-21T16:04:47.677761356Z","type":"response","auth":{"client_token":"hmac-sha256:1ec29e3cd8d52420bedbc4408c0cb2bbe31a6ecc31988de6c8c7dc54936e0857","accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300},"request":{"id":"96877994-df04-1053-2fe5-d139cf9159e2","operation":"update","mount_type":"aws","namespace":{"id":"root"},"path":"auth/aws/login","data":{"iam_http_request_method":"hmac-sha256:05864ff6246edd8910fd4f6c4e04b75c7b1dcad3c0333940b87bf5fbf46f3bea","iam_request_body":"hmac-sha256:a6b996161178463d6e4cda2e9176c129c21bd3e8105a8053fba90fdbd1069117","iam_request_headers":"hmac-sha256:4d22c5076f549af3e00ed95f943481f166cf5a484dacda60653abf64d688a750","iam_request_url":"hmac-sha256:de80ceb22a3568566f33a623d7bcf03cc0933c5439e9ded75013ba07236f6451","role":"hmac-sha256:0796f22fa9fecbc94c1ab6c60c15e88d6eb61cbd552600a4a6373b5268cc8228"},"remote_address":"3.81.200.22","remote_port":58258},"response":{"auth":{"client_token":"hmac-sha256:1ec29e3cd8d52420bedbc4408c0cb2bbe31a6ecc31988de6c8c7dc54936e0857","accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300},"mount_type":"aws"}}
{"time":"2022-03-21T16:04:47.681203967Z","type":"request","auth":{"client_token":"hmac-sha256:1ec29e3cd8d52420bedbc4408c0cb2bbe31a6ecc31988de6c8c7dc54936e0857","accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300,"token_issue_time":"2022-03-21T16:04:47Z"},"request":{"id":"a81b297f-f25c-5b72-9e61-f0d75ec59f50","client_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","operation":"read","mount_type":"database","client_token":"hmac-sha256:60df7c00cd51d1b90731b88c2f8f8df9e6c458fa884873fd5c28206a13cc08e8","client_token_accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","namespace":{"id":"root"},"path":"database/creds/lambda-function","remote_address":"3.81.200.22","remote_port":58258}}
{"time":"2022-03-21T16:04:47.689257463Z","type":"response","auth":{"client_token":"hmac-sha256:1ec29e3cd8d52420bedbc4408c0cb2bbe31a6ecc31988de6c8c7dc54936e0857","accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300,"token_issue_time":"2022-03-21T16:04:47Z"},"request":{"id":"a81b297f-f25c-5b72-9e61-f0d75ec59f50","operation":"read","mount_type":"database","client_token":"hmac-sha256:60df7c00cd51d1b90731b88c2f8f8df9e6c458fa884873fd5c28206a13cc08e8","client_token_accessor":"hmac-sha256:4ed70834f9ef6189d250f3b4f8a3068b75c50334f4535a4f426120ff426795c2","namespace":{"id":"root"},"path":"database/creds/lambda-function","remote_address":"3.81.200.22","remote_port":58258},"response":{"mount_type":"database","secret":{"lease_id":"database/creds/lambda-function/ygpRENQOeivgbsYdInv0kJob"},"data":{"password":"hmac-sha256:a3bf456c0ed398d712d78fdbc5732525999540d9b6e11caa916619262178a1d4","username":"hmac-sha256:0957fb3df371718a327630bfc5563b609bfe29523b3ddba775e8887deb02a287"}}}

Caching

If your use case can benefit from it, caching can be configured for the Vault Lambda Extension local proxy server so that it does not forward every request to the Vault server.

You can enable this caching by setting a VAULT_DEFAULT_CACHE_TTL variable in your lambda.tf configuration with a value that can be parsed as a Go time duration.

For example, to specify a 5 minute TTL you can add VAULT_DEFAULT_CACHE_TTL = "5m" to your variables. Now when the extension is run and clients make GET requests using the header X-Vault-Cache-Control: cache, the request will be returned directly from the cache if there's a cache hit. On a cache miss the request will be forwarded to Vault and the response returned and cached.

If you set VAULT_DEFAULT_CACHE_ENABLED to true as in this example, then all requests are cached; otherwise you must set the HTTP header X-Vault-Cache-Control: cache on requests that you wish to be cached. Consult the Vault Lambda Extension documentation for more detail.

To try caching with this tutorial, you can rename the file lambda-as_an_archive_caching.tf.disabled to lambda.tf.

$ cp lambda-as_an_archive_caching.tf.disabled lambda.tf

Display the contents of the lambda.tf file.

$ cat lambda.tf

Successful output:

resource "aws_lambda_function" "function" {
  function_name = "${var.environment_name}-function"
  description   = "Demo Vault AWS Lambda extension"
  role          = aws_iam_role.lambda.arn
  filename      = "./demo-function/demo-function.zip"
  handler       = "main"
  runtime       = "provided.al2"
  layers        = ["arn:aws:lambda:${var.aws_region}:634166935893:layer:vault-lambda-extension:13"]

  environment {
    variables = {
      VAULT_ADDR           = "http://${aws_instance.vault-server.public_ip}:8200",
      VAULT_AUTH_ROLE      = aws_iam_role.lambda.name,
      VAULT_AUTH_PROVIDER  = "aws",
      VAULT_SECRET_PATH_DB = "database/creds/lambda-function",
      VAULT_SECRET_FILE_DB = "/tmp/vault_secret.json",
      VAULT_DEFAULT_CACHE_ENABLED = true,
      VAULT_DEFAULT_CACHE_TTL = "5m",
      DATABASE_URL         = aws_db_instance.main.address
    }
  }
}

output "lambda" {
  value = <<EOF

The lambda function is ready to run.

    aws lambda invoke --function-name ${aws_lambda_function.function.function_name} /dev/null \
        --log-type Tail \
        --region ${var.aws_region} \
        | jq -r '.LogResult' \
        | base64 --decode

EOF
}

Notice the caching specific entries in this configuration, which set the variables on lines 17 and 18.

Run terraform apply and review the planned actions. Your terminal output indicates the plan that it found and what resources will be created.

$ terraform apply

Enter yes to confirm and resume.

When the terraform apply command completes, the lambda function is modified and the additional output displays how to invoke it.

lambda =
The lambda function is ready to run.

    aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

Invoke the lambda function.

$ aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

Now check the other terminal with audit device log for entries. You will note that there is another pair of request and responses logged.

{"time":"2022-03-21T16:10:12.559749467Z","type":"request","auth":{"token_type":"default"},"request":{"id":"1ddb4b61-ca47-b134-794e-39b0b4f81831","operation":"update","mount_type":"aws","namespace":{"id":"root"},"path":"auth/aws/login","data":{"iam_http_request_method":"hmac-sha256:05864ff6246edd8910fd4f6c4e04b75c7b1dcad3c0333940b87bf5fbf46f3bea","iam_request_body":"hmac-sha256:a6b996161178463d6e4cda2e9176c129c21bd3e8105a8053fba90fdbd1069117","iam_request_headers":"hmac-sha256:5bcfee0729c73dd2f8355e5d1e97cb628fb25304cdc9e228c5b5cfcae8f6b487","iam_request_url":"hmac-sha256:de80ceb22a3568566f33a623d7bcf03cc0933c5439e9ded75013ba07236f6451","role":"hmac-sha256:0796f22fa9fecbc94c1ab6c60c15e88d6eb61cbd552600a4a6373b5268cc8228"},"remote_address":"3.236.58.198","remote_port":33606}}
{"time":"2022-03-21T16:10:12.582299763Z","type":"response","auth":{"client_token":"hmac-sha256:bb2320516bb2fba6bf4b9115a62999183ad158ac54a8a5a13956ed89ec77cf2f","accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300},"request":{"id":"1ddb4b61-ca47-b134-794e-39b0b4f81831","operation":"update","mount_type":"aws","namespace":{"id":"root"},"path":"auth/aws/login","data":{"iam_http_request_method":"hmac-sha256:05864ff6246edd8910fd4f6c4e04b75c7b1dcad3c0333940b87bf5fbf46f3bea","iam_request_body":"hmac-sha256:a6b996161178463d6e4cda2e9176c129c21bd3e8105a8053fba90fdbd1069117","iam_request_headers":"hmac-sha256:5bcfee0729c73dd2f8355e5d1e97cb628fb25304cdc9e228c5b5cfcae8f6b487","iam_request_url":"hmac-sha256:de80ceb22a3568566f33a623d7bcf03cc0933c5439e9ded75013ba07236f6451","role":"hmac-sha256:0796f22fa9fecbc94c1ab6c60c15e88d6eb61cbd552600a4a6373b5268cc8228"},"remote_address":"3.236.58.198","remote_port":33606},"response":{"auth":{"client_token":"hmac-sha256:bb2320516bb2fba6bf4b9115a62999183ad158ac54a8a5a13956ed89ec77cf2f","accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300},"mount_type":"aws"}}
{"time":"2022-03-21T16:10:12.585414511Z","type":"request","auth":{"client_token":"hmac-sha256:bb2320516bb2fba6bf4b9115a62999183ad158ac54a8a5a13956ed89ec77cf2f","accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300,"token_issue_time":"2022-03-21T16:10:12Z"},"request":{"id":"bc1a6869-c41f-f128-04d8-c44634521ff3","client_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","operation":"read","mount_type":"database","client_token":"hmac-sha256:9c46ca0cf4cfba2401142d2567acc446e08783439d8e5a8a50455a887cea0b45","client_token_accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","namespace":{"id":"root"},"path":"database/creds/lambda-function","remote_address":"3.236.58.198","remote_port":33606}}
{"time":"2022-03-21T16:10:12.593161077Z","type":"response","auth":{"client_token":"hmac-sha256:bb2320516bb2fba6bf4b9115a62999183ad158ac54a8a5a13956ed89ec77cf2f","accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","display_name":"aws-vault-lambda-extension-brian-lambda-role","policies":["default","lambda-function"],"token_policies":["default","lambda-function"],"metadata":{"account_id":"561656980159","auth_type":"iam","role_id":"b395511b-5c8c-943e-2bcc-0c4add9755d4"},"entity_id":"3895cdbc-6a17-631b-4224-a8d7cccc5225","token_type":"service","token_ttl":300,"token_issue_time":"2022-03-21T16:10:12Z"},"request":{"id":"bc1a6869-c41f-f128-04d8-c44634521ff3","operation":"read","mount_type":"database","client_token":"hmac-sha256:9c46ca0cf4cfba2401142d2567acc446e08783439d8e5a8a50455a887cea0b45","client_token_accessor":"hmac-sha256:7221d375c4c0a76ff60d076d89f97706e3a5ec8ecc34fc9a273c13d1753a89ad","namespace":{"id":"root"},"path":"database/creds/lambda-function","remote_address":"3.236.58.198","remote_port":33606},"response":{"mount_type":"database","secret":{"lease_id":"database/creds/lambda-function/n8hQDKq1y6H7XxksNMkkiCgw"},"data":{"password":"hmac-sha256:51d179715a37095c95b10373e02ced4ba6338414d5fb19d16a5455ac0d8f44e6","username":"hmac-sha256:f20d229069a6e82b6f3f087fe2e3ed584cf59f96150e2f15de2286cd0a1f7d1a"}}}

Invoke the lambda function once again.

$ aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

Invoke the lambda function a final time.

$ aws lambda invoke --function-name vault-lambda-extension-demo-function /dev/null \
        --log-type Tail \
        --region us-east-1 \
        | jq -r '.LogResult' \
        | base64 --decode

If you check the other terminal with audit device log now, you'll note that there were no entries for the last two invocations of the Lambda.

This is because they successfully used the cache, and did not need to authenticate with Vault.

You can learn more about the caching functionality in the Vault Lambda Extension documentation.

Clean up

In the second terminal displaying the audit log, press CTRL+C to stop tailing the log and exit from the ssh session.

$ exit

Return to the first terminal where you created the cluster and use Terraform to destroy the cluster.

Destroy the AWS resources provisioned by Terraform.

$ terraform destroy -auto-approve

Delete the state file.

$ rm *tfstate*

Next Steps

You built, packaged and deployed a Lambda function written in Go that uses the Vault Lambda Extension, and deployed it with Terraform.

Learn more by reading the blog post Use AWS Lambda Extensions to Securely Retrieve Secrets From HashiCorp Vault or by reviewing the Vault Lambda Extension code repository.

You deployed a Vault server and database with the provided Terraform configuration. Learn more about deploying infrastructure with Terraform Getting Started - AWS.

You used the AWS lambda to authenticate with Vault, request credentials for the PostgreSQL database, and perform a Query. Learn more about Dynamic Secrets: Database Secrets Engine.

You also learned about the caching functionality available in the Vault Lambda Extension, and demonstrated it in action in the audit device logs.

Configure Vault as CM in Kubernetes with Helm

Key management with Azure Key Vault