Terraforming your AWS S3 integration

This guide is intended only as a supplement to the excellent Terraform documentation provided by Hashicorp here.

Please note, it needs to be performed in conjunction with the Encord App Integration setup steps described here

Please do not just copy and paste the below, instead, use it as a template for Terraforming your Private Cloud Integration.

To integrate with Encord, you will need to create:

An S3 Bucket
An IAM Policy
An IAM Role
A CORS Policy

Below are some examples of how this might look:

Declaring your Terraform providers

In the below example, we're using Hashicorp's AWS provider aws and the tfvars utility that allows us to neatly define values to pass into variables.

We also define an alias as well as a Region for the AWS provider, which needs to match the location in which you want your bucket to be provisioned.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.1.0"
    }
    tfvars = {
      source  = "innovationnorway/tfvars"
      version = "0.0.1"
    }
  }
}

provider "aws" {
  alias  = "default"
  region = var.aws_region
}

Declaring your variables

In your variables.tf file, you will need to define the variables into which you want to pass values. An example is below:

variable "bucket_name" {
  description = "Name of the AWS S3 Bucket"
  type        = string
}

variable "policy_name" {
  description = "Name of the IAM Policy"
  type        = string
}

variable "role_name" {
  description = "Name of the IAM Role"
  type        = string
}

variable "external_aws_account_id" {
  description = "Account ID of the external AWS account you're connecting to - default value 312435012576 for Encord"
  type        = string
  default     = "312435012576"
}

variable "external_id" {
  description = "External account id - this is unique to your integration and can be found in the integration setup modal"
  type        = string
}

variable "aws_region" {
  description = "AWS Region in which bucket should be provisioned"
  type        = string
  default     = "eu-west-2"
}

Defining your variables in a `.tfvars` file to avoid having to manually edit the Terraform files

Since we've opted to use tfvars we need to create a corresponding .tfvars file and specify some values we wish to pass into the variables you just defined:

bucket_name             = "encord-test-bucket"
policy_name             = "encord-test-policy"
role_name               = "encord-test-role"
external_aws_account_id = "312435012576"      # This is the same for every integration since it is Encord's AWS account ID
external_id             = "external-id" # This comes from the integration setup modal within the Encord application and is unique for each integration you set up
aws_region              = "eu-west-2"          # Change this to the appropriate region in which your bucket is to be created

Creating the required resources

The resources you need to create include:

The AWS S3 bucket itself
The AWS Bucket CORS Policy to allow Cross Origin Resource Sharing with the Encord domains
The IAM Role
The IAM Policy
The IAM Policy attachment that binds the Role to the Policy

Defining the Bucket, Bucket ACL, and Bucket CORS Policy:

resource "aws_s3_bucket" "bucket_name" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_cors_configuration" "bucket_cors_policy" {
  bucket = var.bucket_name
  cors_rule {
    allowed_headers = [
      "*"
    ]

    allowed_methods = ["GET",
    "PUT"] # The PUT method here is not necessary unless you intend to re-encode videos or work with image groups

    allowed_origins = [
      "https://app.encord.com",
      "https://api.encord.com",
      "https://dicom.encord.com"
    ]
    max_age_seconds = 3600
  }
}

Defining the IAM Policy:

resource "aws_iam_policy" "encord-test-policy" {
  name        = var.policy_name
  path        = "/"
  description = "video testing S3 policy"
  policy      = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": "${aws_s3_bucket.bucket_name.arn}/*"
    }
  ]
}
POLICY
}

resource "aws_iam_role" "encord-test-role" {
  name               = var.role_name
  path               = "/"
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${var.external_aws_account_id}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${var.external_id}"
        }
      }
    }
  ]
}
POLICY
}

Attaching the Role to the Policy:

resource "aws_iam_policy_attachment" "encord-test-policy-policy-attachment" {
  policy_arn = aws_iam_policy.encord-test-policy.arn
  roles      = [var.role_name]
  name       = "${var.policy_name}-policy-attachment"
}

Before applying any changes, run terraform plan to preview the changes and check you are happy with them.

Once your Terraform has been applied, return to the Encord application, and test your integration.

The entire resources file s3-resources.tf now looks like this:

resource "aws_s3_bucket" "bucket_name" {
  bucket = var.bucket_name
}

resource "aws_s3_bucket_cors_configuration" "bucket_cors_policy" {
  bucket = var.bucket_name
  cors_rule {
    allowed_headers = [
      "*"
    ]

    allowed_methods = ["GET",
    "PUT"] # The PUT method here is not necessary unless you intend to re-encode videos or work with image groups

    allowed_origins = [
      "https://app.encord.com",
      "https://api.encord.com",
      "https://dicom.encord.com"
    ]
    max_age_seconds = 3600
  }
}


resource "aws_iam_policy" "encord-test-policy" {
  name        = var.policy_name
  path        = "/"
  description = "video testing S3 policy"
  policy      = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": "${aws_s3_bucket.bucket_name.arn}/*"
    }
  ]
}
POLICY
}


resource "aws_iam_role" "encord-test-role" {
  name               = var.role_name
  path               = "/"
  assume_role_policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::${var.external_aws_account_id}:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "${var.external_id}"
        }
      }
    }
  ]
}
POLICY
}


resource "aws_iam_policy_attachment" "encord-test-policy-policy-attachment" {
  policy_arn = aws_iam_policy.encord-test-policy.arn
  roles      = [var.role_name]
  name       = "${var.policy_name}-policy-attachment"
}

Declaring your Terraform providers​

Declaring your variables​

Defining your variables in a .tfvars file to avoid having to manually edit the Terraform files​

Creating the required resources​

Defining the Bucket, Bucket ACL, and Bucket CORS Policy:​

Defining the IAM Policy:​

Attaching the Role to the Policy:​