AWS: Setting up AWS S3 Policies

There are three use cases covered in this document:

1. Using S3 as Destination.
2. Using S3 as Datasource.
3. Using S3 as Storage (only relevant for Administrators).

Please consider the following AWS Policies as minimum requirement for Datatap in order to ensure proper functionality for the respective use cases. To simplify this step feel free to use the AWS Policy Generator as well.

For use cases 1 and 2 it is possible to use paths in reference to the usual format arn:aws:s3:::BUCKET-NAME/CUSTOM_PATH. To learn more about the arn syntax visit Amazon Resource Names (ARN).

When using deny policies, please keep in mind that an explicit deny in any policy overrides any allows, as described in detail in AWS Policy Evaluation Logic.

1. CONTENTS OF AWS POLICY FILE WHEN USING S3 AS A DESTINATION

All these policies are required but can be limited to certain buckets or paths (folders). If limited these parameters have to be set explicitly within Datatap when setting up the Destination.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::/BUCKETNAME/PATHNAME/*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773756000"
        },
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKETNAME"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773807000",
            "Condition": {
              "StringLike": {
                "s3:prefix": "PATHNAME*"
               }
            }
        }
    ]
}

2. CONTENTS OF AWS POLICY FILE WHEN USING S3 AS A DATASOURCE

We recommend to use the same policy file template as for case 1. Nevertheless basic functionality is available with the following configuration.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::/BUCKETNAME/PATHNAME/*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773756000"
        },
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKETNAME"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773807000",
            "Condition": {
              "StringLike": {
                "s3:prefix": "PATHNAME*"
              }
            } 
        }
    ]
}

3. CONTENTS OF AWS POLICY FILE, USING S3 AS A STORAGE

Currently no paths are supported for Datatap Storages. Therefore please make sure to specify access on bucket level.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::/BUCKETNAME/*"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773756000"
        },
        {
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKETNAME"
            ],
            "Effect": "Allow",
            "Sid": "Stmt1478773807000"
        }
    ]
}