AWS Athena (Pre-release)

Amazon’s serverless, interactive query service that lets you analyze data directly in Amazon S3 using standard SQL.

Supported Connection Methods

Available methods for connections to the target system from Lumi AI:

• Direct

  The default / typical connection option. Interfaces directly with the target system over the Internet with no mediation. Suitable for most cloud-hosted scenarios or public-facing resources. See Network Configuration for more details.

• Gateway An alternative connection method leveraging a Lumi AI Data Gateway. This uses Lumi AI's purpose-built connection agent to mediate communications. Ideal for restricting access to systems within a protected network. For more information, see Data Gateway.

Supported Limits

The following are limits that can be configured for the system to moderate access and usage from users in Lumi AI:

• Cost Limit Before running a query, if the system supports it, the system-specific compute cost (or surrogate) estimate will be processed and compared to an organization-level/admin-set cost limit for systems of this type (if configured/set). If exceeded, the query will not run (and either the workflow will attempt an optimization or the user will be notified).

• Duration Limit An alternative to cost, queries will be stopped the system supports a duration/timeout limit and one is set/configured at the organization level (across systems).

Available Parameters

These properties are the essential source system connection properties that all queries are directed towards.

* Required parameters

Note: The Gateway parameter is common to all systems (when supported) and is only available when gateway is the selected connection method.

AWS Region*

AWS Region where Athena and S3 staging bucket resides in.

👍 Valid Examples

• us-east-1

• us-west-2

👎 Invalid Examples

• US-East-1

• North America

Query Result Location*

The S3 URI (bucket + optional prefix) where Athena writes its query results

👍 Valid Examples

• s3://my-athena-bucket

• s3://my-athena-bucket/staging

👎 Invalid Examples

• /s3/my-athena-bucket

• my-athena-bucket

Database*

The Athena database name.

👍 Valid Examples

• analytics

• default

👎 Invalid Examples

• AwsDataCatalog.analytics

• analytics.products

Role ARN*

The ARN (Amazon Resource Name) of the IAM role that will be assumed by Lumi to run queries against Athena.

👍 Valid Examples

• arn:aws:iam::123456789012:role/lumi-athena-query-runner

👎 Invalid Examples

• 123456789012:role/lumi-athena-query-runner

• lumi-athena-query-runner

System Permissions & Configuration

You will need to create cross-account IAM role whose trust policy allows Lumi's AWS account to run queries against Athena.

Create IAM policy with limited access to your Athena instance

1. Navigate to IAM -> Policies -> Create policy.

2. Select the JSON option and paste the following snippet, replacing your-data-bucket with your-athena-staging-bucket with the name of your S3 Athena data bucket and S3 your staging (query results) bucket, respectively.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAthenaQueries",
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "athena:ListQueryExecutions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "ReadDataBucket",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-data-bucket",
        "arn:aws:s3:::your-data-bucket/*"
      ]
    },
    {
      "Sid": "UseStagingBucket",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::your-athena-staging-bucket/*"
    }
  ]
}

1. Click Next.

2. Name the policy AthenaQueryPermissions and click Create Policy.

Create a Cross-Account IAM Role with the above IAM policy

1. Navigate to IAM → Roles → Create role.

2. Select Custom trust policy option and paste the following snippet, replacing lumi-account-id with Lumi's Account ID and lumi-external-id with Lumi external ID (provided during onboarding).

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::lumi-account-id:role/athena-query-runner"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "lumi-external-id"
        }
      }
    }
  ]
}

1. Click Next.

2. In the Permissions policies, search for AthenaQueryPermissions and select it.

3. Click Next.

4. Name the role athena-query-runner and click Create Role.

Lock Down Your Athena S3 Buckets with Bucket Policies

Data bucket

1. Navigate to your S3 Athena data bucket and click on Permissions

2. Click on Edit button beside Bucket policy and append the following statement to your policy, replacing your-data-bucket with the name of your S3 Athena data bucket and lumi-account-id with Lumi's Account ID

    {
      "Effect":"Allow",
      "Principal": {
        "AWS":"arn:aws:iam::lumi-account-id:role/lumi-athena-query-runner"
      },
      "Action":["s3:ListBucket","s3:GetObject"],
      "Resource":[
        "arn:aws:s3:::your-data-bucket",
        "arn:aws:s3:::your-data-bucket/*"
      ]
    }

1. Click Save changes

Staging (query results) bucket

1. Navigate to your S3 Athena staging (query results) and click on Permissions.

2. Click on Edit button beside Bucket policy and append the following statement to your policy, replacing your-athena-staging-bucket with the name of your S3 Athena staging (query results) bucket and lumi-account-id with Lumi's Account ID.

    {
      "Effect":"Allow",
      "Principal": {
        "AWS":"arn:aws:iam::lumi-account-id:role/lumi-athena-query-runner"
      },
      "Action":["s3:PutObject","s3:GetObject"],
      "Resource":"arn:aws:s3:::your-athena-staging-bucket/*"
    }

1. Click Save changes