AWS Athena (Pre-release)

Amazon’s serverless, interactive query service that lets you analyze data directly in Amazon S3 using standard SQL.

Supported Connection Methods

Available methods for connections to the target system from Lumi AI:

• Direct

  The default / typical connection option. Interfaces directly with the target system over the Internet with no mediation. Suitable for most cloud-hosted scenarios or public-facing resources. See Network Configuration for more details.

• Gateway An alternative connection method leveraging a Lumi AI Data Gateway. This uses Lumi AI's purpose-built connection agent to mediate communications. Ideal for restricting access to systems within a protected network. For more information, see Data Gateway.

Supported Limits

The following are limits that can be configured for the system to moderate access and usage from users in Lumi AI:

• Cost Limit Before running a query, if the system supports it, the system-specific compute cost (or surrogate) estimate will be processed and compared to an organization-level/admin-set cost limit for systems of this type (if configured/set). If exceeded, the query will not run (and either the workflow will attempt an optimization or the user will be notified).

• Duration Limit An alternative to cost, queries will be stopped the system supports a duration/timeout limit and one is set/configured at the organization level (across systems).

Available Parameters

These properties are the essential source system connection properties that all queries are directed towards.

* Required parameters

Note: The Gateway parameter is common to all systems (when supported) and is only available when gateway is the selected connection method.

AWS Role ARN*

The ARN (Amazon Resource Name) of the IAM role that will be assumed by Lumi to run queries against Athena.

👍 Valid Examples

• arn:aws:iam::123456789012:role/lumi-athena-query-runner

👎 Invalid Examples

• 123456789012:role/lumi-athena-query-runner

• lumi-athena-query-runner

AWS Region*

AWS Region where Athena and S3 staging bucket resides in.

👍 Valid Examples

• us-east-1

• us-west-2

👎 Invalid Examples

• US-East-1

• North America

S3 Staging Directory*

The S3 URI (bucket + optional prefix) where Athena writes its query results

👍 Valid Examples

• s3://my-athena-bucket

• s3://my-athena-bucket/staging

👎 Invalid Examples

• /s3/my-athena-bucket

• my-athena-bucket

Database*

The Athena database name.

👍 Valid Examples

• analytics

• default

👎 Invalid Examples

• AwsDataCatalog.analytics

• analytics.products

AWS External ID*

AWS External ID. Can be any arbitrary string that matches aws-external-id of the cross-account IAM role (see below).

👍 Valid Examples

• 6ee2f609-cb2c-4d66-b39b-a3a13985b622

👎 Invalid Examples

• Empty string

System Permissions & Configuration

You will need to create cross-account IAM role whose trust policy allows Lumi's AWS account to run queries against Athena.

Create IAM policy with access to your Athena instance

1. Navigate to IAM -> Policies -> Create policy.

2. Select the JSON option and paste the following snippet, replacing <your-data-bucket> with <your-athena-staging-bucket> with the name of your S3 Athena data bucket and S3 your staging (query results) bucket, respectively. /

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAthenaAccess",
      "Effect": "Allow",
      "Action": [
        "athena:StartQueryExecution",
        "athena:GetQueryExecution",
        "athena:GetQueryResults",
        "athena:ListQueryExecutions",
        "athena:ListDatabases",
        "athena:ListTableMetadata",
        "athena:GetTableMetadata"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowGlueAccess",
      "Effect": "Allow",
      "Action": [
        "glue:GetDatabases",
        "glue:GetDatabase",
        "glue:GetTables",
        "glue:GetTable"
      ],
      "Resource": "*"
   },
    {
      "Sid": "AllowDataBucketReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<your-data-bucket>",
        "arn:aws:s3:::<your-data-bucket>/*"
      ]
    },
    {
      "Sid": "AllowStagingBucketReadWriteAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::<your-athena-staging-bucket>",
        "arn:aws:s3:::<your-athena-staging-bucket>/*"
      ]
    }
  ]
}

1. Click Next.

2. Name the policy AthenaQueryPermissions and click Create Policy.

Create a Cross-Account IAM Role with the above IAM policy

1. Navigate to IAM → Roles → Create role.

2. Select Custom trust policy option and paste the following snippet, replacing <lumi-account-id> with Lumi's Account ID (provided during onboarding) and <aws-external-id> with AWS external ID (can be any arbitrary string)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<lumi-account-id>:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "<aws-external-id>"
        }
      }
    }
  ]
}

1. Click Next.

2. In the Permissions policies, search for AthenaQueryPermissions and select it.

3. Click Next.

4. Name the role LumiAI-athena-query-runner and click Create Role.