AWS Analytics Reference Architecture

The AWS Analytics Reference Architecture is a set of analytics solutions put together as end-to-end examples. It regroups AWS best practices for designing, implementing, and operating analytics platforms through different purpose-built patterns, handling common requirements, and solving customers' challenges.

This project is composed of:

• Reusable core components exposed in an AWS CDK (Cloud Development Kit) library currently available in Typescript and Python. This library contains AWS CDK constructs that can be used to quickly provision analytics solutions in demos, prototypes, proof of concepts and end-to-end reference architectures.
• Reference architectures consumming the reusable components to demonstrate end-to-end examples in a business context. Currently, the AWS native reference architecture is available.

This documentation explains how to get started with the core components of the AWS Analytics Reference Architecture.

Getting started

• AWS Analytics Reference Architecture
  • Getting started
    • Prerequisites
    • Initialization (in Python)
    • Development
    • Deployment
    • Cleanup
  • API Reference
  • Contributing
• License Summary

Prerequisites

1. Create an AWS account
2. The core components can be deployed in any AWS region
3. Install the following components with the specified version on the machine from which the deployment will be executed:
   1. Python [3.8-3.9.2] or Typescript
   2. AWS CDK v2: Please refer to the Getting started guide.
4. Bootstrap AWS CDK in your region (here eu-west-1). It will provision resources required to deploy AWS CDK applications

1export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
2export AWS_REGION=eu-west-1
3cdk bootstrap aws://$ACCOUNT_ID/$AWS_REGION

Initialization (in Python)

1. Initialize a new AWS CDK application in Python and use a virtual environment to install dependencies

1mkdir my_demo
2cd my_demo
3cdk init app --language python
4python3 -m venv .env
5source .env/bin/activate

2. Add the AWS Analytics Reference Architecture library in the dependencies of your project. Update requirements.txt

1aws-cdk-lib==2.51.0
2constructs>=10.0.0,<11.0.0
3aws_analytics_reference_architecture>=2.0.0

3. Install The Packages via pip

1python -m pip install -r requirements.txt

Development

1. Import the AWS Analytics Reference Architecture in your code in my_demo/my_demo_stack.py

1import aws_analytics_reference_architecture as ara

2. Now you can use all the constructs available from the core components library to quickly provision resources in your AWS CDK stack. For example:

• The DataLakeStorage to provision a full set of pre-configured Amazon S3 Bucket for a data lake

1        # Create a new DataLakeStorage with Raw, Clean and Transform buckets configured with data lake best practices
2        storage = ara.DataLakeStorage (self,"storage")     

• The DataLakeCatalog to provision a full set of AWS Glue databases for registring tables in your data lake

1        # Create a new DataLakeCatalog with Raw, Clean and Transform databases
2        catalog = ara.DataLakeCatalog (self,"catalog")     

• The DataGenerator to generate live data in the data lake from a pre-configured retail dataset

1        # Generate the Sales Data
2        sales_data = ara.BatchReplayer(
3            scope=self,
4            id="sale-data",
5            dataset=ara.PreparedDataset.RETAIL_1_GB_STORE_SALE,
6            sink_object_key="sale",
7            sink_bucket=storage.raw_bucket,
8         )
9

1        # Generate the Customer Data
2        customer_data = ara.BatchReplayer(
3            scope=self,
4            id="customer-data",
5            dataset=ara.PreparedDataset.RETAIL_1_GB_CUSTOMER,
6            sink_object_key="customer",
7            sink_bucket=storage.raw_bucket,
8         )
9

• Additionally, the library provides some helpers to quickly run demos:

1        # Configure defaults for Athena console
2        athena_defaults = ara.AthenaDemoSetup(scope=self, id="demo_setup")

1        # Configure a default role for AWS Glue jobs
2        ara.GlueDemoRole.get_or_create(self)

Deployment

Deploy the AWS CDK application

1cdk deploy

The time to deploy the application is depending on the constructs you are using

Cleanup

Delete the AWS CDK application

1cdk destroy

API Reference

More contructs, helpers and datasets are available in the AWS Analytics Reference Architecture. See the full API specification here

Contributing

Please refer to the contributing guidelines and contributing FAQ for details.

License Summary

The documentation is made available under the Creative Commons Attribution-ShareAlike 4.0 International License. See the LICENSE file.

The sample code within this documentation is made available under the MIT-0 license. See the LICENSE-SAMPLECODE file.