Example 3-Account Data Pipeline

• Description of the Example

• Prerequisites

  • credential Setup

    be sure to setup profiles for each of the accounts you want to place parts of the pipeline.
    you’ll need admin access in each of them. setup the accounts using ~/.aws/config and ~/.aws/credentials
    on the device you’re planning to run the terraform from

  • Terraform Setup

    install terraform like: https://learn.hashicorp.com/terraform/getting-started/install.html
    download from here: https://www.terraform.io/downloads.html
    Terraform is licences under MPL-2.0, please refer to https://github.com/hashicorp/terraform for further information

  • Init for deployment

    run: terraform init

  • Variables

    look into example.rtfvar and set the variables as needed.

  • Deployment

    run: terraform plan —var-file ./example.tfvar
    if gone through w/o error
    run: terraform apply —var-file ./example.tfvar

  • Testing

    run: the script “input_sim.sh”
    you will need to pass the profile to the profile used for your ingestion
    you can use the “repeat” program for instance to rerun the command several times to fill up the firebase queue. (or just run multible times)

repeat 100 ./input_sim.sh PROFILE_ACCOUNT_A

successful looks like:

}
{
    "RecordId": "XXXXXXXXXXXXXXXXXXXXXXXXXX",
    "Encrypted": false
}

log into the EMR Cluster Master (look into Management Console or via CLI for the DNS name) using your SSH key used for the environment

vi .aws/config
add your role_arns like:

[default]
region = eu-central-1 
[profile default]
role_arn = arn:aws:iam::ACCOUNT_B_ID:role/storage_emr_crossaccount_role
credential_source = Ec2InstanceMetadata
[profile local]
role_arn = arn:aws:iam::ACCOUNT_A_ID:role/emr_crossaccount_role
credential_source = Ec2InstanceMetadata

then you can list and access the S3 Bucket:

aws s3 ls s3://sourcesfromexamplefirehose/direct

or copy an object to local like:

aws s3 cp s3://sourcesfromexamplefirehose/direct/PATH/TO/OBJECT/ .

• Cleanup

  run: terraform destroy —var-file ./example.tfvar
  note: if you get errors from an non-empty bucket … you should know what you do (in case: delete first all objects manualle … or the whole bucket ..) :)

• Troubleshooting / Common Errors

  ‘Error: Missing required argument
  The argument “region” is required, but was not set.’
  -> Can be ignored.

  Error: you get something like “cannot assume-role” or “No creds in EC2InstanceMetadata”
  Run!

Security

See CONTRIBUTING for more information.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

Terraform is licences under MPL-2.0, please refer to https://github.com/hashicorp/terraform for further information