项目作者: sahanasj

项目描述 :
Cloud Custodian for AWS Cloud Governance (S3 Governance)
高级语言:
项目地址: git://github.com/sahanasj/cloudcustodian-policies.git
创建时间: 2018-06-26T07:55:03Z
项目社区:https://github.com/sahanasj/cloudcustodian-policies

开源协议:

下载


Sahana’s Cloud Custodian Policies

Policies in Production

Policy Description
mailer.yml
Sends email notification via Simple Email Service (SES) using notify action
s3-bucket-versioning.yml
Rectifies and enables all suspended versioning on S3 buckets, then sends notifications.
s3-bucket-public-access.yml
Rectifies and corrects the Global grants and secured S3 buckets as private, then sends notifications.
s3-toggle-logging.yml
Configure New Buckets Settings and Standards such as enabling the default S3 AES256 bucket encryption, turns on object versioning, enables logging on the bucket, and tags the user that created the bucket.

Cloud Custodian Architecture and AWS Services

Cloud Custodian (a.k.a C7N) notifies users in real-time AWS resources behavior changes, Compliance (Security/Access Control, Encryption, Backups, etc) and drives Cost savings (Off-hours, Monitoring and Garbage Collection of unused and underutilized resources).

Getting Started


Quick Install

*** Install dependencies (with virtualenv) *** $ sudo apt-get -y install virtualenv or sudo yum install virtualenv $ virtualenv custodian_env $ source custodian_env/bin/activate *** Install AWS CLI and C7N *** $ pip install awscli c7n ** Configure AWSCLI ** $ aws configure (Configure with AWS Credentials and Region) *** Verify AWSCLI Installation with any CLI command *** $ aws ec2 describe-regions *** To Install Cloud Custodian Mailer *** *** Install repository*** $ git clone https://github.com/capitalone/cloud-custodian $ cd cloud-custodian/tools/c7n_mailer $ pip install -r requirements.txt $ python setup.py develop *** Verify Installation *** $ c7n-mailer $ custodian
For more info, check out Cloud Custodian in GitHub

Usage


Getting Started


  1. Cloud Custodian must be run within a virtual environment.

    $ cd ~
    $ source custodian_env/bin/activate
    $ cd cloudcustodian_scripts (this is the folder where all the custodian policies reside)

    Execute/run the Cloud Custodian Policies

    # Validate the configuration
    $ custodian validate s3-bucket-public-access.yml

    # Dryrun the policies
    $ custodian run dryrun -s check-public-access s3-bucket-public-access.yml
    (Note: Make sure If you get a match (e.g. count > 0), then run the below command)

    # Run the policy
    $ custodian run -s check-public-access s3-bucket-public-access.yml

    Invoking c7n Mailer
    # Validate the configuration
    $ custodian validate s3-bucket-public-access.yml

    # Dryrun the policies
    $ custodian run dryrun -s check-public-access s3-bucket-public-access.yml
    (Note: Make sure If you get a match (e.g. count > 0), then run the below command)

    # Run the policy to invoke custodian mailer
    $ c7n-mailer config mailer.yml update-lambda && custodian run -c s3-bucket-public-access.yml -s .

    When we run this policy, Check the AWS console for a new Lambda named cloud-custodian-mailer.
    The mailer runs every five minutes, so wait a bit and then look for an email in your inbox. (Orelse manually, edit CWE scheduled time less than 5 mins for the quick response)



    Cloud Custodian will create a log files in the ~/cloudcustodian_scripts/check-public-access/ subdirectory IF there are any matches.

C7N Mailer Workflow - AWS SES Sends a mail on violation occurs in S3 Bucket


Workflow

Steps for Cloud Custodian mailer to ensure S3 Governance and Compliance.


Step 1: Create Mailer file



Step 2: Create Custodian Policy for S3 Public read/write Access - Sends email notification via Simple Email Service (SES) using notify action

$ vim s3-bucket-public-access-check.yml .



Step 3: Run a Command that installs the mailer and run a policy that triggers an email to your inbox.

$ c7n-mailer —config mailer.yml —update-lambda && custodian run -c s3-bucket-public-access-check.yml -s .



Step 4: Check the AWS console for a new Lambda and CWE named “cloud-custodian-mailer” and “custodian-s3-public-access”.

Lambda Functions:



CloudWatch Events:



CWE S3 Bucket Logs:



CWE Custodian mailer Logs:



Step 5: Cloud Custodian mailer deployed lambda and sends a customized mail via SES service.



Environment Settings


mailer.yml


  1. #Which queue should we listen to for messages
    queue_url: https://sqs.us-east-1.amazonaws.com/930337447539/c7n_mailer_for_s3_events

    #Standard Lambda Function Config
    region: us-east-1
    role: arn:aws:iam::930337447539:role/lambda-s3-governance

    #Default from address
    from_address: sjayaramu@eplus.com


Cloud Custodian Lambda AWS Role


  1. Note: Based on your use case, additional permissions may be needed.
    Cloud Custodian will generate a msg if that is the case after invocation.
    AWS IAM Role & policies plays an important role to allows Lambda functions to call AWS services. (Make a note of IAM ARN ex: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln)

    Trust relationship:
    Service”: lambda.amazonaws.com




    Reference:
    | AWSS3CustomPolicyForLincoln.json
    | A policy defines the AWS permissions that you can assign to a user, group, or role. |

Schemas Used


s3


  1. (custodian_env) [root@localhost custodian_scripts]# custodian schema s3
    aws.s3:
    actions: [attach-encrypt, auto-tag-user, configure-lifecycle, delete, delete-bucket-notification,
    delete-global-grants, encrypt-keys, encryption-policy, invoke-lambda, mark-for-op,
    no-op, notify, put-metric, remove-statements, remove-website-hosting, set-bucket-encryption,
    set-inventory, set-statements, tag, toggle-logging, toggle-versioning, unmark]
    filters: [and, bucket-encryption, bucket-notification, cross-account, data-events,
    event, global-grants, has-statement, inventory, is-log-target, marked-for-op,
    metrics, missing-policy-statement, missing-statement, no-encryption-statement,
    not, or, value]

    [ OR ]

    For S3 Schema Filters

    (custodian_env) [root@localhost custodian_scripts]# custodian schema s3.filters
    aws.s3:
    filters: [and, bucket-encryption, bucket-notification, cross-account, data-events,
    event, global-grants, has-statement, inventory, is-log-target, marked-for-op,
    metrics, missing-policy-statement, missing-statement, no-encryption-statement,
    not, or, value]

    For S3 Schema actions

    (custodian_env) [root@localhost lfg-custodian]# custodian schema s3.actions
    aws.s3:
    actions: [attach-encrypt, auto-tag-user, configure-lifecycle, delete, delete-bucket-notification,
    delete-global-grants, encrypt-keys, encryption-policy, invoke-lambda, mark-for-op,
    no-op, notify, put-metric, remove-statements, remove-website-hosting, set-bucket-encryption,
    set-inventory, set-statements, tag, toggle-logging, toggle-versioning, unmark]

    To undesrtand a particular filter & action:

    (custodian_env) [root@localhost custodian_scripts]# custodian schema s3.filters.global-grants
    Help
    ——

    Filters for all S3 buckets that have global-grants

    :example:
    .. code-block:: yaml

    policies:
    - name: s3-delete-global-grants
    resource: s3
    filters:
    - type: global-grants
    actions:
    - delete-global-grants
    Schema
    ———

    {
    additionalProperties”: false,
    required”: [
    type
    ],
    type”: object”,
    properties”: {
    allow_website”: {
    type”: boolean
    },
    operator”: {
    enum”: [
    or”,
    and
    ],
    type”: string
    },
    type”: {
    enum”: [
    global-grants
    ]
    },
    permissions”: {
    items”: {
    enum”: [
    READ”,
    WRITE”,
    WRITE_ACP”,
    READ”,
    READ_ACP
    ],
    type”: string
    },
    type”: array
    }
    }
    }


Troubleshooting Tips




  1. Use custodian validate to find syntax errors
    Check name of policy doesnt contain spaces
    Check SQS to see if Custodian payload is entering the queue
    Check cloud-custodian-mailer lambda CloudWatch rule schedule (5 minute by default)
    Check Lambda error logs (this requires CloudWatch logging)
    Check role for lambda(s) have adequate permissions
    Remember to update the cloud-custodian-mailer lambda when making changes to a policy that uses notifications
    Clear the cache if you encounter errors due to stale information (rm ~/.cache/cloud-custodian.cache)


Lambda Code Cheatsheet

  1. mode:
  2. type: cloudtrail
  3. role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  4. events:
  5. - CreateBucket
  1. mode:
  2. type: periodic
  3. role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  4. schedule: "rate(15 minutes)"
  1. mode:
  2. type: periodic
  3. role: arn:aws:iam::930337447539:role/S3-GovernanceForLincoln
  4. schedule: 'cron(0/2 * * * ? *)'


Sending Notifications via SES

actions: - type: notify template: default.html template_format: 'html' priority_header: '5' subject: "ALERT! - S3 : Invalid Global ACL on Bucket [AWS Account: {{ account }} - Region: {{ region }}]" comments: "Violation of S3 policy" violation_desc: <Message_Of_Mail_Body> action_desc: "Actions Taken: Corrects the ACLs/Policy and Notify User" to: - <your-email-address-goes-here> owner_absent_contact: - <your-emails-address-goes-here> transport: type: sqs queue: https://sqs.us-east-1.amazonaws.com/930337447539/c7n_mailer_for_s3_events

Reference:
Schedule Expressions for Rules

Useful Tool: Quick simple editor for cron schedule expressions.

Note

Config: May run in a different region but not cross-account

Event: Only run in the same region and account

Periodic: May run in a different region and different account

Cloud Custodian Important Resources

Cloud Custodian - All Resources

Cloud Custodian - Getting Started

Cloud Custodian - Github

Cloud Custodian - Docs

Cloud Custodian - 400+ actions and 300+ filters to build policies with

Cloud Custodian - Features

Cloud Custodian - S3 Module

Blog - Using Cloud Custodian for Cloud Governance in AWS

Lambda Support

Lambda

AWS CloudWatch Schedule Rules

S3 Data Events

CloudWatch Rules Expressions

Adding Custom Fields to Reports

Custodian Mailer

C7N_Mailer