Xedge>> foxsec-pipeline>> 返回
项目作者： mozilla-services

项目描述 ：
Log analysis pipeline utilizing Apache Beam
高级语言： Java
项目地址: git://github.com/mozilla-services/foxsec-pipeline.git
创建时间： 2018-10-02T19:23:37Z
项目社区：https://github.com/mozilla-services/foxsec-pipeline
开源协议：Mozilla Public License 2.0
下载


foxsec-pipeline

Build Status
Documentation

Apache Beam pipelines for analyzing log data.

Documentation

javadoc documentation is currently updated manually and although should be up to date, may not be current
with master.

Introduction to Beam

To get familiar with developing pipelines in Beam, this repository also contains a small workshop that
provides some guidance on building basic pipelines. The introduction document can be found
here.

Tests

Tests can be executed locally using Docker.

Run all tests

  1. docker build -f Dockerfile-base -t foxsec-pipeline-base:latest .
  2. bin/m test

Run a specific test

  1. docker build -f Dockerfile-base -t foxsec-pipeline-base:latest .
  2. bin/m test -Dtest=ParserTest

CLI Usage

Pipeline RuntimeSecrets

Pipeline runtime secrets can be generated locally using the main method in the RuntimeSecrets class.

  1. bin/m compile exec:java -Dexec.mainClass=com.mozilla.secops.crypto.RuntimeSecrets -Dexec.args='-i testkey -k dataflow -p my-gcp-dataflow-project -r dataflow'

Run the class with no options to see usage information. Note that in this case, the key ring name and key name
are being specified as dataflow. The existing RuntimeSecrets class requires the keys to be accessible
using these identifiers when the pipeline is executing.

The output of the command can be prefixed with cloudkms:// in an option to enable runtime decryption of the secrets
during pipeline execution.

Interacting with Minfraud

Reputation data can be fetched from Minfraud locally using the main method in the Minfraud class.

You must provide the accountid and licensekey plus the IP and/or email you want to get reputation data for. --accountid and --licensekey can either be provided directly or provided as RuntimeSecrets (cloudkms://...).

  1. bin/m exec:java \
  2.   -Dexec.mainClass="com.mozilla.secops.Minfraud" \
  3.   -Dexec.args="-p my-gcp-dataflow-project --accountid 'cloudkms://...' --licensekey 'cloudkms://...' --ip '8.8.8.8' --email 'example@example.com'"

Creating Watchlist entries

Watchlist entries can be created locally using the main method in the Watchlist class.

You must also prefix your command with WITHOUT_DAEMONS=true so that the entry won’t be submitted to the Datastore emulator running within the container.

  1. usage: Watchlist
  2.  -c,--createdby <arg>
  3.  -ne,--neverexpires     Watchlist entry never expires (compared to default
  4.                         of 2 weeks)
  5.  -o,--object <arg>      Object to watch. Can be an IP or email.
  6.  -p,--project <arg>     GCP project name (required if submitting to
  7.                         Datastore)
  8.  -s,--severity <arg>    Severity of Watchlist entry. Can be 'info',
  9.                         'warn', or 'crit'
  10.  -su,--submit           Submit Watchlist entry to Datastore rather than
  11.                         emit json
  12.  -t,--type <arg>        Type of object to watch. Can be 'ip' or 'email'

Example of creating entry without submitting to Datastore

  1. $ bin/m exec:java -Dexec.mainClass="com.mozilla.secops.Watchlist" -Dexec.args="--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info'"
  3. {"type":"ip","severity":"info","expires_at":"2020-02-26T17:45:01.399Z","created_by":"example@example.com","object":"127.0.0.1"}

Example of submitting to Datastore

  1. $ WITHOUT_DAEMONS=true bin/m exec:java -Dexec.mainClass="com.mozilla.secops.Watchlist" -Dexec.args="--object '127.0.0.1' --type 'ip' --createdby 'example@example.com' --severity 'info' --project foxsec-pipeline-nonprod --submit"
  3. Feb 12, 2020 5:41:44 PM com.mozilla.secops.state.State initialize
  4. INFO: Initializing new state interface using com.mozilla.secops.state.DatastoreStateInterface
  5. Feb 12, 2020 5:41:45 PM com.mozilla.secops.state.StateCursor set
  6. INFO: Writing state for 127.0.0.1
  7. Feb 12, 2020 5:41:45 PM com.mozilla.secops.state.State done
  8. INFO: Closing state interface com.mozilla.secops.state.DatastoreStateInterface
  9. Successfully submitted watchlist entry to foxsec-pipeline-nonprod
  10. {"type":"ip","severity":"info","expires_at":"2020-02-26T17:41:43.919Z","created_by":"example@example.com","object":"127.0.0.1"}

Contributing

See the contributing guidelines.