Loads raw FEC filings into a database
A set of flexible command line utilities designed to discover, convert and load raw FEC filings into a database in a fast, streaming manner.
fec is about nine times faster than similar solutions. For example, on a recent MacBook Air, a 2.3 gigabyte ActBlue filing parses in three minutes instead of 23 minutes.
It requires Node. It uses fec-parse.
To try converting a filing to newline-separated JSON without installing fec-loader, paste the following into a terminal:
FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | npx -p github:PublicI/fec-loader#subcommands convert $FILING_ID > $FILING_ID".ndjson"
To install:
npm install -g github:PublicI/fec-loader#subcommands
To set up a Postgres database for FEC filings and the environment variables needed to connect:
export PGHOST=<database host> PGDATABASE=<database name> PGUSER=<database user> PGPASSWORD=<database password>fec init
To load a filing from the FEC into a Postgres database, run:
export PGHOST=<database host> PGDATABASE=<database name> PGUSER=<database user> PGPASSWORD=<database password>FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | fec convert $FILING_ID --format=psql | psql
To list the filings available from the FEC’s RSS feed run:
fec list --rss
To load the most recent five filings from the FEC’s RSS feed, run:
for url in $(fec list --rss --headers=false --columns=fec_url --format=tsv | head -n 5); do FILING_ID=$(echo $url | tr -dc '0-9'); curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | fec convert $FILING_ID --format=psql | psql -v ON_ERROR_STOP=on --single-transaction; done
To get just a summary line as JSON:
FILING_ID=1283013; curl -s "https://docquery.fec.gov/dcdev/posted/"$FILING_ID".fec" | head -n 10 | fec convert | sed -n 2p