assessor-scraper

The goal of this project is to transform the data from the Orleans Parish
Assessor’s Office website into formats that
are better suited for data analysis.

development environment setup

prerequisites

You must have Python 3 installed. You can download it
here.

first setup a python virtual environment

python3 -m venv .venv
. .venv/bin/activate

install the dependencies with pip

pip install -r requirements.txt

Getting started

Set up the database

By default, the scraper is setup to load data into a PostgreSQL database.
Docs on setting up and making changes to the database are here.
You can quickly get the database running locally using Docker.

docker-compose up -d db

If you want to explore how to extract data using scrapy, use the scrapy
shell to interactively
work with the response.

For example,

scrapy shell http://qpublic9.qpublic.net/la_orleans_display.php?KEY=1500-SUGARBOWLDR
owner = response.xpath('//td[@class="owner_value"]/text()').get()
total_value = response.xpath('//td[@class="tax_value"]/text()')[3].get().strip()
next_page = response.xpath('//td[@class="header_link"]/a/@href').get()

Get all the parcel ids

Getting a list of parcel ids allows us to build urls for every property
so we can scrape the data for that parcel. These parcel ids are used
in the url like http://qpublic9.qpublic.net/la_orleans_display.php?KEY=701-POYDRASST,
where 701-POYDRASST is the parcel id.

Running the parcel_id_extractor.py script will cleverly use the owner search to
extract all available parcel ids, then save them in a file parcel_ids.txt.

The file is checked in to the repo, but if you want to run it yourself
to update it with the latest, run

python parcel_id_extractor.py

Running the spider

Running the spider from the command line will crawl the assessors website and
output the data to a destination of your choice.

By default, the spider will output data to a postgres database, which is configured
in scraper/settings.py. You can use a hosted postgres instance or run one locally using
Docker:

Important Note: Scraping should always be done responsibly so check the robots.txt file to ensure the site doesn’t explicitly instruct crawlers to not crawl. Also when running the scraper, be careful not to cause unexpected load to the assessors website - consider running during non-peak hours or profiling the latency to ensure you aren’t overwhelming the servers.

To run the spider,

scrapy runspider scraper/spiders/assessment_spider.py

Warning: this will take a long time to run…you can kill the process with ctrl+c.

To run the spider and output to a csv

scrapy runspider scraper/spiders/assessment_spider.py -o output.csv

Running on Heroku

Set required environment variables:

heroku config:set DATABASE_URL=postgres://user:pass@host:5432/assessordb

You can run the scraper on Heroku by scaling up the worker dyno:

heroku ps:scale worker=1

See the Heroku docs for more info on how to deploy Python code.

Running in aws with Terraform

1) Install terraform
2) cd terraform
3) terraform init
4) terraform plan
5) terraform apply
6) ssh ubuntu@{public_dns}