Simple Friend Recommender

Project to demonstrate stuff a data engineer should know

Getting Started

Starting the Project

./start.sh

Note: If you’re using Mac, run ./start-mac.sh instead.

Then we can go to the survey with the following links:

• http://localhost:8000/questions/0
• http://localhost:8000/admin

For Airflow UI, we go to:

• http://localhost:8080

Stopping the Project

./stop.sh

Note: If you’re using Mac, run ./stop-mac.sh instead.

Testing an Airflow Task

airflow tasks test survey_data_processing transform_data_for_recommender 2021-04-04

Setting Up the Airflow’s Connections

In this talk, we’ll set up 2 connections as follows:

1. Postgres connection:

   

2. File system connection:

   

Starting a Simple Dashboard

It shows an answer count.

python3 -m http.server 8088

Then we go to http://localhost:8088/answer_count.html.

Starting a Jupyter Server (Local)

cd airflow/dags/notebooks
jupyter notebook --ip=0.0.0.0

Go to http://localhost:8888 if your Web browser is not automatically fired up.

Note that We’ll need to set up an virtual environment first under the folder airflow/dags/notebooks in case it hasn’t been done.

python3 -m venv ENV
source ENV/bin/activate
pip install -r requirements.txt

To deactivate, run:

deactivate