Data Science & Engineering

A curated list of Data Science and Engineering frameworks, tools, libraries and related list of tutorials.
This mostly covers python related opensource ones ranging from beginner to intermediate levels.

Table of Contents

• Big Data
  • PySpark - Apache Spark Python API
• Frameworks
  • Apache Airflow
• Libraries
  • Pandas
  • NumPy
  • Alembic
• Tools
  • JupyterLab
  • Google Colab

Big Data

PySpark - Apache Spark Python API - pypi

Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming.

• Homepage
• Installation
• Documentation
• Getting Started - overview os sparkcontext, sqlcontext, spark ml, basic operations, data processing.
• Introduction to Dataframes in PySpark - creating dataframes, commonly used functions on dataframes.
• Introduction to Dataframes in PySpark - Part 2 - hands on dataframe using opensource datasets.
• PySpark SQL - basics of pyspark sql.
• Spark SQL, DataFrames and Datasets Guide - sql, dataframe operations, dataset, aggregation, global temp view, interoperating with RDDs …
• PySpark SQL Cheatsheet
• UDF - User-Defined Functions - register UDFs, invoke UDFs.
• Using Parquet files - loading data, paritions, schema merging, metadata refreshing.
• Structured streaming - batch/interactive processing, stream processing.
• Example scripts - set of example scriupts from Apache spark github repo.
• PySpark Streaming with Kafka/ - using pyspark StreamingContext connect to kafka and process the data from kafka stream.

Study Material

• Learning Apache Spark with Python

Frameworks

Apache Airflow - pypi

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

• Homepage
• Installation
• Getting started - simple airflow setup and running dags.
• @dustinstansbury/understanding-apache-airflows-key-concepts-a96efed52b1a">Key concepts - DAGs, operators, sensors, tasks
• Executors - explanation on different type of executors
• Triggering DAGs - triggering DAG via UI, REST api, Command Line
• Custom Operator and Sensor - write your own operator and sensor.
• Multiple workers with Celery executor - Using celery executor run mutiple airflow workers

  Advanced Concepts:

• Kubernetes operator - explanation on how to use k8s operator
• Using kubernetes executor - run airflow with kubernetes executor in mini-kube using helm.
• Tips, Tricks and Best practises
• Deep Dive into airflow on kubernetes executor - Running airflow reliabley with Kubernetes.

Libraries

Pandas - pypi

Library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

• Homepage
• Documentation
• Basics - Reading files into data frames and selecting.
• Aggregation and grouping - aggregation (such as min, max, sum, count, etc.) and grouping.
• Data wrangling - merge, sort, reset_index, fillna.
• Pivot table - explains the pandas pivot_table function and how to use it for data analysis.
• Pandas plotting - Pandas and Matplotlib to produce the graphs.
• Time series analysis - about powerful time series tools in the pandas library.
• Pandas Cheatsheet

NumPy - pypi

NumPy is the fundamental package for scientific computing with Python.
Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.

• Homepage
• Documentation
• Basics - Data types, Array creation, I/O with NumPy, Indexing, Broadcasting, Byte-swapping, Structured arrays, Subclassing ndarray
• Getting started - a little dive into the above mentioned basic concepts.
• Arrays and Vectorized Computation - ndarrays, array & scalar operations, transposing arrays and swapping axes
• Mathematical function - trigonometric, hyperbolic, exponents and logarithms, complex numeric functions
• UFuncs - Universal Functions - Introduction, expolaration, trigonometric functions, advanced ufuncs, special ufuncs
• Array manipulation - changing shapes, transpose-like operations, changing number of dimensions, changing kind of array, joining-splitting-tilling of arrays, adding-removing & rearraning elements
• Fancy Indexing - fancy indexing, combined indexing
• Numpy Cheatsheet

Alembic - pypi

Alembic is a database migrations tool written by the author of SQLAlchemy. A migrations tool offers the following functionality:

1. Can emit ALTER statements to a database in order to change the structure of tables and other constructs
2. Provides a system whereby “migration scripts” may be constructed; each script indicates a particular series of steps that can “upgrade” a target database to a new version, and optionally a series of steps that can “downgrade” similarly, doing the same steps in reverse.
3. Allows the scripts to execute in some sequential manner.

• Homepage
• Documentation
• Getting started - Setting up alembic connection, running migrations, downgrading
• Auto-generate migrations - Alembic can view the status of the database and compare against the table metadata in the application, generating the “obvious” migrations based on a comparison.
• Batch migratoins - Runnign batch migrations
• Working with branches - merging two divergent source trees
• Alembic and Postgresql/ - Running simple migrations on postgres using alembic.

Tools

JupyterLab

JupyterLab is the next-generation web-based user interface for Project Jupyter.

JupyterLab enables you to work with documents and activities such as Jupyter notebooks, text editors, terminals, and custom components in a flexible, integrated, and extensible manner. You canarrange multiple documents and activities side by side in the work area using tabs and splitters. Documents and activities integrate with each other, enabling new workflows for interactive computing.

• Installation.html
• Documentation
• GitHub Repo
• Using Jupyter Notebook - explains how to install, run, and use Jupyter Notebooks for data science, including tips, best practices, and examples.
• Jupyter notebook tips, tricks & shortcuts - keyboard shortcuts, IPython magic commands.
• JupyterLab evolution of the jupyter notebook - An overview of JupyterLab, the next generation of the Jupyter Notebook.
• Using notebook inside JupyterLab - Using calssic jupyter notebooks inside jupyter lab.
• Running PySpark in Jupyter notebook - PySpark installation, connection pyspark to notebook

Google Colab

Colaboratory is a free Jupyter notebook environment that requires no setup and runs entirely in the cloud.

With Colaboratory you can write and execute code, save and share your analyses, and access powerful computing resources, all for free from your browser.

• Basic Features
• Importing Libraries
• More resources
• GPU Notebooks
• Getting started - a small dive into basic features
• Running PySpark in colab - How to run Apache spark python api in google colab.