Project - Big Data Technology

Built with: Spark Streaming, Kafka, Spark SQL, Javascript

Start Hadoop DFS

/usr/local/hadoop/sbin/start-dfs.sh

Start zookeeper

/usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties

Start Kafka server

/usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties

Project 1 - Twitter Trending hashtag /Realtime/

Run Spark stream

/usr/local/spark/bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.1 spark/sparkStream.py 5

Run Kafka stream

python kafka/twitter.py

Start Node server

node node/server.js

Show the graph

Go to http://localhost:3001

Project 2 - Basketball players dataset querying

Run Spark SQL

cd sql
/usr/local/spark/bin/spark-submit reader.py