Processing the Blockchain with Hadoop and Spark

Check out the full technical report here.
This project involves:

1. Downloading the blockchain as of April 11, 2017.

• A couple sample, compressed blk*.dat files are provided in data/blockchain_sample.

2. Transforming the blockchain into a reduced, text format using Hadoop MapReduce.

• Code for this part of the project can be found in the reduce_blockchain directory.
• Sample output from this step is in data/blockchain_sample_reduced.

3. Using Apache Spark on the text data to uncover statistics/trends on the usage of Bitcoin.

• This code is located in analyze_blockchain/src/main/scala.
• Final results are in the data/output directory.