Hadoop>> cloudcomparer>> 返回
项目作者： vishnuys

项目描述 ：
Compare performance of Apache MapReduce and Spark through SQL queries on Large Dataset
高级语言： Python
项目地址: git://github.com/vishnuys/cloudcomparer.git
创建时间： 2019-09-11T09:12:06Z
项目社区：https://github.com/vishnuys/cloudcomparer
开源协议：BSD 3-Clause "New" or "Revised" License
下载


Cloud Project: Compare performance of Hadoop vs Spark

Rest API to run SQL query on large set of data using Hadoop and Spark.

Requirements:

  • Python 3
  • Hadoop
  • Hadoop Streamer Path
  • HDFS CSV Files Path

Installation:

1) Create a virtualenv Cloud and activate it. If you do not have virtualenv installed, install it. (Installation: Windows, @garimajdamani/https-medium-com-garimajdamani-installing-virtualenv-on-ubuntu-16-04-108c366e4430">Linux & MAC OS)
2) In the project home folder, Install the necessary packages using the command pip install -r requirements .
3) Create config.py in cloudproject folder.
4) Add the following code to config.py

  1. ```python
  2. HADOOP_STREAMER_PATH = "<Path of the Hadoop Streamer JAR file>"
  3. HDFS_CSV_PATH = "<Path of the CSV Files in HDFS>"
  4. ```

5) Execute the following command to apply migrations: python manage.py migrate.
6) To run the server in localhost with default port, execute the command python manage.py runserver.

Usage:

This REST API doesn’t have any authentication. To access this API, send a POST request using any request managers like Postman with data having key query with value as the SQL query.

The output of the query contains two keys: mapreduce and spark. Each key contains the time it took to execute the query on the specified data and the result of the query.

You can learn more about APIs and how to use them here.