直播技术>> ConferenceTweetMapper>> 返回
项目作者： identitymonk

项目描述 ：
Export Tweets from Twitter into JSON file then publish as a Graph objects in Neo4j DB
高级语言： Python
项目地址: git://github.com/identitymonk/ConferenceTweetMapper.git
创建时间： 2018-10-17T04:39:51Z
项目社区：https://github.com/identitymonk/ConferenceTweetMapper
开源协议：
下载


ConferenceTweetMapper

General usage

This repository contains three Python scripts:

  • archive.py

    This script allows to search Twitter for a particular conference hashtag and to return all the resulting tweets in a JSON file

  • graphify.py

    This script takes a JSON file containing Tweets and to transform them to graph oriented object representing another view of the timeline of Tweets, Retweets, Quote, Hashtag, and Handles. In case of Retweet/Quote/Reply, this script will also drills for original tweet even if outside the scope of the file.

  • stream.py

    This script allows to search Twitter for a particular conference hashtag and to transform them to graph oriented object representing another view of the timeline of Tweets, Retweets, Quote, Hashtag, and Handles. In case of Retweet/Quote/Reply, this script will also drills for original tweet even if outside the scope of the search filter.

Pre-requisites

In order to use those scripts you must have:

  • Python

    • 2.7: All not in the Python 3.X list below
    • 3.7: For graphify.py, archive.py

  • Pip installed

  • Python packages installed:

    • tweepy
    • json
    • time
    • configparser
    • argparse
    • py2neo
    • asyncio
      • requires Python 3.X to work

  • Neo4j Db installed, configured, and ready for connection

    I won’t detail here how to do this part, there are plenty of good tutorials on the Web

archive.py

This script can be used as follow:

  1. usage: archive.py [-h] {file,line} ...
  3. Export tweets that match the search query
  5. positional arguments:
  6.   {file,line}  Add configuration from Ini file or through arguments
  7.     file       Adding configuration from a file (Default: Ini/Default.ini)
  8.     line       Adding configuration from a arguments in the command line
  10. optional arguments:
  11.   -h, --help   show this help message and exit

The file subcommand supports the following syntax:

  1. usage: stream.py file [-h] [-i INI_FILE]
  3. positional arguments:
  4.   cmd
  6. optional arguments:
  7.   -h, --help            show this help message and exit
  8.   -i INI_FILE, --ini_file INI_FILE
  9.                         Path to the Ini file (Default: Ini/Default.ini)

The line subcommand supports the following syntax:

  1. usage: stream.py line [-h] -s SEARCH -ck CONSUMER_KEY -cs CONSUMER_SECRET -ak
  2.                       ACCESS_KEY -as ACCESS_SECRET -o OUTPUT_FILENAME
  3.                       [-b BACKUP_INI_FILE_NAME]
  5. positional arguments:
  6.   cmd
  8. optional arguments:
  9.   -h, --help            show this help message and exit
  10.   -s SEARCH, --search SEARCH
  11.                         Twitter search filter
  12.   -ck CONSUMER_KEY, --consumer_key CONSUMER_KEY
  13.                         Twitter consumer key obtained from your Twitter account
  14.   -cs CONSUMER_SECRET, --consumer_secret CONSUMER_SECRET
  15.                         Twitter consumer secret obtained from your Twitter account
  16.   -ak ACCESS_KEY, --access_key ACCESS_KEY
  17.                         Twitter access key obtained from your Twitter account
  18.   -as ACCESS_SECRET, --access_secret ACCESS_SECRET
  19.                         Twitter access_secret obtained from your Twitter account
  20.   -o OUTPUT_FILENAME, --output_filename OUTPUT_FILENAME
  21.                         Name of the results output file
  22.   -b BACKUP_INI_FILE_NAME, --backup_ini_file_name BACKUP_INI_FILE_NAME
  23.                         Name of the Ini file to backup from this request parameters

graphify.py

This script can be used as follow:

  1. usage: graphify.py [-h] {file,line} ...
  3. Import tweets in a Graph DB
  5. positional arguments:
  6.   {file,line}  Add configuration from Ini file or through arguments
  7.     file       Adding configuration from a file (Default: Ini/Default.ini)
  8.     line       Adding configuration from a arguments in the command line
  10. optional arguments:
  11.   -h, --help   show this help message and exit

The file subcommand supports the following syntax:

  1. usage: graphify.py file [-h] [-i INI_FILE]
  3. positional arguments:
  4.   cmd
  6. optional arguments:
  7.   -h, --help            show this help message and exit
  8.   -i INI_FILE, --ini_file INI_FILE
  9.                         Path to the Ini file (Default: Ini/Default.ini)

The line subcommand supports the following syntax:

  1. usage: graphify.py line [-h] [-type DB_TYPE] [-proto PROTOCOL]
  2.                         [-lang LANGUAGE] [-server SERVER_NAME]
  3.                         [-port SERVER_PORT] -pwd DB_PASSWORD [-set RESULT_SET]
  4.                         -name CONFERENCE_NAME -loc CONFERENCE_LOCATION -time
  5.                         CONFERENCE_TIME_ZONE -start CONFERENCE_START_DATE -end
  6.                         CONFERENCE_END_DATE [-purge PURGE_BEFORE_IMPORT]
  7.                         [-fname FILTER_ORGANIZER_TWITTER_SCREENAME]
  8.                         [-fhash FILTER_CONFERENCE_HASHTAG]
  9.                         [-b BACKUP_INI_FILE_NAME]
  11. positional arguments:
  12.   cmd
  14. optional arguments:
  15.   -h, --help            show this help message and exit
  16.   -type DB_TYPE, --db_type DB_TYPE
  17.                         For future use: indicate db type
  18.   -proto PROTOCOL, --protocol PROTOCOL
  19.                         For future use: indicate protocol to connect to db
  20.   -lang LANGUAGE, --language LANGUAGE
  21.                         For future use: indicate language to query the db
  22.   -sec SECURE, --secure SECURE
  23.                         Flag for secure connection
  24.   -server SERVER_NAME, --server_name SERVER_NAME
  25.                         FQDN of the db server
  26.   -port SERVER_PORT, --server_port SERVER_PORT
  27.                         server socket hosting the db service
  28.   -pwd DB_PASSWORD, --db_password DB_PASSWORD
  29.                         service password to access the db
  30.   -set RESULT_SET, --result_set RESULT_SET
  31.                         Result set file from streaming script (Default: Output/search.json)
  32.   -name CONFERENCE_NAME, --conference_name CONFERENCE_NAME
  33.                         Name of the conference for the master node
  34.   -loc CONFERENCE_LOCATION, --conference_location CONFERENCE_LOCATION
  35.                         Location of the conference for the master node
  36.   -time CONFERENCE_TIME_ZONE, --conference_time_zone CONFERENCE_TIME_ZONE
  37.                         Number of (+/-) hours from UTC reference of the conference's timezone
  38.   -start CONFERENCE_START_DATE, --conference_start_date CONFERENCE_START_DATE
  39.                         First day of the conference in dd/mm/yyyy format
  40.   -end CONFERENCE_END_DATE, --conference_end_date CONFERENCE_END_DATE
  41.                         Last day of the conference in dd/mm/yyyy format
  42.   -purge PURGE_BEFORE_IMPORT, --purge_before_import PURGE_BEFORE_IMPORT
  43.                         Indicate if the graph must be deleted before importing (Default: false)
  44.   -fname FILTER_ORGANIZER_TWITTER_SCREENAME, --filter_organizer_twitter_screename FILTER_ORGANIZER_TWITTER_SCREENAME
  45.                         Twitter screename that helps to filter out organizer tweets and retweets
  46.   -fhash FILTER_CONFERENCE_HASHTAG, --filter_conference_hashtag FILTER_CONFERENCE_HASHTAG
  47.                         Hashtag of the conference
  48.   -b BACKUP_INI_FILE_NAME, --backup_ini_file_name BACKUP_INI_FILE_NAME
  49.                         Name of the Ini file to backup from this request parameters

stream.py

This script can be used as follow:

  1. usage: stream.py [-h] {file,line} ...
  3. Export tweets that match the search query
  5. positional arguments:
  6.   {file,line}  Add configuration from Ini file or through arguments
  7.     file       Adding configuration from a file (Default: Ini/Default.ini)
  8.     line       Adding configuration from a arguments in the command line
  10. optional arguments:
  11.   -h, --help   show this help message and exit

The file subcommand supports the following syntax:

  1. usage: stream.py file [-h] [-i INI_FILE]
  3. positional arguments:
  4.   cmd
  6. optional arguments:
  7.   -h, --help            show this help message and exit
  8.   -i INI_FILE, --ini_file INI_FILE
  9.                         Path to the Ini file (Default: Ini/Default.ini)

The line subcommand supports the following syntax:

  1. usage: stream.py file [-h] [-i INI_FILE]
  3. positional arguments:
  4.   cmd
  6. optional arguments:
  7.   -h, --help            show this help message and exit
  8.   -i INI_FILE, --ini_file INI_FILE
  9.                         Path to the Ini file (Default: Ini/Default.ini)
  11. (base) C:\Users\User\Documents\Git\Work\ConferenceTweetMapper>python stream.py line -h
  12. usage: stream.py line [-h] -s SEARCH -ck CONSUMER_KEY -cs CONSUMER_SECRET -ak
  13.                       ACCESS_KEY -as ACCESS_SECRET -o OUTPUT_FILENAME
  14.                       [-type DB_TYPE] [-proto PROTOCOL] [-lang LANGUAGE]
  15.                       [-server SERVER_NAME] [-port SERVER_PORT] -pwd
  16.                       DB_PASSWORD [-set RESULT_SET] -name CONFERENCE_NAME -loc
  17.                       CONFERENCE_LOCATION -time CONFERENCE_TIME_ZONE -start
  18.                       CONFERENCE_START_DATE -end CONFERENCE_END_DATE
  19.                       [-purge PURGE_BEFORE_IMPORT]
  20.                       [-fname FILTER_ORGANIZER_TWITTER_SCREENAME]
  21.                       [-fhash FILTER_CONFERENCE_HASHTAG]
  22.                       [-b BACKUP_INI_FILE_NAME]
  24. positional arguments:
  25.   cmd
  27. optional arguments:
  28.   -h, --help            show this help message and exit
  29.   -s SEARCH, --search SEARCH
  30.                         Twitter search filter
  31.   -ck CONSUMER_KEY, --consumer_key CONSUMER_KEY
  32.                         Twitter consumer key obtained from your Twitter account
  33.   -cs CONSUMER_SECRET, --consumer_secret CONSUMER_SECRET
  34.                         Twitter consumer secret obtained from your Twitter account
  35.   -ak ACCESS_KEY, --access_key ACCESS_KEY
  36.                         Twitter access key obtained from your Twitter account
  37.   -as ACCESS_SECRET, --access_secret ACCESS_SECRET
  38.                         Twitter access_secret obtained from your Twitter account
  39.   -o OUTPUT_FILENAME, --output_filename OUTPUT_FILENAME
  40.                         Name of the results output file
  41.   -type DB_TYPE, --db_type DB_TYPE
  42.                         For future use: indicate db type
  43.   -proto PROTOCOL, --protocol PROTOCOL
  44.                         For future use: indicate protocol to connect to db
  45.   -lang LANGUAGE, --language LANGUAGE
  46.                         For future use: indicate language to query the db
  47.   -sec SECURE, --secure SECURE
  48.                         Flag for secure connection
  49.   -server SERVER_NAME, --server_name SERVER_NAME
  50.                         FQDN of the db server
  51.   -port SERVER_PORT, --server_port SERVER_PORT
  52.                         server socket hosting the db service
  53.   -pwd DB_PASSWORD, --db_password DB_PASSWORD
  54.                         service password to access the db
  55.   -set RESULT_SET, --result_set RESULT_SET
  56.                         Result set file from streaming script (Default: Output/search.json)
  57.   -name CONFERENCE_NAME, --conference_name CONFERENCE_NAME
  58.                         Name of the conference for the master node
  59.   -loc CONFERENCE_LOCATION, --conference_location CONFERENCE_LOCATION
  60.                         Location of the conference for the master node
  61.   -time CONFERENCE_TIME_ZONE, --conference_time_zone CONFERENCE_TIME_ZONE
  62.                         Number of (+/-) hours from UTC reference of the conference's timezone
  63.   -start CONFERENCE_START_DATE, --conference_start_date CONFERENCE_START_DATE
  64.                         First day of the conference in dd/mm/yyyy format
  65.   -end CONFERENCE_END_DATE, --conference_end_date CONFERENCE_END_DATE
  66.                         Last day of the conference in dd/mm/yyyy format
  67.   -purge PURGE_BEFORE_IMPORT, --purge_before_import PURGE_BEFORE_IMPORT
  68.                         Indicate if the graph must be deleted before importing (Default: false)
  69.   -fname FILTER_ORGANIZER_TWITTER_SCREENAME, --filter_organizer_twitter_screename FILTER_ORGANIZER_TWITTER_SCREENAME
  70.                         Twitter screename that helps to filter out organizer tweets and retweets
  71.   -fhash FILTER_CONFERENCE_HASHTAG, --filter_conference_hashtag FILTER_CONFERENCE_HASHTAG
  72.                         Hashtag of the conference
  73.   -b BACKUP_INI_FILE_NAME, --backup_ini_file_name BACKUP_INI_FILE_NAME
  74.                         Name of the Ini file to backup from this request parameters

Ini file example

In the Ini folder you should find a Default.ini file describing the format expected for a global Ini file:

  1. #Default initialization filter
  2. #All dates shall be in format dd/mm/yyyy
  4. [DEFAULT]
  5. output_filename = Output/search.json
  6. search = #Identiverse
  8. [Twitter]
  9. consumer_key = <your_consumer_key>
  10. consumer_secret = <your_consumer_secret>
  11. access_key = <your_access_key>
  12. access_secret = <your_access_secret>
  14. [Graph]
  15. db_type = Neo4j
  16. protocol = bolt
  17. language = cypher
  18. server_name = localhost
  19. server_port = 7687
  20. db_password = Identiverse
  22. [Processing]
  23. result_set = Output/search.json
  24. conference_name = Identiverse 2018
  25. conference_location = Boston
  26. conference_time_zone = -4
  27. conference_start_date = 24/06/2018
  28. conference_end_date = 27/06/2018
  30. [Misc]
  31. purge_before_import = false
  32. filter_organizer_twitter_screename = Identiverse
  33. filter_conference_hashtag = Identiverse

General limitations and advices

Using those scripts, you understand that:

  • Having two scripts allows to separate the two operations independently
  • Scripts do not check for file existence at the time of exporting (results and configuration), so be careful if you don’t want one to be overwritten
  • Twitter search public API will not return unindexed results, some results older than 7 days, or maybe all the results you may get by using the UI version of it
  • stream.py search filter aims has been designed to target conference hashtag… but it is a standard Twitter search filter supporting all the options Twitter allows
  • graphify.py does only support Neo4j, bolt protocol, and cipher language as for now

Example of the result

If successful you should be able to use Neo4j tools to visualize and drill your Tweet Graph:
Tweet Graph

Example of the drilling of a Retweet/Quote/Reply:
Drilling

Here are some interesting Cipher request examples

Roadmap

  • Generates statistics (1rst level Tweeters, 1rst level Tweets, engaged Twitters, engaged Tweets, etc.)
  • Follow RT, Reply, Quote up and down a la treeverse <- Partially solved, will need script Expand
  • Script Redox: Merge similar RT into only one RT-Tweet
  • Script Expand: Import all the retweets by retweets_of_status_id and replies by in_reply_to_status_id Prenium Search parameters
  • Script Append: Continue an import or update an import with a list of tweets. Look before if tweet is alredy imported or not.
  • Switch script function to Async https://www.aeracode.org/2018/02/19/python-async-simplified/
  • Think about KPIs: Tweet rate, Top for User/Tweet/Hashtag/Mention (see Generates statistics)
  • WebUI to see Graph online
  • Update logging to console to be more dynamic
  • Better date management
  • Change the Post and Pre conference period id to something speciifc to the conference upload to prevent cross mapping
  • Change the Days of conference period id to something speciifc to the conference upload to prevent cross mapping
  • Correct name attribue of object Source to remove href