Part 2 of the Web Data Mining course @ DWS MSc

Our report

Repository structure

• In the root of this repository there are the required files to deploy the website. There is already an online version available at Netlify. (Just press the button above)
• In the dataset/ folder there is a dump of the MongoDB database that we used for our analyses.
• In the analysis-scripts/ folder there are all the various scripts that we used for our analyses.

Usage

All of the steps below require that you have already installed:

• Node.js 12.13 or later
• Python 3.6 or later
• MongoDB 4.0 or later

Website

• Install depencencies:

  npm i

• Run the development server:

  npm run develop

• Or build and serve the production bundle:

  npm run build
  npm run serve

Mongo dataset

• Download the dump from the dataset/ folder.
• Join the three parts of the .gz file into one, using something like:
  • Windows: type twitter_search.bson.gz.* > twitter_search.bson.gz
  • Linux/MacOS: cat twitter_search.bson.gz.* > twitter_search.bson.gz
• Use the mongorestore command, in conjuction with the --gzip option.

Python scripts

• Install dependencies from analysis-scripts/requirements.txt

• Download necessary nltk files. In a python3 shell:

  >>> import nltk
  >>> nltk.download('stopwords')
  >>> nltk.download('punkt')

• Make sure to set each script's required environment variables before executing it!

• Also, again, make sure that you have at least 3.6 Python, because these script use f literals.