Psychology topic models

Requires: docker, node, python

1. Parsing, storing in db

in pubmed-to-db:

1.1 Install dependencies: npm install

1.2 Start database: docker-compose up

1.3 Create database table structure: node db-setup.js

1.4 Download xml-results from the source: https://www.ncbi.nlm.nih.gov/pubmed/?term=psychology

1.5 Parse file: node parse-from-xml.js <path-to-xml-file>

(xml-file refers to an export from pubmed)

2. Preprocessing

in topic-modelling:

2.1 Install modules pip install nltk spacy gensim

2.2 Download nltk stopwords: python download_stopwords.py

2.3 Download spacy en module: python -m spacy download en

2.4 Preprocess the texts: python preprocess.py

2.5 Transform corpus to dictionary and bag-of-words structure: python -c "from transform_corpus import *; save_corpus_and_dictionary_to_file()"

3. LDA

in topic-modelling:

3.1 Download Mallet, set MALLET_PATH environment variable

3.2 Generate a topic model: python generate_model.py <'gensim' | 'mallet'> ...topic_number_configurations

3.3 Calculate coherence scores: python get_coherence.py ...model_names

3.4 Extract topics to a csv file: python extract_topics.py <model-name> <number of topics>