Advanced Data Mining Project - data scraping, analysis and processing; models' training and evaluation

1. Data scraping & EDA

In order to scrape the Google Maps reviews from a chosen set of restaurants, I needed an automated pipeline that is able to interact with the service in a similar way a human would. This is crucial since only this way I can leverage the internal Google’s algorithms for restaurant lookup, as well as loading, sorting, and automated translation of the reviews. To this end, we incorporated Microsoft’s package Playwright, which allows us to automatically interact with a browser engine.

The EDA involved analysing the scraped raw data in order to understand its structure, distribution, and potential insights. The distribution of various features was examined with respect to data categories, such as whether the review was originally in English.

Insight into authors' review writing patterns

Analysis of the distributions w.r.t. the location size

Analysis of the distributions w.r.t. the review original language

Analysis of the categorical options

2. Data Processing & Post-processing Analysis

Processing the data involved two steps. First, the processor modules were fit the training data and used to produce the numerical features. Then, basic stats, such as mean and standard deviation, were calculated on the extracted features. Next step involved using the produced stats as well as the saved processors' states to obtain the test dataset. This two-step process ensures that there's no data leakage between the training and test sets.

The following classes of features were extracted from the raw data.

1. BERT-based embeddings
   • sentence-level embeddings obtained from the [CLS] BERT's special token
   • available in two modes: a sequence of sentence embeddings or a single averaged embedding for the review
2. Word count-based features
   • based on classical text processing techniques involving counting word occurrences
   • available as tensors containing features for the top-k most common words to control model size
   • used by the model in several possible modes: raw count, normalized count, TF-IDF and Binary Bag-of-Words
3. POS-based features
   • a vector with part-of-speech tag frequencies
   • available in raw count and normalized count forms
4. Categorical features
   • extracted based on a chosen subset of categorized options' values
   • an additional feature: quantized number of reviews written by the author
5. Trace features
   • (velocity, volume) pairs inspired by this paper
   • calculated using spatial relationships between chunks created from token-level BERT embeddings using a given chunk size and step size
   • velocity indicates the pace with which a given review jumps from topic to topic
   • volume indicates the semantical richness of the review

   

Processing details

Analysis of the processed dataset

3. Model architecture & metrics

The ultimate goal of this work is to develop a neural network capable of predicting the rating of the given review. The model is designed to use a chosen subset of the data features and encode them using dedicated encoders. Depending on the nature of the input data, the model uses either Linguistic Encoders, that consist of a stack of fully-connected layers with ReLU activations, normalization and dropout, or Numerical Encoders, that leverage Kolmogorov-Arnold networks, which take longer to train but are more expressive.

All encoded features are subsequently concatenated and passed through a post-net that produces two kinds of rating prediction output: regressive rating prediction and rating classification either of which can be used to train the model. Optionally, classification of the model original language can be used as a secondary task.

The following metrics are used during the training (the regression output is always rounded and clamped to produce a single rating class):

1. Accuracy
2. Precision, Recall - calculated with respect to each class and averaged in either the macro or weighted setup.
3. Accuracy, Precision and Recall for translation flag prediction task
4. Area Under the ROC Curve for translation flag prediction task
5. Metrics for Coarse prediction - includes precision and recall calculated for the coarse-grained rating prediction, where ratings 1 and 2 are treated as the "bad" class and 3-5 are treated as the "good" class.

4. Experiments

The experiments involved training different setups of the model, where a single experiment's study was treated as standard hyperparameter tuning, used to obtain the best model for a given set of scientific parameters. Each experiment could involve comparing the best models of several studies to gain insight into the relationship between a certain parameter and the model's performance.

Due to the imperfect nature of the reviews rating task, there's no perfect algorithm to achieve 100% accuracy, as the rating is in practice often strongly subjective. It is reflected in the level of confusion in the prediction of neighboring classes. For this reason, I focused on evaluating the model's ability to predict the coarse "bad" and "good" classes. I believe this to be a more realistic and meaningful evaluation for this type of task.

Obtaining the baseline model

Ablation studies

The ablation studies were conducted to evaluate the contribution of each component in the model. For each study, the best model was chosen from a set of trained configurations. The optimized hyperparameters within each study included regularization and optimizer parameters.

5. Evaluation

The evaluation was conducted on the test set which is independent from the training data. This separation aims to reflect testing the model in production - the models are optimized over the validation dataset and evaluated on independent data.

Slice-based evaluation

Slice-based evaluation aims to get insight into the model's performance on chosen categories of the data. This type of evaluation allows to be aware of the possible model's weaknesses, e.g. bad performance on scarcely represented groups.