# Semantic Textual Similarity
 
## ❓ Purpose
The overarching goal of this project is to increase the understanding of masked language models (MLM) and bring forth new ideas about their way of functioning. We will specifically look at SBERT models, which are fine-tuned only on NLI. For more details, please see our [project outline](https://gitlab.cl.uni-heidelberg.de/reichelt/semantic-textual-similarity/-/blob/master/project_outline_sts.pdf). There you can also find information on the models and data sets we used.
 
## GitLab
This project can be accessed on [GitLab](https://gitlab.cl.uni-heidelberg.de/reichelt/semantic-textual-similarity) by any logged in user.
 
## Directory Structure
In this directory, you can find our trained models, data sets, scripts we used to perform probing and evaluation of the models, the output of these scripts as well as some additional project configuration and information files.
 
Note: On GitLab, the repository does *not* include the trained models due to their size. These can be found in [this](https://heibox.uni-heidelberg.de/d/04a6e6ca5c154f1ba6bc/) heiBOX folder.
 
### src
Our scripts and modules can be found in the `src directory`.
 
There, you will find two classes STSB_Model and STSB_Dataset which are, essentially, wrapper modules with some handy functions for our models and datasets for easier access in other modules. Additionally, there is a file named paws_training.py which was created for one-time use to fine-tune the sts-bert-large model on the PAWS data set, as explained in our [project report](https://gitlab.cl.uni-heidelberg.de/reichelt/semantic-textual-similarity/-/blob/master/project_report_sts.pdf). All other files in the src directory contain probing scripts which are described further in their documentation and the Usage section of this README.
 
### data
Here you can find all the STS Data Sets for training, developing and testing the models along with a text file containing further information about the dataset.
 
### outputs
 
This section contains all the visual and some informational output of our scripts.
The directories’ names indicate the respective topics.
 
## Requirements
 
Our code depends on many external libraries which you can install using our requirements.txt file:
 
```bash
pip install -r requirements.txt
```
 
Alternatively, you can install the required packages using a conda installation with our environment export `s-trans.yml`:
 
```bash
conda env create -f s-trans.yml
```
 
Should you encounter issues with the installation of en-core-web-sm, the simplest solution is to download it separately:
 
```bash
python -m spacy download en_core_web_sm
```
 
## Installation
Our project does not currently have a method of installation implemented. You can view the output of our scripts in the outputs folder. If you wish to run the scripts yourself and do not have a copy of them locally yet, you need to manually download them from GitLab. When downloading from GitLab, please remember to also download the trained models from the link provided above and store them in a directory `./models`. To download other modules required to run our scripts, please refer to the requirements section of this README.
 
## Usage
 
Since our probing scripts all have separate purposes, it does not make sense to connect them in a `main.py` file or similiar. Instead, you can run the specific script for which you wish to see the outcome. Here, we briefly explain the scripts and how to use them. See also: code documentation, project report.
 
### Sentence similarity
sentence_similarity.py is a basic script which loads a BERT model and the STS-B data set. It can print the performance evaluation metrics and/or create visualizations for the embeddings. You can choose which functionalities you want by commenting out or enabling code. 


### Masking
To recreate our masking experiments, import probing to SBERT_Model.py.
When loading the dataset, call one of the methods in probing.py (e.g. `mask_token("NOUN")`) and assign it to the dataset variable.
POS-sensitive methods will require one argument of type string, POS-non-sensitive methods should be called without any arguments.
The string should be a valid POS like "NOUN" or "ADJ".
The methods will return the changed dataset.
 
Example:
`dataset = probing.mask_token("VERB")`
 
### Sentence length
sentence_length.py tests how the length of sentences influence how well similarity can be embedded. You can simply run the script without any changes or parameters. It will print some statistics concerning sentence lengths and the performance evaluation on data sets with different sentence lengths.
 
### Attention visualization
attention_visualization.py produce a heat map for the attentions used by BERT to create the embedding of a specified sentence. You can specify the sentence you want to use as an argument when running the script, for example:
 
```bash
python attention_visualization.py "[CLS] US drone strike kills 10 in Pakistan"
```
 
Note the use of `[CLS]` as a first token. You should ideally include this in your input. The reason for this is explained in sectinon 4.5 (Attention heat map analysis) of our project report.
 
Per default, the attention values in relation to the first token of the sentence after `[CLS]` are used. You can change this by adjusting line 19 of the script.
 
### WIViT generation

To generate WIViT, first initialize a model using SBERT_MODEL class and execute the `get_html_word_importance` method on it, with a list of sentence pairs as an argument. A sentence pair should be a tuple of strings.

Example:
```python
ft_model = SBERT_Model("stsb-bert-large", f'{models_path}/stsb-bert-large/', dataset)
ft_model.get_html_word_importance(sentence_pairs)
```

### PAWS evaluation

paws_evaluation.py essentially creates an instance of the SBERT_Model class with all three models and the PAWS data set and calls functions that evaluate those models on it.
 
## 🔬 Results
 
For our results, please see our [project report](https://gitlab.cl.uni-heidelberg.de/reichelt/semantic-textual-similarity/-/blob/master/project_report_sts.pdf).
 
## 👥 Authors
Daniel Podrażka \
Aileen Reichelt \
Nico Schnurr \
Ivo Schaper
 
## 📧 Contact us
{podrazka, reichelt, schnurr, schaper}@cl.uni-heidelberg.de
 
## 📄 License
This project is licensed under the MIT license.

