Skip to content
Snippets Groups Projects
Commit 75f1c67f authored by burkhardt's avatar burkhardt
Browse files

Merge branch 'master' of https://gitlab.cl.uni-heidelberg.de/burkhardt/lsem-rc-in-nominal-compounds

parents 3638a2d4 8f057d83
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
README.md
+ 25
− 6
View file @ 75f1c67f
# Lsem-RC in nominal compounds
# Lexical Semantic Relation Classification for nominal compounds (Lsem-RC-NC)
Ein System soll mit Substantiv Verbindung der Form NC = noun1 noun2 und Paraphrasen,
die die Relation zwischen w1 und w2 beschreiben, trainiert werden. Es soll nun überprüft
werden, ob semantische Beziehungen zwischen Komponenten von Substantivverbindung
gelernt wurden und reproduziert werden können. Dafür wird getestet, inwiefern die in
Paraphrasen maskierten Komponenten - die Verben - maschinell ergänzt werden können.
## Introduction 👋🏼
This repository is part of our project for the course ```Formale Semantik``` at the University Heidelberg. The project task can be summarized as the classification of lexical semantic relations between the components of nominal compounds. If this topic caught your interest the [Project Report]() offers a detailed insight into the project and its outcomes.
## Task 📝
A system is to be trained with a noun compound of the form NC = noun1 noun2 and paraphrases describing the relation between noun1 and noun2. It was now to be tested whether semantic relations between the two components of a noun compound, head and modifier, have been learned and can be reproduced. For this purpose, we tested to what extent the components masked in paraphrases - the verbs - can be completed by a machine and how well relations can be predicted for a nominal compound occurring in a sentence by a fine-tuned model.
## Prerequisites 🗂
In order to be able to run the code in this repository certain prerequisites need to be met. The needed packages and modules are listed in the [requirements.txt](requirements.txt) file. The requirements can be installed using the following command:
```
pip install -r requirements.txt
```
## Structure of this Repository 🧶
| subdirectory | content | README
| ---- | ---- | ---- |
| data | contains all data needed for probing and fine-tuning | [README]()
| documents | contains the first plan for our [**Project Outline**]() and the final [**Project Report**](documents/Gruppe_9__NC-RC_-_Outline.pdf)
| fine_tuning | contains code to fine-tune models, the fine-tuned models, test results and evaluation| [README](fine_tuning/README.md)
| probing | contains code for probing and its evaluation | [README](probing/README.md)
## Authors & Contact 📮
Do you have any questions or comments? Feel free to reach out to us! 💥 <br>
The ELSE-Team consists of
[**E**ike Burkhardt](mailto:eike.burkhardt@gmx.net), [Janine **L**üger](mailto:lueger@stud.uni-heidelberg.de),[**S**tella Wernicke](mailto:stella.wernicke@stud.uni-heidelberg.de) and [Katharina **E**ngel](mailto:a-katharina.engel@stud.uni-heidelberg.de).
\ No newline at end of file
fine_tuning/README.md
+ 8
− 4
View file @ 75f1c67f
......@@ -71,8 +71,12 @@ The mode (fine-tuning == 'f' or testing == 't') and the input file are given to
The files should be comma separated with at least two columns **Label** and **Sentence**. Fine-tuning requires two input files for tuning and validation. One is provided by command line and the other one has to be exactly the same name in the same directory only with val instaed of train e.g. `file_train.csv` and `file_val.csv`.
Examples:
- fineBert.py f file_train.csv
- fineBert.py t file_test.csv
```
fineBert.py f file_train.csv
```
```
fineBert.py t file_test.csv
```
### Results
The resulting model of the fine-tuning process is stored in a directory named after the input file, batch size and learning rate. We output numpy arrays with true labels and predictions which are saved in the working directory. For better clarity we moved them in the results directory.
......@@ -105,13 +109,13 @@ Visualized results for fine grained relations:
</div></p>
For each batch size a similar behavior can be seen. While the training loss is the highest after the first epoch of training it is reduced to under 0.1 in every scenario after the second epoch. The training loss gets even more reduced with more epochs, whereas the validation loss is quite high after one epoch, remains roughly on the same level only between the first and second epoch and increases after that. This is a sign of overfitting, which means that our trained model corresponds too closely to the train data. In our case the increase of the validation loss is not drastic but shows that two epochs are sufficient for training our model. <br>
The accuracy the model achieves stays more or less the same over all epochs and for every batch size. <br>
The accuracy the model achieves stays more or less the same over all epochs, learning rates and for every batch size. <br>
If the three plots are compared one can see that the results for the different learning rates are very close to another and they also vary with every execution of the training process. As a trade-off between economy (learning rate not too small) and performance (high accuracy) we decided on a **batch size of 32** and a **learning rate of 3e-05** that lead to the validation loss of around 1.89 and the accuracy of 0.66. <br>
The same goes for the coarse grained relations. A comparable behavior can be seen in the plot below.
<p><div align="center">
![](plots/model_sep_coarse_50_train_set_16.png)
![](plots/model_sep_coarse_50_train_set_32.png)
</div></p>
......
probing/README.md
+ 17
− 17
View file @ 75f1c67f
......@@ -111,11 +111,11 @@ complement | book title | A {n1} {n2} is a {n2} that {mask} the {n1} | character
containment | cigarette pack | A {n1} {n2} is a {n2} that {mask} of {n1}s | consists
loc_part_whole | hotel lobby | A {n1} {n2} is a {n2} that is {mask} in {n1} | located
objective | pipeline explosion | A {n1} {n2} is a {n2} that {mask} on {n1} | focusses
other | fund system |
other | fund system | ?? | ??
owner_emp_use | navy commander | A {n1} {n2} is a {n2} that {mask} by a {n1} | owned
purpose | alarm clock | A {n1} {n2} is a {n2} that is {mask} for {n1} | used
time | night work | A {n1} {n2} is a {n2} that {mask} place at {n1} | takes
topical | student database |
topical | student database | A {n1} {n2} is a {n2} that {mask} {n1} | regards
</div>
......@@ -131,39 +131,39 @@ Relation | Example | Template | Gold Verb
ADJ-LIKE_NOUN | mass destruction | A {n1} {n2} is a {n2} that is {mask} as a {n1} | considered
AMOUNT-OF | credit volume | A {n1} {n2} is a {n2} that {mask} {n1} | quantifies
CONTAIN | photo album | A {n1} {n2} is a {n2} that {mask} a {n1} | contains
CREATE-PROVIDE-GENERATE-SELL | drug office |
CREATE-PROVIDE-GENERATE-SELL | drug office | A {n1} {n2} is a {n2} that {mask} {n1} | creates
CREATOR-PROVIDER-CAUSE_OF | hate crime | A {n1} {n2} is a {n2} that is {mask} by {n1} | caused
EMPLOYER | government agent | A {n1} {n2} is a {n2} that is {mask} by the {n1} | employed
EQUATIVE | consultation process | A {n1} {n2} is a {n2} that {mask} for a {n1} | is
EXPERIENCER-OF-EXPERIENCE | consumer comfort | A {n1} {n2} is a {n2} that is {mask} by a {n1} | experienced
JUSTIFICATION | conspiracy trial |
JUSTIFICATION | conspiracy trial | A {n1} {n2} is a {n2} that is {mask} by a {n1} | justified
LEXICALIZED | pay phone | A {n1} {n2} is a {n2} that {mask} for {n1} | is
LOCATION | eye makeup | A {n1} {n2} is a {n2} that is {mask} in a {n1} | located
MEANS | hunger strike |
MEANS | hunger strike | A {n1} {n2} is a {n2} that {mask} {n1} | uses
MEASURE | mile track | A {n1} {n2} is a {n2} that {mask} over the {n1} | extends
MITIGATE&OPPOSE | crime unit | A {n1} {n2} is a {n2} that is used to {mask} the {n1} | mitigate
OBJECTIVE | poverty reduction |
OBJECTIVE | poverty reduction | ?? | ??
OBTAIN&ACCESS&SEEK | ownership right | A {n1} {n2} is a {n2} that has the function to {mask} a {n1} | achieve
ORGANIZE&SUPERVISE&AUTHORITY | government authority |
ORGANIZE&SUPERVISE&AUTHORITY | government authority | A {n1} {n2} is a {n2} that {mask} {n1} | supervises
OTHER | sports organization | A {n1} {n2} is a {n2} that is {mask} to {n1} | related
OWNER-USER | government property
PARTIAL_ATTRIBUTE_TRANSFER | mushroom lamp
OWNER-USER | government property | A {n1} {n2} is a {n2} that is {mask} by a {n1} | owned
PARTIAL_ATTRIBUTE_TRANSFER | mushroom lamp | A {n1} {n2} is a {n2} that {mask} a {n1} | resembles
PART&MEMBER_OF_COLLECTION&CONFIG&SERIES | citizen committee | A {n1} {n2} is a {n2} that {mask} of {n1}s | consists
PERFORM&ENGAGE_IN | fax machine
PERSONAL_NAME | Josh Gorbachev | -
PERSONAL_TITLE | Ms. Patterson | -
PERFORM&ENGAGE_IN | fax machine | A {n1} {n2} is a {n2} that {mask} a {n1} | performs
PERSONAL_NAME | Josh Gorbachev | -- | --
PERSONAL_TITLE | Ms. Patterson | -- | --
PURPOSE | court system | A {n1} {n2} is a {n2} that is {mask} for {n1} | used
RELATIONAL-NOUN-COMPLEMENT | eye color | A {n1} {n2} is a {n2} that is {mask} to a {n1} | related
SUBJECT | family violence |
SUBJECT | family violence | A {n1} {n2} is a {n2} that is {mask} by a {n1} | conducted
SUBSTANCE-MATERIAL-INGREDIENT | aluminum can | A {n1} {n2} is a {n2} that is {mask} of {n1} | made
TIME-OF1 | advance sale | A {n1} {n2} is a {n2} that {mask} during a {n1} | happens
TIME-OF2 | watermelon season |
TOPIC | soundtrack album |
TOPIC_OF_COGNITION&EMOTION | phone fanatic |
TIME-OF2 | watermelon season | A {n1} {n2} is a {n2} that {mask} the {n1} | dates
TOPIC | soundtrack album | A {n1} {n2} is a {n2} that {mask} with {n1} | deals
TOPIC_OF_COGNITION&EMOTION | phone fanatic | A {n1} {n2} is a {n2} that {mask} to the {n1} | relates
TOPIC_OF_EXPERT | technology expert | A {n1} {n2} is a {n2} that {mask} with {n1} | deals
USER_RECIPIENT | employee benefit | A {n1} {n2} is a {n2} that is {mask} for a {n1} | provided
VARIETY&GENUS_OF | grape variety | A {n1} {n2} is a {n2} that {mask} down {n1} | breaks
WHOLE+ATTRIBUTE&FEATURE& QUALITY_VALUE_IS_CHARACTERISTIC_OF | butter texture |
WHOLE+ATTRIBUTE&FEATURE& QUALITY_VALUE_IS_CHARACTERISTIC_OF | butter texture | A {n1} {n2} is a {n2} that {mask} {n1} | characterizes
WHOLE+PART_OR_MEMBER_OF | whale tongue | A {n1} {n2} is a {n2} that {mask} part of a {n1} | is
</div>
......
requirements.txt 0 → 100644
+ 9
− 0
View file @ 75f1c67f
transformers
gensim
tensorflow
tqdm
seaborn
pandas
numpy
re
os
temp/README.md
+ 4
− 0
View file @ 75f1c67f
# Data
## Table of Contents
- [Data](#data)
- [Table of Contents](#table-of-contents)
- [Searching for data](#searching-for-data)
- [Forcing BERTs Attention on the compound](#forcing-berts-attention-on-the-compound)
- [Limiting NC Occurrences](#limiting-nc-occurrences)
......
temp/data_split.ipynb deleted 100644 → 0
+ 0
− 474
View file @ 3638a2d4
%% Cell type:code id: tags:
``` python
import random
import pandas as pd
```
%% Cell type:code id: tags:
``` python
df = pd.read_csv("sentences_fine_200.csv")
```
%% Cell type:code id: tags:
``` python
sent_amount = len(df)
perc_train = sent_amount*0.6
perc_test = sent_amount*0.25
perc_val = sent_amount*0.15
print(perc_train)
print(perc_val)
print(perc_test)
```
%% Output
104350.8
26087.7
43479.5
%% Cell type:code id: tags:
``` python
# Create different DataFrames for each set
train_set = pd.DataFrame(columns = ['Relation', 'Label', 'NC', 'Sentence'])
val_set = pd.DataFrame(columns = ['Relation', 'Label', 'NC', 'Sentence'])
test_set = pd.DataFrame(columns = ['Relation', 'Label', 'NC', 'Sentence'])
```
%% Cell type:code id: tags:
``` python
grouped_NC = df.groupby(['Label', 'NC']).size().reset_index(name="Count")
grouped_rel = df.groupby(['Label']).size().reset_index(name="Count_rel")
sum = 0
# iterate over relations to get amount of sentences from each relation
for index, row in grouped_rel.iterrows():
count = row['Count_rel']
amount = count*0.6
print(f"sum {sum}")
print("Label {}".format(row['Label']))
print(f" not over {amount}")
# iterate over NC of relation to get sentences and dont split up NC in different sets
# amount of sentences should stay below amount calculated above
if sum < perc_train:
sum2 = 0
# get only nc from relevant relation
NC_rel = grouped_NC.where(grouped_NC['Label']== row['Label']).dropna().reset_index(drop=True)
for index2, row2 in NC_rel.iterrows():
count2 = row2['Count']
temp = sum2 + count2
if temp < amount and sum+sum2 < perc_train:
df1 = df[df['NC'].str.contains(row2['NC'])]
train_set = pd.concat([train_set, df1], axis=0)
sum2 += count2
sum = len(train_set)
train_set.to_csv(f'./train_data/train_set.csv')
```
%% Output
sum 0
Label 1
not over 1210.2
sum 1197
Label 2
not over 1104.6
sum 2301
Label 3
not over 930.5999999999999
sum 3231
Label 4
not over 4637.4
sum 7868
Label 5
not over 1587.0
sum 9453
Label 6
not over 3437.4
sum 12890
Label 7
not over 6597.599999999999
sum 19797
Label 8
not over 297.59999999999997
sum 20094
Label 9
not over 103.8
sum 20192
Label 10
not over 1199.3999999999999
sum 21390
Label 11
not over 5962.8
sum 27352
Label 12
not over 1728.0
sum 29079
Label 13
not over 6036.599999999999
sum 35115
Label 14
not over 731.4
sum 35846
Label 15
not over 15852.0
sum 51706
Label 16
not over 1271.3999999999999
sum 52977
Label 17
not over 1659.6
sum 54732
Label 18
not over 4556.4
sum 59292
Label 19
not over 2518.7999999999997
sum 61824
Label 20
not over 1667.3999999999999
sum 63489
Label 21
not over 385.2
sum 63871
Label 22
not over 13704.0
sum 77574
Label 23
not over 2521.2
sum 80098
Label 24
not over 6627.599999999999
sum 86737
Label 25
not over 3777.0
sum 90523
Label 26
not over 1959.6
sum 92483
Label 27
not over 2916.6
sum 95400
Label 28
not over 819.6
sum 96217
Label 29
not over 5308.8
sum 100331
Label 30
not over 156.6
sum 100392
Label 31
not over 319.8
sum 100708
Label 32
not over 789.0
sum 101496
Label 33
not over 60.599999999999994
sum 101513
Label 34
not over 474.0
sum 101975
Label 35
not over 1441.2
%% Cell type:code id: tags:
``` python
merge_set = pd.merge(df, train_set, how='outer', indicator=True)
reduced = merge_set.loc[merge_set._merge == 'left_only', ['Relation','Label','NC','Sentence']]
print(reduced.head)
```
%% Output
<bound method NDFrame.head of Relation Label NC \
1173 ADJ-LIKE_NOUN 1 mass destruction
1174 ADJ-LIKE_NOUN 1 mass destruction
1175 ADJ-LIKE_NOUN 1 mass destruction
1176 ADJ-LIKE_NOUN 1 mass destruction
1177 ADJ-LIKE_NOUN 1 mass destruction
... ... ... ...
173965 WHOLE+PART_OR_MEMBER_OF 35 wing tip
173966 WHOLE+PART_OR_MEMBER_OF 35 wing tip
173967 WHOLE+PART_OR_MEMBER_OF 35 wing tip
173968 WHOLE+PART_OR_MEMBER_OF 35 wing tip
173969 WHOLE+PART_OR_MEMBER_OF 35 wing tip
Sentence
1173 one used a primitive revolver; the other a wea...
1174 american obligations to come to israel’s defen...
1175 archbishop jose h. gomez of los angeles in a j...
1176 as robert draper recently us, those in the adm...
1177 didn’t powell say that iraq had ‘weapons of ma...
... ...
173965 snip out the bird’s backbone and add it to the...
173966 the arrest was made following tip-off received...
173967 the incident occurred when police, including g...
173968 apical meristem or growing tip.
173969 lesser is a smaller bird, with slimmer build, ...
[70748 rows x 4 columns]>
%% Cell type:code id: tags:
``` python
val_set = pd.DataFrame(columns = ['Relation', 'Label', 'NC', 'Sentence'])
grouped_NC = reduced.groupby(['Label', 'NC']).size().reset_index(name="Count")
grouped_rel = reduced.groupby(['Label']).size().reset_index(name="Count_rel")
sum = 0
# iterate over relations to get amount of sentences from each relation
for index, row in grouped_rel.iterrows():
count = row['Count_rel']
amount = count*0.15
print(f"sum {sum}")
print("Label {}".format(row['Label']))
print(f" not over {amount}")
# iterate over NC of relation to get sentences and dont split up NC in different sets
# amount of sentences should stay below amount calculated above
if sum < perc_val:
sum2 = 0
# get only nc from relevant relation
NC_rel = grouped_NC.where(grouped_NC['Label']== row['Label']).dropna().reset_index(drop=True)
for index2, row2 in NC_rel.iterrows():
count2 = row2['Count']
temp = sum2 + count2
if temp < amount and sum+sum2 < perc_val:
df1 = reduced[reduced['NC'].str.contains(row2['NC'])]
val_set = pd.concat([val_set, df1], axis=0)
sum2 += count2
sum = len(val_set)
val_set.to_csv(f'./val_data/val_set.csv')
```
%% Output
sum 0
Label 1
not over 123.0
sum 116
Label 2
not over 110.55
sum 224
Label 3
not over 93.14999999999999
sum 317
Label 4
not over 463.79999999999995
sum 780
Label 5
not over 159.0
sum 937
Label 6
not over 343.8
sum 1280
Label 7
not over 613.35
sum 1893
Label 8
not over 29.849999999999998
sum 1919
Label 9
not over 11.25
sum 1919
Label 10
not over 120.14999999999999
sum 2036
Label 11
not over 595.9499999999999
sum 2631
Label 12
not over 172.95
sum 2803
Label 13
not over 603.75
sum 3405
Label 14
not over 73.2
sum 3477
Label 15
not over 1585.35
sum 5062
Label 16
not over 127.19999999999999
sum 5180
Label 17
not over 166.04999999999998
sum 5346
Label 18
not over 455.7
sum 5801
Label 19
not over 252.14999999999998
sum 6052
Label 20
not over 167.4
sum 6216
Label 21
not over 39.0
sum 6251
Label 22
not over 1356.1499999999999
sum 7607
Label 23
not over 252.14999999999998
sum 7859
Label 24
not over 662.85
sum 8521
Label 25
not over 376.34999999999997
sum 8897
Label 26
not over 196.04999999999998
sum 9093
Label 27
not over 291.75
sum 9382
Label 28
not over 82.35
sum 9460
Label 29
not over 710.1
sum 10170
Label 30
not over 30.0
sum 10170
Label 31
not over 32.55
sum 10200
Label 32
not over 79.05
sum 10279
Label 33
not over 12.6
sum 10279
Label 34
not over 49.199999999999996
sum 10279
Label 35
not over 174.45
%% Cell type:code id: tags:
``` python
merge_set2 = pd.merge(reduced, val_set, how='outer', indicator=True)
test_set = merge_set2.loc[merge_set2._merge == 'left_only', ['Relation','Label','NC','Sentence']]
print(test_set.head)
test_set.to_csv(f'./test_data/test_set.csv')
```
%% Output
<bound method NDFrame.head of Relation Label NC \
116 ADJ-LIKE_NOUN 1 mass extinction
117 ADJ-LIKE_NOUN 1 mass extinction
118 ADJ-LIKE_NOUN 1 mass extinction
119 ADJ-LIKE_NOUN 1 mass extinction
120 ADJ-LIKE_NOUN 1 mass extinction
... ... ... ...
70743 WHOLE+PART_OR_MEMBER_OF 35 wing tip
70744 WHOLE+PART_OR_MEMBER_OF 35 wing tip
70745 WHOLE+PART_OR_MEMBER_OF 35 wing tip
70746 WHOLE+PART_OR_MEMBER_OF 35 wing tip
70747 WHOLE+PART_OR_MEMBER_OF 35 wing tip
Sentence
116 earth is at the start of a sixth mass extincti...
117 the heisei era indicates that godzilla was a d...
118 the impact would have thrown trillions of tons...
119 across england and wales, towns and villages a...
120 geographically widespread organisms fare bette...
... ...
70743 snip out the bird’s backbone and add it to the...
70744 the arrest was made following tip-off received...
70745 the incident occurred when police, including g...
70746 apical meristem or growing tip.
70747 lesser is a smaller bird, with slimmer build, ...
[60295 rows x 4 columns]>
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment