@@ -135,13 +135,13 @@ Apart from the regular sentences containing the nominal compound we changed the
For all these different test sets two to three different scenarios were tested:
- just the sentence
- the sentence with the nominal compound added after a [SEP] token (_sep_)
- if one of the components got deleted, the sentence with the still occurring component added after a [SEP] token or in the case of _flipnc_ both components flipped (_alt_)
- if one of the components got deleted, the sentence with the still occurring component added after a [SEP] token, for _rndsent_ the original NC belonging to the sentence or in the case of _flipnc_ both components flipped (_alt_)
The results of these test runs with our fine-tuned model for coarse and fine grained data can be seen below. In general it can be stated that the accuracy and the loss scores have a similar distribution for both kinds of relations. The best results are achieved when using our model on test data, that has the nominal compound added with [SEP] tokens after the sentence. This leads to higher self-attention scores for the nominal compound in the sentence as it occurred twice and helps the model to recognize these two words as especially important for the classification task. The performance on a test set without these added tokens the loss is nearly three times as high and the accuracy drops drastically. <br>
The results of these test runs with our fine-tuned model for coarse and fine grained data can be seen below. In general it can be stated that the accuracy and the loss scores have a similar distribution for both kinds of relations. The best results are achieved when using our model on test data, that has the nominal compound added with [SEP] tokens after the sentence. The performance on a test set without these added tokens the loss is nearly three times as high and the accuracy drops drastically. <br>
With variations of leaving out the compound's head or modifier possible biases were tested, like if the model just learns that a NC belongs to a certain relation because of its head or modifier that occurs multiple times in that same relation. <br>
To our relief the accuracy drops significantly without the head or modifier in the sentence and the loss increases. The head seems to play a more important role than the modifier for a good classification as test on a "nomodi" set get higher accuracies than test on a "nohead" set. But we can conclude that our model does not seem to solely learn the components of the nominal compounds and the relation they belong to as there is a significant influence on the performance if one component is deleted in the sentence.<br>
Another interesting insight was that the accuracy, when classifying NCs occurring in a random sentence, still is nearly as high as the accuracy on the set with them in the correct sentence.<br>
The last testing experiment was on the test set with the components flipped. Interestingly enough the order of them occurring seems to play a role in the performance as there is a drop of accuracy of a third (coarse) or half in the case of finegrained relations.
The last testing experiment was on the test set with the components flipped. Interestingly enough, the order of them occurring seems to play a role in the performance as there is a drop of accuracy of a third for coarse and half in the case of fine-grained relations. If the nominal compound is added after the [SEP] the accuracy does not change much, but as soon as the addition is also swapped, we see the impact the order seems to have. <br>
