@@ -36,20 +36,22 @@ After the human needs have been successfully assigned for all essays, we evaluat
</p>
> The system scored an accuracy of roughly 28% for Maslow categories, while scoring merely 7.7% for Reiss motives. Just as we had conjectured at the beginning of our project (and also experienced ourselves during the Inter-Annotor-Agreement phase), the assigment of Maslow categories is significantly easier for both the system and us human. This is due to the more coarse-grained nature of the Maslow categories, which only consist of 5 categories and thus leavey the system and us human less options to choose from. With a broad spectrum of possibile categories to choose from, the system struggles more and has significant difficulties to assign the correct Reiss motive (though it did assign the correct Maslow category for many examples). <br> The more fine-grained nature of the Reiss motives provide much more options, some of which correspond heavily to one another, which makes the assigment of a correct Reiss motive even more difficult. This confirms our study conjectures at the beginning: <br> **The classification of more coarse-grained categories works signiifcantly better than that of fine-grained categories.**
>>Though the results may seem disappointing at first sight, we're still somewhat satisfied with them as we had to use a different, more naive approach, which was still effective in assigning human needs. For the accurate consideration of our results, we also need to take the nature of our texts into account. Entire paragraphs provide much more tokens and thus also much more information, which also makes them more difficult to process. Another factor that contributes to the difficulty of these texts is that they are all of varying lengths. Still, at least for Maslow, the system was able to score satisfying results.
>>Though the results may seem disappointing at first sight, we're still somewhat satisfied with them as we had to use a novel, more naive approach, which was still effective in assigning human needs. For the accurate consideration of our results, we also need to take the nature of our texts into account. Entire paragraphs provide much more tokens and thus also much more information, which also makes them more difficult to process. Another factor that contributes to the difficulty of these texts is that they are all of varying lengths. Still, at least for Maslow, the system was able to score satisfying results.
<palign="center">
<imgsrc="pictures/Recall_test.png"/>
</p>
> The system scored a macro-averaged recall of roughly 20% for Maslow categories, while scoring just 3.54% for Reiss motives. Again, these values for the recall metric confirm our study conjecture that fine-grained categories like Reiss motives leave the system too much options to choose from, which lowers the system's performance and confidence significantly. As mentioned before, another contributing factor to this is that some Reiss motives are heavily connected to each other and could possibly be seen as interchangeable, which explains the low results for recall. Of particular interest here are the Reiss motives "order" and "curiosity": motive "order" had a precision of 50% but a recall value of only 7%, meaning that the system struggled to retrieve the relevant examples for this class. For "curiosity", the opposite is true, as the system scored a precision of 16% but a recall of 25%. Given a much smaller range of options, the system is much more confident and thus provides even better performance for Maslow categories. Especially "spiritual growth" and "stability" provide good results here with 38% each, which is not suprising given that they are the most frequently assigned Maslow categories in our gold annotations. However, classes which are assigned with an insignificant frequency in the gold data perform poorly, as is visible by the results for the class "physiological needs" (and its corresponding Reiss motives).
>> Again, the results may seem insatisfactory, but considering aforementioned nature of the texts, problems regarding lack of frequency in our gold data and our novel approach, the results again are somewhat acceptable. Just like for precision, a larger and, crucially, more balanced corpus could potentially improve these results.
<palign="center">
<imgsrc="pictures/Precision_test.png"/>
</p>
> The system scored a macro-averaged precision of roughly 19% for Maslow categories, while scoring only 5.4% for Reiss categories. Yet again the system struggles much more with the classification of the correct Reiss motive due to their fine-grained nature. The possibility to choose from a much wider range of possible classes makes negative impacts on the systems performance. Again, nature of the Reiss motives too negatively influences the system's performance due to aforementioned reasons. However, provided with a much more limited variety of classes to choose from, the system's performance increases heavily, as is visible in the precision scores of Maslow categories. It is interesting to take a quick glance at the scores for each class: As we had expected, classes that are barely seen in the gold data produce poor results due to their lack of generalization. In our case, "physiological needs" scored 0% precision because of this. On the other hand, classes which are more frequent in the gold data score much better results, as visible by the results for the classes "spiritual growth" and "stability". These results confirm our study conjectures about the frequency of certain classes, in which frequent classes scored significantly higher than classes which are barely existent in the gold data. Classes in the middle of this range provide expected results, in which they don't score very high but also not dramatically low.
>> Though the results may again seem utterly dissatisfying and not quite what we had hoped for, it is still somewhat accaptable due to the more difficult and tricky nature of our texts and the approach we had to take. The problems that arise from the lack of gold data for certain classes (i.e. "physiological needs") will have to be mitigated either way. A more balanced distribution of texts could possibly solve this problem, but the content of the corpus at use could make this approach quite difficult, as it was the case.
>> Though the results may again seem utterly dissatisfying and not quite what we had hoped for, it is still somewhat accaptable due to the more difficult and tricky nature of our texts and the novel approach we had to take. The problems that arise from the lack of gold data for certain classes (i.e. "physiological needs") will have to be mitigated either way. A more balanced distribution of texts could possibly solve this problem, but the content of the corpus at use could make this approach quite difficult, as it was the case.
