Commit 53037791 authored by pirapakaran's avatar pirapakaran
Browse files

Update project_report.md

parent c9ee2fe4
Loading
Loading
Loading
Loading
+6 −0
Original line number Diff line number Diff line
@@ -84,6 +84,12 @@ After the human needs have been successfully assigned for all essays, we evaluat
  <img src="pictures/Matrix_4_test.png" />
</p>

#### General Comments on Evaluation

As mentioned before, all evalution metrics confirmed our initial study conjectures.
1. Performance of Maslow category classification and Reiss motive classification

    During the annotation phase of our gold data (and especially during the calculation of the Inter-Annotator-Agreement), we had already noticed that assigning the proper and most fitting class for Maslow and especially for Reiss was not as straightforwards as expected. The coarse-grained nature of Maslow categories provided us much less options to choose from, which in turn made our annotation itself more confident. Reiss motives, however, were much more difficult approach as they were much more fine-grained and provided such a wide range of possible options that, at times, it became very difficult us to choose the right motive. In addition to that, two or even more motives would have been equally fitting for one essay which increased the difficulty of the task. This notion was first supported by our Inter-Annotator-Agreement values, in which the annotations for Maslow categories showed much higher agreement between us than for those of the Reiss motives. Unsurprisingly, the system matched our pattern and intution and scored significantly higher for the coarse-grained Maslow categories than for the fine-grained Reiss motives. The conclusion which we can draw from this is that, given two groups of classes to choose from, the system will perform better on the coarse-grained classes and show be less effective on fine-grained classes. For future research, it could be of special interest to shift the focus to the correct classification of fine-grained classes and categories, as fulfillment of this task could also potentially entail the correct classification of the coarse-grained classes.