Here we can see that the attention reacts on specific words and if it does not react at all then we can assume that there was a learning only from the captions not from the vector we provided.
## Bad visualisation
##### Bad visualisation
The attention can't be mapped correctly because we do not know the dimensions of the pictures and how the CNN changed the data during the learning.
We see here that "surfer" and "riding" is in the same area but we do not know how to correctly map it on the surfer in the picture.
@@ -222,13 +208,25 @@ We see here that "surfer" and "riding" is in the same area but we do not know ho
- early information on how to use attention after a two model approach with CNN and LSTM is hard to understand
- Mapping of attention and picture data
### Example captionresult
##### Example captionresult
singlelabel:
“A man is holding a toothbrush in his mouth with his mouth open . <END>”
multilabel:
“A woman holding a video game controller in a living room . <END>”
### Evaluation
| Measure | Singlelabel | Multilabel | Show & Tell (2015) | SOTA (2017) |
