This gitlab contains code for the software project course during Sommersemester 2018.
The Goal is to produce automatic generated captions for images (see [System](#system))
We based our work on the following paper: **Show and tell: A neural image caption generator** and its successor.
## Additional material
@@ -16,6 +17,49 @@ Here are the slides (in german) for the two presentations we gave during the sem
## Dataset
We used the [**COCO**](http://cocodataset.org/#explore) Dataset [[PDF](https://arxiv.org/pdf/1405.0312.pdf)].
It provides:
* ~118000 pictures, manually annotated with contained objects
* 80 object categories
* 5 example captions per picture
Example:
Captions:
* a cat playing inside a cardboard space ship on the floor
* a cat is in a small model plane
* a cat sitting in a little toy plane
* a cat inside of a small toy air plane
* a cat sits inside of a paper airplane toy
## References
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017). **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**. *IEEE transactions on pattern analysis and machine intelligence*, 39(4), 652-663.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). **Show, attend and tell: Neural image caption generation with visual attention**. In **International conference on machine learning** (pp. 2048-2057).
[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). **Rethinking the inception architecture for computer vision**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).
[PDF](https://arxiv.org/pdf/1512.00567.pdf)
He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).
[PDF](https://arxiv.org/pdf/1512.03385.pdf)
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014, September). **Microsoft coco: Common objects in context**. In *European conference on computer vision* (pp. 740-755). Springer, Cham.
[PDF](https://arxiv.org/pdf/1405.0312.pdf)
for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki
## Usage
**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html))
@@ -316,8 +360,10 @@ considering that we used only ~7% of the trainings data for the LSTM.
* Another effect which does not show in the evaluation metric is that even though the scores of *Singlelabel* and *Multilabel* are quite similar,
we observed some difference when we looked at it manually. *Singlelabel* often produced more gramatically sound sentences, but not fitting to the objects in the image,
while *Multilabel* recognized the objects better, but had slightly more errors in sentence structure (for an example see [Example captionresult](#example-captionresult)).
* It has been shown that automatic metrics don't work so well for that particular task. For reference see
"Why we need new evaluation metrics for nlg" [PDF](https://arxiv.org/pdf/1707.06875.pdf) and "The price of debiasing automatic metrics in natural language evaluation" [PDF](https://arxiv.org/pdf/1807.02202.pdf)
* It has been shown that automatic metrics don't work particular well for that particular task.
For reference see:
-*Why we need new evaluation metrics for nlg*[[PDF](https://arxiv.org/pdf/1707.06875.pdf)]
-*The price of debiasing automatic metrics in natural language evaluation*[[PDF](https://arxiv.org/pdf/1807.02202.pdf)]
## Demo
@@ -333,25 +379,6 @@ while *Multilabel* recognized the objects better, but had slightly more errors i
## Conclusion
## References
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017). **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**. *IEEE transactions on pattern analysis and machine intelligence*, 39(4), 652-663.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). **Show, attend and tell: Neural image caption generation with visual attention**. In **International conference on machine learning** (pp. 2048-2057).
[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). **Rethinking the inception architecture for computer vision**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).
[PDF](https://arxiv.org/pdf/1512.00567.pdf)
He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).
[PDF](https://arxiv.org/pdf/1512.03385.pdf)
for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki
## LICENSE
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details
