This gitlab contains code for the software project course during Sommersemester 2018.
The Goal is to produce automatic generated captions for images (see [System](#system))
We based our work on the following paper: **Show and tell: A neural image caption generator** and its successor.
We based our work on the following paper: **Show and tell: A neural image caption generator**[PDF](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.761.3284&rep=rep1&type=pdf)
and its successors **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**[PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7505636)
and **Show, attend and tell: Neural image caption generation with visual attention**[PDF](http://proceedings.mlr.press/v37/xuc15.pdf).
## Additional material
@@ -37,29 +40,6 @@ Captions:
* a cat sits inside of a paper airplane toy
## References
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017). **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**. *IEEE transactions on pattern analysis and machine intelligence*, 39(4), 652-663.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). **Show, attend and tell: Neural image caption generation with visual attention**. In **International conference on machine learning** (pp. 2048-2057).
[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). **Rethinking the inception architecture for computer vision**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).
[PDF](https://arxiv.org/pdf/1512.00567.pdf)
He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).
[PDF](https://arxiv.org/pdf/1512.03385.pdf)
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014, September). **Microsoft coco: Common objects in context**. In *European conference on computer vision* (pp. 740-755). Springer, Cham.
[PDF](https://arxiv.org/pdf/1405.0312.pdf)
for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki
## Usage
**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html))
@@ -378,6 +358,34 @@ while *Multilabel* recognized the objects better, but had slightly more errors i
## Conclusion
## References
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017). **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**. *IEEE transactions on pattern analysis and machine intelligence*, 39(4), 652-663.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). **Show, attend and tell: Neural image caption generation with visual attention**. In **International conference on machine learning** (pp. 2048-2057).
[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). **Rethinking the inception architecture for computer vision**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).
[PDF](https://arxiv.org/pdf/1512.00567.pdf)
He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).
[PDF](https://arxiv.org/pdf/1512.03385.pdf)
Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014, September). **Microsoft coco: Common objects in context**. In *European conference on computer vision* (pp. 740-755). Springer, Cham.
[PDF](https://arxiv.org/pdf/1405.0312.pdf)
Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017). **Why we need new evaluation metrics for nlg**. *arXiv preprint arXiv:1707.06875*.
[PDF](https://arxiv.org/pdf/1707.06875.pdf)
Chaganty, A. T., Mussman, S., & Liang, P. (2018). **The price of debiasing automatic metrics in natural language evaluation**. *arXiv preprint arXiv:1807.02202*.
[PDF](https://arxiv.org/pdf/1807.02202.pdf)
for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki
