Merge branch 'master' of gitlab.cl.uni-heidelberg.de:jmrs/caption

# O Caption! My Caption!

![https://xkcd.com/1425/](https://imgs.xkcd.com/comics/tasks.png)

This gitlab contains code for the software project course during Sommersemester 2018.

The Goal is to produce automatic generated captions for images (see [System](#system))

We based our work on the following paper: **Show and tell: A neural image caption generator** [[PDF](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.761.3284&rep=rep1&type=pdf)]

and its successors **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge** [[PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7505636)]

and **Show, attend and tell: Neural image caption generation with visual attention** [[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)].

## Requirements

```

pip install -r requirements.txt

```

## Additional material

Here are the slides (in german) for the two presentations we gave during the semester:

* [Forschungsplan](/Demo/forschungsplan.pdf) - our research plan: possible datasets, outline for the project and schedule of time and workload.

* [Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf) contains our final presentation with demonstrations, results and evaluation.

## System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

## Dataset

We used the [**COCO**](http://cocodataset.org/#explore) 2017 Dataset [[PDF](https://arxiv.org/pdf/1405.0312.pdf)].  

It provides:  

* ~118000 pictures, manually annotated with contained objects

* 80 object categories

* 5 example captions per picture

Example:  

<p float="left">

  <img src="/Diagrams/cat_plain.png" width="420" />

  <img src="/Diagrams/cat_plane.png" width="420" /> 

</p>

Captions:  

* a cat playing inside a cardboard space ship on the floor

* a cat is in a small model plane

* a cat sitting in a little toy plane

* a cat inside of a small toy air plane

* a cat sits inside of a paper airplane toy

## Usage

**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html)) 

### System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

### Requirements

Python3.6 or newer is needed to run the code!

```

pip install -r requirements.txt

```

for coco related requirements see [required input](#reqired-input)

### CNN

#### Visualization (Lucid)

As an additional way to evaluate the CNN output we want to visualize the learned features.

* We used [https://github.com/tensorflow/lucid](https://github.com/tensorflow/lucid) (for creating own model have a look at this [colab-notebook](https://colab.research.google.com/drive/1PPzeZi5sBN2YRlBmKsdvZPbfYtZI-pHl#scrollTo=06q00d8zvwqc))  

* Chosen Visualization style [Activation Grid](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/building-blocks/ActivationGrid.ipynb)

### LSTM

#### Setup

Um das komplette LSTM verwenden zu können sind die mscocotools notwendig. Des weiteren muss man einige Packete auf Python3 installiert haben.

* python 3.6.5^

For using the LSTM it is required to have mscocotools installed.

The following modules are needed to run the code and can be installed with pip3 for Python3.

* python 3.6.5 *

* os *

* division *

* tensorflow *

* matplotlib *

* skimage *

* scipy *

* ipywidgets

* ipywidgets *

To configure the LSTM please edit 'current settings' in [configuration.py](/caption_lib/lstm/configuration.py)

Um das LSTM zu konfigurieren editiert man die "current settings" in der [configuration.py](/caption_lib/lstm/configuration.py)

```

Default trainingsconfig

class default_trainer(object):

    batch_randomization = True

```

Zum starten des LSTMS einfach in den ordner caption_lib navigieren und von dort aus [start.py](/caption_lib/start.py) ausführen.

To start the LSTM navigate to the folder caption_lib and run [start.py](/caption_lib/start.py).

#### Attention

#### visualisation  

| Bleu-4        | 0.141         | 0.140       | 0.309              | **0.363**   |

| SPICE         | 0.092         | 0.089       | 0.182              | **0.213**   |

* We used the provided toolkit from the coco website to evaluate our data, but had to adjust the code to fit the 2017-data-version we used.

* For nearly all metrics (except to CIDEr) we have a difference of 0.1 to 0.2 - which is quite a margin but can be explained by

considering that we extracted only ~7% (8000 out of 118000) of the image data to train the LSTM.  

* More reasons for our lower results: the cnn wasn't fully trained (precision was still low), no extended hyperparameter tuning for LSTM, no beam search implemented

* Another effect which does not show in the evaluation metric is that even though the scores of *Singlelabel* and *Multilabel* are quite similar, 

we observed some difference when we looked at it manually. *Singlelabel* often produced more gramatically sound sentences, but not fitting to the objects in the image, 

while *Multilabel* recognized the objects better, but had slightly more errors in sentence structure (for an example see [Example captionresult](#example-captionresult)).

* It has been shown that automatic metrics don't work particular well for that particular task.  

  For reference see: 

    -  *Why we need new evaluation metrics for nlg* [[PDF](https://arxiv.org/pdf/1707.06875.pdf)]

    -  *The price of debiasing automatic metrics in natural language evaluation* [[PDF](https://arxiv.org/pdf/1807.02202.pdf)]

## Demo

[Demo Notebook](/caption_lib/LSTM.ipynb)

## Additional material

[Forschungsplan](/Demo/forschungsplan.pdf)  

[Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf)  

## Conclusion

We used several [pretrained image classification networks](#retraining) and retrained them onto the categories from the [coco dataset](#dataset). Here the *Multilabel* approach proved 

best for capturing the content of the images (and could have turned out to be overall best if trained for longer) and the *Singlelabel* approach was more fitted onto the language model.

To gain a grasp on the learned features we utilized [lucid](#visualization-lucid) by visualizing various activation grids, but were confronted with several problems and an output 

which is stunning but sometimes hard to interpret.  

After [extracting the feature vectores](#extraction) for a subset of our images in our dataset (performance issues!), we used this as input for our [LSTM](#lstm) and trained it. 

At this step we implemented [attention](#attention), but ran into problems because we decided earlier to extract the 2048 dimensional feature vector, which could not backtrack the 

input image proportions that varied to a high degree.   

This is reflected in our [evaluation](#evaluation) scores. As automatic scores are unreliable for that task we would have loved to do a manual evaluation - but that is left for 

future work, as well as proper implementation of attention, beam search, parameter tuning for the LSTM, extraction of all feature vectors from the dataset and extended 

training time for the CNN.

## References

He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).  

[PDF](https://arxiv.org/pdf/1512.03385.pdf)

Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., ... & Zitnick, C. L. (2014, September). **Microsoft coco: Common objects in context**. In *European conference on computer vision* (pp. 740-755). Springer, Cham.  

[PDF](https://arxiv.org/pdf/1405.0312.pdf)

Novikova, J., Dušek, O., Curry, A. C., & Rieser, V. (2017). **Why we need new evaluation metrics for nlg**. *arXiv preprint arXiv:1707.06875*.  

[PDF](https://arxiv.org/pdf/1707.06875.pdf)

Chaganty, A. T., Mussman, S., & Liang, P. (2018). **The price of debiasing automatic metrics in natural language evaluation**. *arXiv preprint arXiv:1807.02202*.  

[PDF](https://arxiv.org/pdf/1807.02202.pdf)

for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki

## LICENSE

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details

pycocotools==2.0

scikit-image==0.13.1

matplotlib==2.1.1

lucid==0.1.0

 No newline at end of file