Merge branch 'master' of https://gitlab.cl.uni-heidelberg.de/jmrs/caption

![https://xkcd.com/1425/](https://imgs.xkcd.com/comics/tasks.png)

## Requirements

```

pip install -r requirements.txt

```

## Usage

**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html)) 

### System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

### CNN

#### Coco API

We used [https://github.com/cocodataset/cocoapi](https://github.com/cocodataset/cocoapi)  

Installation:  

```

git clone https://github.com/cocodataset/cocoapi.git

cd cocoapi/PythonAPI

python setup.py install

```

TODO: coco preprocessing code

#### Retraining

Tensorflow pretrained image classification models available at [https://github.com/tensorflow/models/tree/master/research/slim](https://github.com/tensorflow/models/tree/master/research/slim)

The pretrained model is retrained onto the 80 categories from coco.

##### Singlelabel Model (InceptionV3) [PDF](https://arxiv.org/pdf/1512.03385.pdf

* How to retrain Singlelabel:  

For training with default hyperparameters you need to pass a directory with all the images.  

The directory has to contain subdirectories which each are named after one of the labels and contain only images with that label.   

```

python3 retrain.py \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/  

```

You can also pass different hyperparameters as arguments:

```

python3 retrain.py \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/ \  

--how_many_training_steps 4000 \  

--flip_left_right True \  

--random_scale 20 \  

--random_brightness 20 \  

--random_crop 20

```

* **Results:**  

[Trained model (default)](/models/singlelabel_default/)  

[Trained model (+Hyperparameters)](/models/singlelabel_hp/)

##### ResNet Model (singlelabel) [PDF](https://arxiv.org/pdf/1512.03385.pdf)

* How to retrain ResNet:  

The training is almost the same as with the classic singlelabel model.  

Just pass the download link to the ResNet model as hyperparameter.  

Example with tuned hyperparameters:  

```

python3 retrain.py \  

--tfhub_module https://tfhub.dev/google/imagenet/inception_resnet_v2/feature_vector/1 \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/ \  

--how_many_training_steps 4000 \  

--flip_left_right True \  

--random_scale 20 \  

--random_brightness 20 \  

--random_crop 20

```

* **Results:**  

[Trained model (default)](/models/resnet_default/)  

[Trained model (+Hyperparameters)](/models/resnet_hp/)

##### Multilabel Model (InceptionV3)

* How to retrain multilabel:  

```

bash retrain.sh

```

The retrain.sh calls the version of the retrain.py which is modified for multilabel training.  

It is possible to change the hyperparamters in the retrain.sh.  

Example for retrain.sh with tuned hyperparameters:

```

python retrain.py \

--bottleneck_dir=bottlenecks \

--how_many_training_steps 50000 \

--model_dir=model_dir \

--output_graph=retrained_graph.pb \

--output_labels=retrained_labels.txt \

--summaries_dir=retrain_logs \

--flip_left_right=True \

--random_scale=20 \

--random_brightness=20 \

--random_crop=20 \

--image_dir=/softpro/ss18/caption/cocoapi/multilabel_outer/ 

```

The image directory contains all the training images without them being in subdirectories.  

Additionally you have to open the the retrain.py and change the path of IMAGE_LABELS_DIR and ALL_LABELS_FILE.  

IMAGE_LABELS_DIR is a directory which contains a text file for every image in image_dir describing all labels on that image.  

ALL_LABELS_FILE is a text file that contains all the labels.

* **Results:**  

[Trained model (default)](/models/multilabel_default/)  

[Trained model (+Hyperparameters)](/models/multilabel_hp/)

#### Extraction

* **Restriction:**  

  Because extracting all vectors from images took too long we limited the dataset to 100 pictures per category, resulting in 8000 vectors total (which are used as input for the LSTM). We expect that this decision will get us inferior end results, but had to be done to stay within time limits.

* to create an equal distribution without intersecting pictures we used the following [script](cnn/preprocessing/100.py):

```

python 100.py

```

* The final distribution can be seen in the following chart:

  ![distribution of categories](/Diagrams/sb_category_frequencies.png)

#### Visualization (Lucid)

* We used [https://github.com/tensorflow/lucid](https://github.com/tensorflow/lucid) (for creating own model have a look at this [colab-notebook](https://colab.research.google.com/drive/1PPzeZi5sBN2YRlBmKsdvZPbfYtZI-pHl#scrollTo=06q00d8zvwqc))  

* Chosen Visualization style [Activation Grid](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/building-blocks/ActivationGrid.ipynb)

![ActivationGrid](https://storage.googleapis.com/lucid-static/building-blocks/notebook_heroes/activation-grid.jpeg)

* **Workflow:**  

See [example notebook](/lucid/Visualization - singlelabel.ipynb)

* **Results:**  

[take a trip](/visual/README.md)

* **Problems:**

  - only limited selection of layers possible (early layers not available due to lack of memory, later layers not available due to getting "kernel died" message from jupyter notebook)

  - no discernable features on some images

  - if underlying picture is not known before =>  hard to interpret features (if underlying picture is known => prone to overinterpretation and bias)

  - hard to debug

### LSTM

### Evaluation

| Measure       | Singlelabel   | Multilabel  | Show & Tell (2015) | SOTA (2017) |

| -------------:|:-------------:|:-----------:|:------------------:|:-----------:|

| CIDEr         | 0.366         | 0.375       | 0.943              | **1.179**   |

| Meteor        | 0.141         | 0.140       | 0.254              | **0.277**   |

| Rouge-L       | 0.402         | 0.407       | 0.530              | **0.573**   |

| Bleu-1        | 0.514         | 0.511       | 0.713              | **0.795**   |

| Bleu-2        | 0.338         | 0.336       | 0.542              | **0.635**   |

| Bleu-3        | 0.217         | 0.217       | 0.407              | **0.485**   |

| Bleu-4        | 0.141         | 0.140       | 0.309              | **0.363**   |

| SPICE         | 0.092         | 0.089       | 0.182              | **0.213**   |

## Demo

### CNN

[Demo Notebook](/Demo/CNN.ipynb)

### LSTM

[Demo Notebook](/caption_lib/LSTM.ipynb)

## Additional material

Forschungsplan

[Forschungsplan](/Demo/forschungsplan.pdf)

TODO: Spezifiktaionsvortrag  

[Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf.pdf)

## References

Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).  

[PDF](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.761.3284&rep=rep1&type=pdf)

Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2017). **Show and tell: Lessons learned from the 2015 mscoco image captioning challenge**. *IEEE transactions on pattern analysis and machine intelligence*, 39(4), 652-663.  

[PDF](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7505636)

Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., ... & Bengio, Y. (2015, June). **Show, attend and tell: Neural image caption generation with visual attention**. In **International conference on machine learning** (pp. 2048-2057).  

[PDF](http://proceedings.mlr.press/v37/xuc15.pdf)

Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). **Rethinking the inception architecture for computer vision**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 2818-2826).  

[PDF](https://arxiv.org/pdf/1512.00567.pdf)

He, K., Zhang, X., Ren, S., & Sun, J. (2016). **Deep residual learning for image recognition**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 770-778).  

[PDF](https://arxiv.org/pdf/1512.03385.pdf)

for even more check the [resources page](https://gitlab.cl.uni-heidelberg.de/jmrs/caption/wikis/organization/resources) from our wiki

## LICENSE

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details