Merge branch 'master' of gitlab.cl.uni-heidelberg.de:jmrs/caption

**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html)) 

### System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

### CNN

#### Coco API

python setup.py install

```

TODO: coco preprocessing code

#### Retraining

Tensorflow pretrained image classification models available at [https://github.com/tensorflow/models/tree/master/research/slim](https://github.com/tensorflow/models/tree/master/research/slim)

The pretrained model is retrained onto the 80 categories from coco.

##### Singlelabel Model [PDF](https://arxiv.org/pdf/1512.03385.pdf)

##### Singlelabel Model (InceptionV3) [PDF](https://arxiv.org/pdf/1512.03385.pdf

* How to retrain Singlelabel:  

TODO

For training with default hyperparameters you need to pass a directory with all the images.  

The directory has to contain subdirectories which each are named after one of the labels and contain only images with that label.   

```

python3 retrain.py \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/  

```

You can also pass different hyperparameters as arguments:

```

python3 retrain.py \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/ \  

--how_many_training_steps 4000 \  

--flip_left_right True \  

--random_scale 20 \  

--random_brightness 20 \  

--random_crop 20

```

* **Results:**  

[Trained model (default)](/models/singlelabel_default/)  

[Trained model (+Hyperparameters)](/models/singlelabel_hp/)

##### ResNet Model [PDF](https://arxiv.org/pdf/1512.03385.pdf)

##### ResNet Model (singlelabel) [PDF](https://arxiv.org/pdf/1512.03385.pdf)

* How to retrain ResNet:  

The training is almost the same as with the classic singlelabel model.  

Just pass the download link to the ResNet model as hyperparameter.  

Example with tuned hyperparameters:  

```

python3 retrain.py \  

--tfhub_module https://tfhub.dev/google/imagenet/inception_resnet_v2/feature_vector/1 \  

--image_dir /softpro/ss18/caption/cocoapi/train_all_objects/ \  

--how_many_training_steps 4000 \  

--flip_left_right True \  

--random_scale 20 \  

--random_brightness 20 \  

--random_crop 20

```

* **Results:**  

[Trained model (default)](/models/resnet_default/)  

[Trained model (+Hyperparameters)](/models/resnet_hp/)

##### Multilabel Model

##### Multilabel Model (InceptionV3)

* How to retrain multilabel:  

```

bash retrain.sh

```

The retrain.sh calls the version of the retrain.py which is modified for multilabel training.  

It is possible to change the hyperparamters in the retrain.sh.  

Example for retrain.sh with tuned hyperparameters:

```

python retrain.py \

--bottleneck_dir=bottlenecks \

--how_many_training_steps 50000 \

--model_dir=model_dir \

--output_graph=retrained_graph.pb \

--output_labels=retrained_labels.txt \

--summaries_dir=retrain_logs \

--flip_left_right=True \

--random_scale=20 \

--random_brightness=20 \

--random_crop=20 \

--image_dir=/softpro/ss18/caption/cocoapi/multilabel_outer/ 

```

The image directory contains all the training images without them being in subdirectories.  

Additionally you have to open the the retrain.py and change the path of IMAGE_LABELS_DIR and ALL_LABELS_FILE.  

IMAGE_LABELS_DIR is a directory which contains a text file for every image in image_dir describing all labels on that image.  

ALL_LABELS_FILE is a text file that contains all the labels.

* **Results:**  

[Trained model (default)](/models/multilabel_default/)  

[Trained model (+Hyperparameters)](/models/multilabel_hp/)

TODO:

[Demo Notebook]()

## Additional material

TODO: Forschungsplan  

TODO: Spezifiktaionsvortrag  

TODO: Abschlussvortrag

## References

Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015). **Show and tell: A neural image caption generator**. In *Proceedings of the IEEE conference on computer vision and pattern recognition* (pp. 3156-3164).