Update README.md

![https://xkcd.com/1425/](https://imgs.xkcd.com/comics/tasks.png)

## Requirements

This gitlab contains code for the software project course during Sommersemester 2018.

The Goal is to produce automatic generated captions for images (see [System](#system))

```

pip install -r requirements.txt

```

## Additional material

for coco related requirements see [required input](#reqired-input)

Here are the slides (in german) for the two presentations we gave during the semester:

* [Forschungsplan](/Demo/forschungsplan.pdf) - our research plan: possible datasets, outline for the project and schedule of time and workload.

* [Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf) contains our final presentation with demonstrations, results and evaluation.

## System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

## Usage

**Note:** viewing jupyter-notebooks on git may lack some features (not showing output or choosing options from [ipywidget](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html)) 

### System

![Pipeline](/Diagrams/Highlevel_Diagram.png)

### Requirements

Python3.6 or newer is needed to run the code!

```

pip install -r requirements.txt

```

for coco related requirements see [required input](#reqired-input)

### CNN

#### Visualization (Lucid)

As an additional way to evaluate the CNN output we want to visualize the learned features.

* We used [https://github.com/tensorflow/lucid](https://github.com/tensorflow/lucid) (for creating own model have a look at this [colab-notebook](https://colab.research.google.com/drive/1PPzeZi5sBN2YRlBmKsdvZPbfYtZI-pHl#scrollTo=06q00d8zvwqc))  

* Chosen Visualization style [Activation Grid](https://colab.research.google.com/github/tensorflow/lucid/blob/master/notebooks/building-blocks/ActivationGrid.ipynb)

### LSTM

#### Setup

Um das komplette LSTM verwenden zu können sind die mscocotools notwendig. Des weiteren muss man einige Packete auf Python3 installiert haben.

* python 3.6.5^

For using the LSTM it is required to have mscocotools installed.

The following modules are needed to run the code and can be installed with pip3 for Python3.

* python 3.6.5 *

* os *

* division *

* tensorflow *

* matplotlib *

* skimage *

* scipy *

* ipywidgets

* ipywidgets *

To configure the LSTM please edit 'current settings' in [configuration.py](/caption_lib/lstm/configuration.py)

Um das LSTM zu konfigurieren editiert man die "current settings" in der [configuration.py](/caption_lib/lstm/configuration.py)

```

Default trainingsconfig

class default_trainer(object):

    batch_randomization = True

```

Zum starten des LSTMS einfach in den ordner caption_lib navigieren und von dort aus [start.py](/caption_lib/start.py) ausführen.

To start the LSTM navigate to the folder caption_lib and run [start.py](/caption_lib/start.py).

#### Attention

#### visualisation  

| Bleu-4        | 0.141         | 0.140       | 0.309              | **0.363**   |

| SPICE         | 0.092         | 0.089       | 0.182              | **0.213**   |

* For nearly all metrics (except to CIDEr) we have a difference of 0.1 to 0.2 - which is quite a margin but can be explained by

considering that we used only ~7% of the trainings data for the LSTM.  

* More reasons for our lower results: the cnn wasn't fully trained (precision was still low), no extended hyperparameter tuning for LSTM, no beam search implemented

* Another effect which does not show in the evaluation metric is that even though the scores of *Singlelabel* and *Multilabel* are quite similar, 

we observed some difference when we looked at it manually. *Singlelabel* often produced more gramatically sound sentences, but not fitting to the objects in the image, 

while *Multilabel* recognized the objects better, but had slightly more errors in sentence structure (for an example see [Example captionresult](#example-captionresult)).

## Demo

[Demo Notebook](/caption_lib/LSTM.ipynb)

## Additional material

[Forschungsplan](/Demo/forschungsplan.pdf)  

[Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf)  

## Conclusion

## References