Tensorflow pretrained image classification models available at [https://github.com/tensorflow/models/tree/master/research/slim](https://github.com/tensorflow/models/tree/master/research/slim)
The pretrained model is retrained onto the 80 categories from coco.
##### Singlelabel Model (InceptionV3) [PDF](https://arxiv.org/pdf/1512.03385.pdf
##### Singlelabel Model (InceptionV3) [PDF](https://arxiv.org/pdf/1512.03385.pdf)
* How to retrain Singlelabel:
For training with default hyperparameters you need to pass a directory with all the images.
@@ -152,6 +152,68 @@ See [example notebook](/lucid/Visualization - singlelabel.ipynb)
### LSTM
#### Setup
Um das komplette LSTM verwenden zu können sind die mscocotools notwendig. Des weiteren muss man einige Packete auf Python3 installiert haben.
* python 3.6.5^
* os *
* division *
* tensorflow *
* collections *
* json *
* datetime *
* time *
* numpy *
* pickle *
* re *
* matplotlib *
* skimage *
* scipy *
* ipywidgets
Um das LSTM zu konfigurieren editiert man die "current settings" in der [configuration.py](/caption_lib/lstm/configuration.py)
```
Default trainingsconfig
class default_trainer(object):
train_on = False
train_fresh_and_override = True
verbose = False
num_layers = 1
num_epochs = 60
num_steps = 35
vector_size = 2048
batch_size = 20
dropout_on = True
dropout = 0.5
forget_bias = 0.4
hidden_size = 1024
learning_rate = 0.001
batch_randomization = True
```
Zum starten des LSTMS einfach in den ordner caption_lib navigieren und von dort aus [start.py](/caption_lib/start.py) ausführen.
#### Attention
#### visualisation
Here we can see that the attention reacts on specific words and if it does not react at all then we can assume that there was a learning only from the captions not from the vector we provided.
#### Bad visualisation
The attention can't be mapped correctly because we do not know the dimensions of the pictures and how the CNN changed the data during the learning.
We see here that "surfer" and "riding" is in the same area but we do not know how to correctly map it on the surfer in the picture.
***Problems:**
- Hard to understand with zero experience. Also setup is not easy to grasp.
- early information on how to use attention after a two model approach with CNN and LSTM is hard to understand
- Mapping of attention and picture data
#### Example captionresult
singlelabel:
“A man is holding a toothbrush in his mouth with his mouth open . <END>”
multilabel:
“A woman holding a video game controller in a living room . <END>”
### Evaluation
@@ -166,6 +228,7 @@ See [example notebook](/lucid/Visualization - singlelabel.ipynb)
| Bleu-4 | 0.141 | 0.140 | 0.309 | **0.363** |
| SPICE | 0.092 | 0.089 | 0.182 | **0.213** |
## Demo
### CNN
@@ -174,14 +237,13 @@ See [example notebook](/lucid/Visualization - singlelabel.ipynb)
