Merge branch 'master' of gitlab.cl.uni-heidelberg.de:jmrs/caption

python setup.py install

```

TODO: coco preprocessing code

[Coco Preprocessing Code](/preprocessing/coco_preprocessing/)

#### Reqired Input

The following folder structure is required to run the retraining process.

#### Before Preprocessing:

Assumption: Your pictures from coco is in jpg files in the `cocoapi/images/train/` subfolder and the coco annotation files are in the `cocoapi/annotations/train/` subfolder.

```

cocoapi/                    

    images/

        train/

            *.jpg

    annotations/

        train/

            instances_train2017.json

            captions_train2017.json

```

#### After Preprocessing:

* [exract_multilabel_pics.py](/preprocessing/coco_preprocessing/extract_multilabel_pics.py) will create the following folders and files (images for multilabel training):

    ```

    cocoapi/

        multilabel_outer/

            multilabel_images/

                *.jpg

    ```

* [categories_of_pictures.py](/preprocessing/coco_preprocessing/categories_of_pictures.py) will create the following folders and files (labels for multilabel training):

    ```

    cocoapi/

        images_txt/

            *.txt

    ```

* [categories_labels.py](/preprocessing/coco_preprocessing/categories_labels.py) will create the following file (categories of coco):

    ```

    cocoapi/

        labels.txt

    ```

#### Retraining

* **Restriction:**  

  Because extracting all vectors from images took too long we limited the dataset to 100 pictures per category, resulting in 8000 vectors total (which are used as input for the LSTM). We expect that this decision will get us inferior end results, but had to be done to stay within time limits.

* to create an equal distribution without intersecting pictures we used the following [script](cnn/preprocessing/100.py):

* to create an equal distribution without intersecting pictures we used the following [script](preprocessing/coco_preprocessing/100.py):

```

python 100.py

```

* **Workflow:**  

See [example notebook](/lucid/Visualization - singlelabel.ipynb)  

Note that different models have different architectures and layers.  

To extract the layers of a given model we used [this](/preprocessing/graph_reader.py) script.

* **Results:**  

[take a trip](/visual/README.md)

## Additional material

[Forschungsplan](/Demo/forschungsplan.pdf)  

TODO: Spezifiktaionsvortrag  

[Abschlussvortrag](/Demo/abschlussvortrag_softwareprojekt.pdf)  

## References

import os

path = '/softpro/ss18/caption/cocoapi/images_txt/'

coco = COCO('/softpro/ss18/caption/cocoapi/annotations/train/instances_train2017.json')

coco = COCO('../../cocoapi/annotations/train/instances_train2017.json')

cats = coco.loadCats(coco.getCatIds())

nms = [cat['name'] for cat in cats]

with open('labels.txt', 'w') as handle:

with open('../../cocoapi/labels.txt', 'w') as handle:

    for category in nms:

        out = category + '\n'

        handle.write(out)

import os

path = '/softpro/ss18/caption/cocoapi/images_txt/'

coco = COCO('/softpro/ss18/caption/cocoapi/annotations/train/instances_train2017.json')

path = '../../cocoapi/images_txt/'

if not os.path.exists(path):

    print(f'Folder images_txt for image labels not existing.\nCreatue folder: {path}')

    os.mkdir(path)

coco = COCO('../../cocoapi/annotations/train/instances_train2017.json')

cats = coco.loadCats(coco.getCatIds())

nms = [cat['name'] for cat in cats]

coco = COCO(

    '/softpro/ss18/caption/cocoapi/annotations/train/instances_train2017.json')

    '../../cocoapi/annotations/train/instances_train2017.json')

cats = coco.loadCats(coco.getCatIds())

nms = [cat['name'] for cat in cats]

path = '../../cocoapi/multilabel_outer/'

subfolder = path + 'multilabel_images/'

if not os.path.exists(path):

    print(f'Folder multilabel not existing.\nCreatue folder: {path}')

    os.mkdir(path)

    print(f'Create Subfolder: {subfolder}')

    os.mkdir(subfolder)

for category in nms:

    print(category)

    catIds = coco.getCatIds(catNms=[category])

    imgIds = coco.getImgIds(catIds=catIds)

    for pic in imgIds:

        name = coco.loadImgs(pic)[0]['file_name']

        shutil.copy2('images/train/' + name, 'multilabel_images/' + name)

        shutil.copy2('../../cocoapi/images/train/' + name, subfolder + name)