evaluator added

*.pyc

Microsoft COCO Caption Evaluation

===================

Evaluation codes for MS COCO caption generation.

## Requirements ##

- java 1.8.0

- python 2.7

## Files ##

./

- cocoEvalCapDemo.py (demo script)

./annotation

- captions_val2014.json (MS COCO 2014 caption validation set)

- Visit MS COCO [download](http://mscoco.org/dataset/#download) page for more details.

./results

- captions_val2014_fakecap_results.json (an example of fake results for running demo)

- Visit MS COCO [format](http://mscoco.org/dataset/#format) page for more details.

./pycocoevalcap: The folder where all evaluation codes are stored.

- evals.py: The file includes COCOEavlCap class that can be used to evaluate results on COCO.

- tokenizer: Python wrapper of Stanford CoreNLP PTBTokenizer

- bleu: Bleu evalutation codes

- meteor: Meteor evaluation codes

- rouge: Rouge-L evaluation codes

- cider: CIDEr evaluation codes

- spice: SPICE evaluation codes

## Setup ##

- You will first need to download the [Stanford CoreNLP 3.6.0](http://stanfordnlp.github.io/CoreNLP/index.html) code and models for use by SPICE. To do this, run:

    ./get_stanford_models.sh

- Note: SPICE will try to create a cache of parsed sentences in ./pycocoevalcap/spice/cache/. This dramatically speeds up repeated evaluations. The cache directory can be moved by setting 'CACHE_DIR' in ./pycocoevalcap/spice. In the same file, caching can be turned off by removing the '-cache' argument to 'spice_cmd'. 

## References ##

- [Microsoft COCO Captions: Data Collection and Evaluation Server](http://arxiv.org/abs/1504.00325)

- PTBTokenizer: We use the [Stanford Tokenizer](http://nlp.stanford.edu/software/tokenizer.shtml) which is included in [Stanford CoreNLP 3.4.1](http://nlp.stanford.edu/software/corenlp.shtml).

- BLEU: [BLEU: a Method for Automatic Evaluation of Machine Translation](http://www.aclweb.org/anthology/P02-1040.pdf)

- Meteor: [Project page](http://www.cs.cmu.edu/~alavie/METEOR/) with related publications. We use the latest version (1.5) of the [Code](https://github.com/mjdenkowski/meteor). Changes have been made to the source code to properly aggreate the statistics for the entire corpus.

- Rouge-L: [ROUGE: A Package for Automatic Evaluation of Summaries](http://anthology.aclweb.org/W/W04/W04-1013.pdf)

- CIDEr: [CIDEr: Consensus-based Image Description Evaluation](http://arxiv.org/pdf/1411.5726.pdf)

- SPICE: [SPICE: Semantic Propositional Image Caption Evaluation](https://arxiv.org/abs/1607.08822)

## Developers ##

- Xinlei Chen (CMU)

- Hao Fang (University of Washington)

- Tsung-Yi Lin (Cornell)

- Ramakrishna Vedantam (Virgina Tech)

## Acknowledgement ##

- David Chiang (University of Norte Dame)

- Michael Denkowski (CMU)

- Alexander Rush (Harvard University)

#from textatistic import Textatistic

from pycocotools.coco import COCO

from pycocoevalcap.eval import COCOEvalCap

import matplotlib.pyplot as plt

#import skimage.io as io

import pylab

pylab.rcParams['figure.figsize'] = (10.0, 8.0)

import json

from json import encoder

encoder.FLOAT_REPR = lambda o: format(o, '.3f')

# set up file names and pathes

dataDir='.'

dataType='val2014' # name of annotation and result file

algName = 'fakecap'# name of dummy captions file

annFile='%s/annotations/captions_%s.json'%(dataDir,dataType)

subtypes=['results', 'evalImgs', 'eval']

[resFile, evalImgsFile, evalFile]= \

['./results/captions_%s_%s_%s.json'%(dataType,algName,subtype) for subtype in subtypes]

# create coco object and cocoRes object

coco = COCO(annFile)

cocoRes = coco.loadRes(resFile)

# create cocoEval object by taking coco and cocoRes

cocoEval = COCOEvalCap(coco, cocoRes)

# evaluate on a subset of images by setting

# cocoEval.params['image_id'] = cocoRes.getImgIds()

# please remove this line when evaluating the full validation set

cocoEval.params['image_id'] = cocoRes.getImgIds()

# evaluate results

# SPICE will take a few minutes the first time, but speeds up due to caching

cocoEval.evaluate()

# print output evaluation scores

for metric, score in cocoEval.eval.items():

    print '%s: %.3f'%(metric, score)

"""

# compute flesch score

sentences = u''

for image in cocoRes.dataset["annotations"]:

	sentences += image["caption"] + '. '

print ("SENTENCES:", sentences)

flesch = Textatistic(sentences).flesch_score

print ("Flesch score: ",flesch)

"""