Evaluate with this.

================================================================================

 EVALUATION TOOL FOR SEMEVAL-2013 TASK #11:

 Evaluating Word Sense Induction & Disambiguation within An End-User Application

 http://www.cs.york.ac.uk/semeval-2013/task11/

================================================================================

The aim of this task is to provide a framework for the objective evaluation and 

comparison of Word Sense Disambiguation and Induction algorithms in an end-user 

application, namely Web Search Result Clustering.

Web Search Result Clustering is a task consisting of grouping into clusters 

the snippet results returned by a search engine for an input query. Results 

in a given cluster are assumed to be semantically related to each other and 

each cluster is expected to  represent a specific meaning of the input query.

A Word Sense Induction (WSI) system will be asked to identify the meaning of 

the input query and cluster the snippets into semantically-related groups 

according to their meanings. Instead, a Word Sense Disambiguation (WSD) 

system will be requested to sense-tag the above snippets with the appropriate 

senses of the input query and this, again, will implicitly result in a 

clustering of snippets (i.e., one cluster per sense).

WSD and WSI systems will then be evaluated in an end-user application, i.e., 

according to their ability to diversify the search results for the input query. 

This evaluation scheme, previously proposed for WSI by Navigli and Crisafulli 

(2010) and Di Marco and Navigli (2013), is extended here to WSD and WSI systems 

and is aimed at overcoming the limitations of in vitro evaluations. In fact, 

the quality of the output clusters will be assessed in terms of their ability 

to diversify the snippets across the query meanings.

No training data will be provided.

---------------------------------------

IMPORTANT CONSTRAINTS ABOUT THE SYSTEMS

---------------------------------------

Since both WSI and WSD systems can participate, we will perform two separate 

comparative evaluations: one involving sense induction systems, the other 

involving disambiguation systems. 

The following constraints apply:

- WSI systems can ONLY use RAW corpora for inducing the senses of a query; 

  no sense annotated corpus, existing sense inventory or lexical resource (such 

  as WordNet or BabelNet) is allowed.

- WSD systems can use ANY kind of information, as declared by the participants.

  For instance, supervised systems will use training data, knowledge-rich 

  systems will use resources like WordNet, BabelNet, etc.

Since search results come with URL, title and text snippet, systems can exploit

any of this information for performing the induction/disambiguation task.

If systems want to exploit the URL for retrieving additional information, they

will have to declare this at submission time.

All the information about the submitted systems (such as corpora, resources, URL, 

web page, etc. used by the system) will be reported in the task paper.

------------------

ABOUT THIS PACKAGE

------------------

This package contains the evaluation tool for the Semeval-2013 task #11. 

The tool performs:

1) an intrinsic evaluation in terms of Adjusted RandIndex, Jaccard Index and F1 measure 

2) an extrinsic evaluation in terms of the degree of Web search snippet diversification

   calculated as Subtopic Recall@K and Precision@r

The above measures are calculated as explained in the following reference paper:

A. Di Marco, R. Navigli. Clustering and Diversifying Web Search Results with Graph-Based Word Sense Induction. Computational Linguistics, 39(4), MIT Press, 2013.

--------

CONTENTS

--------

This package contains the following main components:

  README.txt   	      # this file

  datasets            # folder containing the MORESQUE development query dataset together with its gold standard assignments

                      # NOTE: this is NOT the test dataset (the test dataset will, instead, be used for evaluating the participating systems)

  clustering_example  # folder containing an example of clustering output produced by a WSI algorithm

  config	      # configuration folder

  WSI-Evaluator.jar   # WSI evaluation system

------------

INSTALLATION

------------

    # unpack the archive

    tar xvfz semeval-2013_task11_evaluator.tar.gz

---------------------------------

USING WSI-Evaluator.jar

---------------------------------

The evaluator needs as input:

1) a query dataset together with gold standard assignments, e.g. datasets/MORESQUE 

   (obtained from http://lcl.uniroma1.it/moresque/, and already included in the package)

Note that the MORESQUE dataset is provided for development purposes, but 

a new dataset of ambiguous queries will be provided before the start of the competition.

2) the output of your Word Sense Induction/Disambiguation algorithm applied to the snippets returned 

   for each query. For an example, refer to the clustering_example folder.

To run the evaluator do the following:

    # run

    java -jar WSI-Evaluator.jar <query dataset folder> <WSI algorithm output file>

For instance, the following command evaluates the clustering example provided

in this package on the MORESQUE development dataset:

    # output format from SemEval-2010 Word Sense Induction & Disambiguation Task #14 

    java -jar WSI-Evaluator.jar datasets/MORESQUE clustering_example/MORESQUE/output_2010.txt

    # output format of SemEval-2013 Evaluating Word Sense Induction & Disambiguation within An End-User Application Task #11 

    java -jar WSI-Evaluator.jar datasets/MORESQUE clustering_example/MORESQUE/output_2013.txt 

Note that two possible formats are allowed for the clustering output by your 

Word Sense Induction/Disambiguation algorithm (recognized automatically by the evaluator):

- The same format used in the SemEval-2010 Word Sense Induction & Disambiguation Task #14

  (http://www.cs.york.ac.uk/semeval2010_WSI/)

- A new format structured as follows. The first line is fixed:

subTopicID	resultID

  All other lines are pairs of the following kind:

<queryID.subtopicID>	<queryID.snippetID>

- queryID is the ID for a given query, coming from the file datasets/<DATASET>/topics.txt

- queryID.subtopicID is a system ID for a certain cluster output by the system

- queryID.snippetID comes from the file datasets/<DATASET>/results.txt.

  For example 80.1 denotes the air sense of liquid_air: Liquid air - Wikipedia, the free encyclopedia	For the automobile, see Liquid Air. Liquid air is air that has been cooled to very low ... Liquid air can absorb heat rapidly and revert to its gaseous state. ...

The results.txt file contains the results returned by the search engine for the input queries. For instance, the following line in MORESQUE/results.txt:

80.1    http://en.wikipedia.org/wiki/Liquid_air Liquid air - Wikipedia, the free encyclopedia   For the automobile, see Liquid Air. Liquid air is air that has been cooled to very low ... Liquid air can absorb heat rapidly and revert to its gaseous state. ...

is the first result returned for query 80 (liquid_air) and contains:

- URL (http://en.wikipedia.org/wiki/Liquid_air);

- title (Liquid air - Wikipedia, the free encyclopedia)

- snippet text (For the automobile, see Liquid Air. Liquid air is air etc...)

Note that a hard clustering is assumed, i.e. each snippet has to belong to at most one cluster.

------------------

EVALUATOR'S OUTPUT

------------------

As a result of running the evaluator, several operations are reported in the wsi_eval.log file.

The final evaluation output, with all the evaluation measures, can be found in result.log.

----------

CONCLUSION

----------

Good Luck!

-------

AUTHORS

-------

Roberto Navigli, Sapienza University of Rome

(navigli@di.uniroma1.it)

Daniele Vannella, Sapienza University of Rome

(vannella@di.uniroma1.it)

-------

CONTACT

-------

Please feel free to get in touch with us for any question or problem you

may have using the following Google group:

  http://groups.google.com/group/semeval-2013-wsi-in-application?hl=en

---------------

ACKNOWLEDGMENTS

---------------

The authors gratefully acknowledge the support of the MultiJEDI ERC Starting Grant No. 259234 

(http://lcl.uniroma1.it/multijedi) and the CASPUR High-Performance Computing Grant 475/2011 and 117/2012.

############################ 

# WSI EVALUATION  PROPERTIES

############################

evaluation.createBucketCluster = TRUE

##############

# EVALUATION #

##############

# K for Subtopic-Recall@K

evaluation.subtopicRecall.K = 100

# The minimum value of recall from which to begin calculating the precision (expressed in percentage)

evaluation.subtopicPrecision.r = 0.4