created instructions

Coming soon

# Accuracy prediction for AMR parsing

predicts 33 accuracy metrics for a given sentence and its (automatic) AMR parse

### Training and testing a model

Put files with amr-parses into the directories ../data/training and ../data/dev and ../data/test

Each file is the output of an AMR parser, where parses are separated by emtpy lines. 

Two example files of four different systems (JAMR,CAMR,GPLA, GOLD) are contained in each directory.

Now we prepare the targets by comparison to gold

```

cd src

./split_into_single_files.sh

./run_full_eval.sh

'''

Now we prepare training, dev and test json-files

```

cd src

python create_data_dicts.py

'''

Now we train a toy model (which will be bad because of the small data)

```

cd src

python quamr_dep.py exampledict-gold.json.train,exampledict-camr.json.train,exampledict-jamr.json.train,exampledict-gpla.json.train exampledict-gold.json.dev,exampledict-camr.json.dev,exampledict-jamr.json.dev,exampledict-gpla.json.dev exampledict-gold.json.train,exampledict-camr.json.test,exampledict-jamr.json.test,exampledict-gpla.json.test 99 1

'''

training dev and testing data are separated with space. 

99 is some run-index and 1 refers to a specifig setup in config.py

## Using a trained model for predicting new data

Given: File with (automatic) parses separated by single line,

each AMR must have a \# snt:: line which is the orig. sentence of the corresponding parse.

### Prepare data for prediction

```

cd src

python create_data_dict_for_prediction.py INPUTFILE OUTPUTFILE

'''

for example

```

cd src

python create_data_dict_for_prediction.py ../data/test/exampledata-gpla.txt gpla-parses.json

'''

A pretrained model can be found in src/models, now we can predict the accuracies

```

cd src

python -u quamr_dep_predict.py gpla-parses.json 0 1

'''

0 is the model index of the pretrained model (I put one under src/models/) and 1 is the config setup

this prints out json lines with predicted accuracies which can be used e.g., for determining trusty parses for your downstream task.

### Citation

```

@inproceedings{opitzfrank:2019b,

  author = {Juri Opitz and Anette Frank},

  title = {{Automatic Accuracy Prediction for AMR Parsing}},

  year = {2019},

  booktitle = {Proceedings of the 8th Joint Conference on Lexical and Computational Semantics (*SEM)},

  address = {Minneapolis, Minnesota},

  note = {to appear},

  url = {https://arxiv.org/abs/1904.08301v1}

}

'''