Commit 69c28d09 authored by schaper's avatar schaper
Browse files

Initial commit with some structure

parent 5e0553de
Loading
Loading
Loading
Loading

paws.py

0 → 100644
+16 −0
Changes for paws.py: 16 added lines, 0 removed lines.
Original line number Diff line number Diff line
from datasets import load_dataset
from sentence_transformers import SentenceTransformer

dataset = load_dataset('paws', 'labeled_final', split='test')

dataset.set_format(type='pandas')

foo = dataset[:]

model = SentenceTransformer('models/stsb-bert-large/')

# predict dataset instances with model and evaluate with confusion matrix
# calculate spearman rank correlation between cosine-similarity of the sentence embeddings and the gold labels
# (or if too hard pearson correlation)

# use code from evaluation_stsbenchmark.py