our demo version

Protokoll, 14.01, Dienstag

Es gibt zwei Wege, wie wir weiter machen:

1. Verknüpfung von unserer Seite zu MTurk in einem Schritt zu machen. In diesem Fall brauchen wir Deployment nicht, es wird alles lokal verknüpft.

2. Verknüpfung von unserer Seite zu MTurk in zwei Schritte: erst Deployment (z.B., mit Heroku) und dann Verknüpfung mit MTurk, dann läuft alles auch extern

Wir haben 1. Variante ausgewählt, weil wir schon einige Info. davor im Internet recherchiert haben.

Foto von Schema habe ich in unsere Whatsapp-Gruppe geschickt.

Plan, was muss gemacht werden:

1. Wenn man ein Batch aus einem Project auswählt, muss ein Key-Paar generiert werden (etwas ähnliches wie für Annotatoren).

2. Danach wird man zu dem entsprechendem HIT (=Batch) auf MTurk weitergeleitet. Also, man braucht noch Template zusäztlich zu erstellen.

Link dazu: https://blog.mturk.com/tutorial-a-beginners-guide-to-crowdsourcing-ml-training-data-with-python-and-mturk-d8df4bdf2977

Hier muss es auch "Accept-it"-Button erstellt werden

3. Wenn man auf "Accept-it"-Button klick, wird man zu Seite geleitet, wo leeres Fenster für Key und Link zu unsere Batch mit Umfrage gibts.

Man klickt auf den Link um die Daten zu annotieren, gleichzeitig muss man die Seite mit leerem Key-Fenster nicht zuschliessen.

4. Auf die Umfrage-Seite neben dem Submit-Button Key-Fenster zusätzlich einbauen. Wenn man fertig mit der Annotation ist und die Annotation submitted, wird alles überprüft,

und wenn alles Ok ist bekommt man in Key-Fenster Key, die man auf der Seite aus dem Punkt 3 eingibt, wenn es mit davor generiertem Key-Paar übereinstimmt, dann wird von der Umfrage-Seite "ok" zu MTurk geschickt,

dann wird die Annotation akzeptiert und Annotator (Worker) bekommt Geld. Die annotierte Daten werden wie davor in DB von unserer Oberfläche gespeichert und BWS berechnet.

Plan für die Abschlusspräsentation am 04.02:

1. Allgemeine Information über das Projekt - Ziele, Beschreibung, Erwartungen (kann von der ersten Präsentation übernommen werden)

2. Finale Version (was man tatsächlich bekommen hat)

3. Demo (tatsächlich zeigen). Dafür sinnvole Daten vorbereiten, z.B., Sentiment Analyse

4. Schwierigkeiten, was ist gut, was ist schlecht.

import boto3

MTURK_SANDBOX = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com'

mturk = boto3.client('mturk',

   aws_access_key_id = "AKIAI57NPWPWYHGOWIPQ",

   aws_secret_access_key = "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",

   region_name='us-east-1',

   endpoint_url = MTURK_SANDBOX

)

# You will need the following library

# to help parse the XML answers supplied from MTurk

# Install it in your local environment with

# pip install xmltodict

import xmltodict

# Use the hit_id previously created

hit_id = '3PKVGQTFIHR9FF0SG3QBFNHWVI7RY3'#'3DZKABX2ZICO5JG4TVDWJTEGD0ZVCP'

# We are only publishing this task to one Worker

# So we will get back an array with one item if it has been completed

worker_results = mturk.list_assignments_for_hit(HITId=hit_id, AssignmentStatuses=['Submitted'])

response = mturk.list_review_policy_results_for_hit(HITId=hit_id)

'''worker_results looks like that:

{

    'AssignmentId': 'string',

    'WorkerId': 'string',

    'HITId': 'string',

    'AssignmentStatus': 'Submitted'|'Approved'|'Rejected',

    'AutoApprovalTime': datetime(2015, 1, 1),

    'AcceptTime': datetime(2015, 1, 1),

    'SubmitTime': datetime(2015, 1, 1),

    'ApprovalTime': datetime(2015, 1, 1),

    'RejectionTime': datetime(2015, 1, 1),

    'Deadline': datetime(2015, 1, 1),

    'Answer': 'string',

    'RequesterFeedback': 'string'

}

'''

#print ("A new HIT has been created. You can preview it here:")

#print ("https://workersandbox.mturk.com/mturk/preview?groupId=" + mturk_example.new_hit['HIT']['HITGroupId'])

#print ("HITID = " + hit_id + " (Use to Get Results)")

# Remember to modify the URL above when you're publishing HITs to the live marketplace.

# Use: https://worker.mturk.com/mturk/preview?groupId=

print(response)

print('\n')

if worker_results['NumResults'] > 0:

   for assignment in worker_results['Assignments']:

      xml_doc = xmltodict.parse(assignment['Answer'])

      print("Worker's answer was:")

      if type(xml_doc['QuestionFormAnswers']['Answer']) is list:

         # Multiple fields in HIT layout

         for answer_field in xml_doc['QuestionFormAnswers']['Answer']:

            print("For input field: " + answer_field['QuestionIdentifier'])

            print("Submitted answer: " + answer_field['FreeText'])

      else:

         # One field found in HIT layout

         print("For input field: " + xml_doc['QuestionFormAnswers']['Answer']['QuestionIdentifier'])

         print("Submitted answer: " + xml_doc['QuestionFormAnswers']['Answer']['FreeText'])

else:

   print("No results ready yet")

'''

test if our keys work and if we can connect to the Mturk engine

'''

import boto3

MTURK_SANDBOX = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com' 

mturk = boto3.client('mturk',

   aws_access_key_id = "AKIAI57NPWPWYHGOWIPQ",

   aws_secret_access_key = "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",

   region_name='us-east-1',

   endpoint_url = MTURK_SANDBOX) # later delete this argument in order to connect a real MTurk account

#connect to the mturk api and get the account balance 

print ("I have $" + mturk.get_account_balance()['AvailableBalance'] + " in my Sandbox account")

#print(mturk)

#defining a HIT

#print(render_without_request('questions.xml'))

#question = open('questions.xml', 'r').read()

question = """

<HTMLQuestion xmlns="http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2011-11-11/HTMLQuestion.xsd">

<HTMLContent><![CDATA[

<html>

<head>

<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>

<script type='text/javascript' src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js'></script>

</head>

<body>

<form name='mturk_form' method='post' id='mturk_form' action='https://workersandbox.mturk.com/mturk/externalSubmit'><input type='hidden' value='' name='assignmentId' id='assignmentId'/>

<h2>{{title}} Is this Tweet happy, angry, excited, scared, annoyed or upset? Type in one word to describe the main emotion in the message. If it is unclear, type in "unclear".</h2>

<h3> {{description}} Tweet: "I am really looking forward to the next Seahawks game!" take a test</h3>

<ul>

  <li> You will receive a keyword to type in the field below after annotating this batch. Don't close this window! Type it in correctly or you won't get paid for your annotations. If you don't want to do this HIT, just close the window with the questionnaire, don't submit it! </li>

  <li> Click <a href='#' target='_blank' rel='noopener noreferrer'> here </a> to go to the site with questionnaire ! </li>

</ul>

<div class="form-group col-md-4">

    <input type='text' name='keyword' placeholder='Type in your keyword' />

  </div>

  <div class="form-group col-md-3"> <input class="btn btn-primary" type='submit' id='submitButton' value='Submit' /> </div>

<script language='Javascript'>turkSetAssignmentID();</script></body></html>

<!-- YOUR HTML ENDS -->

]]>

</HTMLContent>

<FrameHeight>600</FrameHeight>

</HTMLQuestion>

"""

new_hit = mturk.create_hit(

    Title = 'Is this Tweet happy, angry, excited, scared, annoyed or upset? or whatever it is called with something more',

    Description = 'Read this tweet and type out one word to describe the emotion of the person posting it: happy, angry, scared, annoyed or upset',

    Keywords = 'text, quick, labeling',

    Reward = '0.15',

    MaxAssignments = 1,

    LifetimeInSeconds = 172800,

    AssignmentDurationInSeconds = 600,

    AutoApprovalDelayInSeconds = 14400,

    Question = question,

    AssignmentReviewPolicy = {

    'PolicyName':'ScoreMyKnownAnswers/2011-09-01',

    'Parameters': [ {'Key':'AnswerKey', 'MapEntries':[{ 'Key':'keyword', 'Values':['happy']}]},

                    #{'Key':'ApproveIfKnownAnswerScoreIsAtLeast', 'Values':['1']},

                    {'Key':'RejectIfKnownAnswerScoreIsLessThan', 'Values':['1']},

                    {'Key':'RejectReason', 'Values':['Sorry, we could not approve your submission as you did not choose the write emotion.']}

                  ]

    }

)

print ("A new HIT has been created. You can preview it here:")

print ("https://workersandbox.mturk.com/mturk/preview?groupId=" + new_hit['HIT']['HITGroupId'])

print ("HITID = " + new_hit['HIT']['HITId'] + " (Use to Get Results)")

# Remember to modify the URL above when you're publishing HITs to the live marketplace.

# Use: https://worker.mturk.com/mturk/preview?groupId=

'''

test if our keys work and if we can connect to the Mturk engine

'''

import boto3

from botocore import UNSIGNED

from botocore.client import Config

from flask import render_template, make_response

import jinja2

def render_without_request(template_name, **template_vars):

    """

    Usage is the same as flask.render_template:

    render_without_request('my_template.html', var1='foo', var2='bar')

    """

    env = jinja2.Environment(

        loader=jinja2.PackageLoader('demo_app','demo_app')

    )

    template = env.get_template(template_name)

    return template.render(**template_vars)

MTURK_SANDBOX = 'https://mturk-requester-sandbox.us-east-1.amazonaws.com' 

mturk = boto3.client('mturk',

                      region_name='us-east-1',

                      endpoint_url = MTURK_SANDBOX)

mturk._request_signer.sign = (lambda *args, **kwargs: None)

'''

mturk = boto3.client('mturk',

   aws_access_key_id = "AKIAI57NPWPWYHGOWIPQ",

   aws_secret_access_key = "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",

   region_name='us-east-1',

   endpoint_url = MTURK_SANDBOX) # later delete this argument in order to connect a real MTurk account

'''

#connect to the mturk api and get the account balance 

print ("I have $" + mturk.get_account_balance()['AvailableBalance'] + " in my Sandbox account")

print(mturk)

#defining a HIT

#print(render_without_request('questions.xml'))

question = open('questions.xml', 'r').read()

'''

question = """

<html>

<head>

<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'/>

<script type='text/javascript' src='https://s3.amazonaws.com/mturk-public/externalHIT_v1.js'></script>

</head>

<body>

<form name='mturk_form' method='post' id='mturk_form' action='https://www.mturk.com/mturk/externalSubmit'><input type='hidden' value='' name='assignmentId' id='assignmentId'/>

<h2>Is this Tweet happy, angry, excited, scared, annoyed or upset? Type in one word to describe the main emotion in the message. If it is unclear, type in "unclear".</h2>

<h3> Tweet: "I am really looking forward to the next Seahawks game!"</h3>

<div>

  <input type='text' name='reported_emotion' placeholder='Type in your answer here'>

</div>

<p><input type='submit' id='submitButton' value='Submit' /></p></form>

<script language='Javascript'>turkSetAssignmentID();</script></body></html>

<!-- YOUR HTML ENDS -->

"""

'''

new_hit = mturk.create_hit(

    Title = 'Is this Tweet happy, angry, excited, scared, annoyed or upset?',

    Description = 'Read this tweet and type out one word to describe the emotion of the person posting it: happy, angry, scared, annoyed or upset',

    Keywords = 'text, quick, labeling',

    Reward = '0.15',

    MaxAssignments = 1,

    LifetimeInSeconds = 172800,

    AssignmentDurationInSeconds = 600,

    AutoApprovalDelayInSeconds = 14400,

    Question = question,

)

'''

print ("A new HIT has been created. You can preview it here:")

print ("https://workersandbox.mturk.com/mturk/preview?groupId=" + new_hit['HIT']['HITGroupId'])

print ("HITID = " + new_hit['HIT']['HITId'] + " (Use to Get Results)")

# Remember to modify the URL above when you're publishing HITs to the live marketplace.

# Use: https://worker.mturk.com/mturk/preview?groupId=

'''

 No newline at end of file