Create prompt_task_map.json #737

yeganehkordi · 2022-03-11T21:23:06Z

Here is the map of the shared task between our tasks and training datasets of the T0 model.
They have trained the model on the 35 datasets. We have 31 shared tasks and lots of missing tasks that I've listed in the spreadsheet.
Also, for the paws, duorc, amazon_us_reviews, and hotpotqa datasets, our tasks don't have a specific subset. So, we may need to add them again.

danyaljj · 2022-03-11T21:26:59Z

doc/prompt_task_map.json

+    ],
+    "social_i_qa": [
+        "task384_socialiqa_question_classification",
+		"task580_socialiqa_answer_generation"


Hmm ... isn't social_i_qa a question answering task?

if so, I am confused about why it is mapped to an "answer_generation" task.

Yes, I mapped our tasks to the datasets. They have created the following tasks types from social_i_qa:

answer verification (task384_socialiqa_question_classification)

multiple-choice question answering (task580_socialiqa_answer_generation)

contextual question answering without options (missing task)

question generation from the given context and answer (missing task)
(They have one prompt for answer with option index and one for answer with the option string. I just add one of them as a missing task.)

Could you share the pointers to these?

answer verification (task384_socialiqa_question_classification)
multiple-choice question answering (task580_socialiqa_answer_generation)

Yeah, answer verification (task384_socialiqa_question_classification) -> data in sheet, task
multiple-choice question answering (task580_socialiqa_answer_generation) -> data in sheet, task
You can see the prompts in Hosted version of PromptSource.

I've added the prompt name and id as a key in the json files.
Also, I can change this file to a prompt to task map.

I've added the prompt name and id as a key in the json files.

Maybe I'm missing something here. Does the name of these json files tell us whether they correspond to socialiqa_question_classification or socialiqa_question_classification?

My understanding is that, here we have a 1-to-2 mapping (as opposed to 1-to-1 mapping). It's possible that I am missing something, in which case, help me see it! :)

No, you're right! I think I can use better architecture here.
Script gets the name of the dataset and generates a task for each prompt, and the name of the json files is based on the dataset and prompt names (e.g., task_socialiqa_Generate_answer).
The mapping is based on the datasets and our tasks and it hasn't mapped the tasks to the prompts. So, I need to add the prompt-task correspondence.

Yeah exactly.

I've updated the file to map the tasks to the prompt. I'll add T0p and T0pp datasets tmrw.

Palipoor · 2022-03-13T18:31:02Z

Hi! I am sorry for my absence @danyaljj @yeganehkordi.
This list looks good, do we have a list of task types that we have and they don't? (e.g. task246_dream_question_generation)

yeganehkordi · 2022-03-13T18:39:06Z

Hi! I am sorry for my absence @danyaljj @yeganehkordi. This list looks good, do we have a list of task types that we have and they don't? (e.g. task246_dream_question_generation)

Yes, The list of the missing tasks is in the spreadsheet (based on the dataset and prompt name).
A list of the tasks that we have is in this Json file.

yizhongw · 2022-03-13T18:57:50Z

@yeganehkordi Question: are you including only the T0 training datasets here? It seems some common datasets (e.g., COPA, BoolQ, WSC, etc.) are missing. Those are used for training T0p, T0pp as well as for evaluation. Could we add them?

yeganehkordi · 2022-03-13T18:59:54Z

@yeganehkordi Question: are you including only the T0 training datasets here? It seems some common datasets (e.g., COPA, BoolQ, WSC, etc.) are missing. Those are used for training T0p, T0pp as well as for evaluation. Could we add them?

Yes, They are only T0 training datasets. I'll add T0p and T0pp training datasets.

yizhongw · 2022-03-13T19:06:08Z

Yes, They are only T0 training datasets. I'll add T0p and T0pp training datasets.

Thanks! And also the test tasks mentioned here. We have their mapping before in our working doc, but it's not in a prompt-to-task format.

yeganehkordi · 2022-03-13T19:09:46Z

Yes, They are only T0 training datasets. I'll add T0p and T0pp training datasets.

Thanks! And also the test tasks mentioned here. We have their mapping before in our working doc, but it's not in a prompt-to-task format.

Will do.

yizhongw

I think the organization of datasets, NI tasks and PS prompts are much more clear. I left some comments for minor fix. @Palipoor Could you work on the missing datasets?

yizhongw · 2022-03-18T18:07:34Z

doc/prompt_task_map.json

+            ]
+        }
+    ],
+    "hotpot_qa/distractor": [


T0 casts hotpot_qa as a closed_book qa task. Can we add a hotpot_qa/closed_book placeholder here, and add the task together with other missing tasks?

Can you elaborate more?
They don't have any closed_book prompt. are you suggesting adding this task without prompt?

It seems they used the kilt version of hotpotqa, according the the list here.

Thanks! I'll add it.

doc/prompt_task_map.json

yizhongw · 2022-03-18T18:17:17Z

doc/prompt_task_map.json

+    ],
+    "winogrande/winogrande_l": [
+    ],
+    "winogrande/winogrande_debiased": [


For winogrande, could you confirm that the xl/xs/s/m/l/debiased settings only differ in the training sizes? I think we can remove the other and only keep this debiased setting.

Yes, Their only difference is their sizes.
Sure, will do.

yizhongw · 2022-03-18T18:19:32Z

doc/prompt_task_map.json

+            ]
+        }
+    ],
+    "super_glue/copa": [


Does task828_copa_commonsense_cause_effect or task827_copa_commonsense_reasoning correspond to this? We have been using these two tasks for our current evaluation.

There is a slight difference between our task and their tasks. Our task is to choose the completion which is the cause or effect of the first sentence, but they have specified that the completion should be which one of them in the instances.
So, I think our task is more general and probably more difficult.

Got it. Are we going to add a task similar to theirs?

Yeah, I will.

yizhongw · 2022-03-18T18:26:03Z

doc/prompt_task_map.json

+    ],
+    "wiki_hop/masked": [
+    ],
+    "adversarial_qa/adversarialQA": [


I think it's fine to keep all the four settings for adversarial_qa here. But when we add the missing tasks later, please just use this adversarialQA setting. Or, probably we can just drop the other three now to avoid confusion?

I don't know the setting of our task and I'm not sure if it is adversarial_qa or not.
If we don't need all the subsets, we can change the instances of this task and just keep adversarial_qa.

yizhongw · 2022-03-18T18:27:46Z

doc/prompt_task_map.json

+            ]
+        }
+    ],
+    "qasc": [


We have qasc tasks right? task040_qasc_question_generation, task041_qasc_answer_generation.

Yes, but we don't have any shared task with them.

yizhongw · 2022-03-18T18:32:19Z

doc/prompt_task_map.json

+    ],
+    "wiqa": [
+    ],
+    "cosmos_qa": [


I noticed that you included the question generation tasks for some QA datasets. This is good because I remember the original T0 also uses such prompts. But for some QA datasets, you didn't include the question generation task (e.g., task023_cosmosqa_question_generation can be included here for cosmos_qa). Can we add all of them if we already have them in our current data?

I've only added the tasks that had an equivalent prompt. In this case, we have a question generation task, but they don't have a question generation prompt for cosmos_qa.

danyaljj · 2022-03-26T01:26:25Z

Merging this PR since we iterated over it several times now. If anything is missing let's address in another PR.

Create prompt_task_map.json

daaa905

danyaljj requested review from Palipoor and yizhongw March 11, 2022 21:26

danyaljj reviewed Mar 11, 2022

View reviewed changes

yeganehkordi added 2 commits March 16, 2022 03:51

Update prompt_task_map.json

a5a22f4

Update prompt_task_map.json

a3eeebd

yizhongw requested changes Mar 18, 2022

View reviewed changes

Update prompt_task_map.json

f9f5652

danyaljj merged commit bbee421 into allenai:master Mar 26, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create prompt_task_map.json #737

Create prompt_task_map.json #737

yeganehkordi commented Mar 11, 2022 •

edited

Loading

danyaljj Mar 11, 2022

danyaljj Mar 11, 2022

yeganehkordi Mar 11, 2022 •

edited

Loading

danyaljj Mar 11, 2022

yeganehkordi Mar 11, 2022 •

edited

Loading

yeganehkordi Mar 13, 2022

danyaljj Mar 13, 2022

yeganehkordi Mar 13, 2022

danyaljj Mar 13, 2022

yeganehkordi Mar 16, 2022

Palipoor commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw left a comment •

edited

Loading

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

yizhongw Mar 18, 2022

yeganehkordi Mar 18, 2022

danyaljj commented Mar 26, 2022 •

edited

Loading

Create prompt_task_map.json #737

Create prompt_task_map.json #737

Conversation

yeganehkordi commented Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeganehkordi Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeganehkordi Mar 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Palipoor commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw commented Mar 13, 2022

yeganehkordi commented Mar 13, 2022

yizhongw left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danyaljj commented Mar 26, 2022 • edited Loading

yeganehkordi commented Mar 11, 2022 •

edited

Loading

yeganehkordi Mar 11, 2022 •

edited

Loading

yeganehkordi Mar 11, 2022 •

edited

Loading

yizhongw left a comment •

edited

Loading

danyaljj commented Mar 26, 2022 •

edited

Loading