Tasks 1400-1425: Assigning categories, domains and addressing feedback (

#651)

* Update tasks

* Update tasks

* Update definitions

doc/task-hierarchy.md

@@ -92,8 +92,8 @@
   - `Identification -> Object Identification`
   - `Identification -> Predicate Identification`
   - `Identification -> Subject Identification`
-- `Language Identification`
-  - `Language Identification -> Verification`
+  - `Identification -> Language Identification`
+    - `Identification -> Language Identification -> Verification`
 - `Mathematics`
   - `Mathematics -> Algebra`
   - `Mathematics -> Arithmetic`

tasks/task1400_obqa_incorrect_answer_generation.json

@@ -6,9 +6,13 @@
         "OpenBookQA: https://allenai.org/data/open-book-qa"
     ],
     "Categories": [
-        "Incorrect Answer Generation"
+        "Question Answering -> Incorrect Answer Generation -> Contextual Question Answering -> Abstractive",
+        "Reasoning -> Factual Reasoning"
     ],
     "Definition": "This task is about generating an incorrect answer to a question given the question and a true statement related to the question. The answer must be related to the question, but incorrectly answer it given the context.",
+    "Domains": [
+        "Natural Science -> School Science Textbooks"
+    ],
     "Positive Examples": [
         {
             "input": "Fact: deep sea animals live deep in the ocean. Question: Frilled sharks and angler fish live far beneath the surface of the ocean, which is why they are known as?",

tasks/task1401_obqa_sentence_generation.json

@@ -6,9 +6,12 @@
         "OpenBookQA: https://allenai.org/data/open-book-qa"
     ],
     "Categories": [
-        "Sentence Generation"
+        "Text Generation -> Sentence Generation -> Explanation Generation"      
     ],
     "Definition": "In this task, you are given a question and a corresponding answer. Your task is to generate a fact statement that is useful in answering the given question.",
+    "Domains": [
+        "Natural Science -> School Science Textbooks"
+    ],
     "Positive Examples": [
         {
             "input": "Question: Frilled sharks and angler fish live far beneath the surface of the ocean, which is why they are known as? Answer: deep sea fish",

tasks/task1402_clue_question_generation.json

@@ -6,7 +6,7 @@
         "CLUE CMRC2018: /~https://github.com/ymcui/cmrc2018"
     ],
     "Categories": [
-        "Question Generation"
+        "Question Generation -> Contextual Question Generation"
     ],
     "Input_language": [
         "Chinese"
@@ -18,6 +18,9 @@
         "English"
     ],
     "Definition": "Given a passage in simplified Chinese, generate a reading comprehension question. The question should be unambiguous and the answer to this question should be in the passage.",
+    "Domains": [
+        "Wikipedia"
+    ],
     "Positive Examples": [
         {
             "input": "Passage: 《战国无双3》（）是由光荣和ω-force开发的战国无双系列的正统第三续作。本作以三大故事为主轴，分别是以武田信玄等人为主的《关东三国志》，织田信长等人为主的《战国三杰》，石田三成等人为主的《关原的年轻武者》，丰富游戏内的剧情。此部份专门介绍角色，欲知武器情报、奥义字或擅长攻击类型等，请至战国无双系列1.由于乡里大辅先生因故去世，不得不寻找其他声优接手。从猛将传 and Z开始。2.战国无双 编年史的原创男女主角亦有专属声优。此模式是任天堂游戏谜之村雨城改编的新增模式。本作中共有20张战场地图（不含村雨城），后来发行的猛将传再新增3张战场地图。但游戏内战役数量繁多，部分地图会有兼用的状况，战役虚实则是以光荣发行的2本「战国无双3 人物真书」内容为主，以下是相关介绍。（注：前方加☆者为猛将传新增关卡及地图。）合并本篇和猛将传的内容，村雨城模式剔除，战国史模式可直接游玩。主打两大模式「战史演武」&「争霸演武」。系列作品外传作品",

tasks/task1403_check_validity_date_mmddyyyy.json

@@ -6,9 +6,13 @@
         "synthetic"
     ], 
     "Categories": [
-        "Answer Generation"
+        "Classification -> Verification",
+        "Reasoning -> Commonsense Reasoning -> Numerical Commonsense Reasoning"
     ], 
     "Definition": "In this task, you are given a date in \"mm/dd/yyyy\" format. You need to check if the date is valid or not. Return 1 if it is valid, else return 0. A date is valid is the components month(\"mm\"), day(\"dd\") and year(\"yyyy\") are all valid individually. A day(dd) is valid if it is greater than or equal to 1 and less than 30 or 31 depending upon the month(mm). Months which have 31 days are January, March, May, July, August, October, December. Rest of the months have 30 days except February which has 28 days if it is not a leap year and 29 days if it is a leap year. A month(mm) is valid if it lies in the range from 1 to 12 as there are 12 months in a year. A year is always valid if it is expressed in the form of \"yyyy\".", 
+    "Domains": [      
+        "Commonsense -> Concepts and Relations"
+    ],
     "Positive Examples": [
         {
             "input": "14/25/1405", 

tasks/task1404_date_conversion.json

@@ -6,9 +6,14 @@
         "synthetic"
     ], 
     "Categories": [
-        "Answer Generation"
+        "Text Generation -> Command Execution -> Code Execution",
+        "Text Modification -> Structured Text Modification",
+        "Reasoning -> Commonsense Reasoning"
     ], 
     "Definition": "In this task, you are given a date in a particular format and you need to convert to another format. If given format is \"dd/mm/yyyy\" then convert to \"mm/dd/yyyy\". If given format is \"mm/dd/yyyy\" then convert to \"dd/mm/yyyy\".", 
+    "Domains": [      
+        "Commonsense -> Concepts and Relations"
+    ],
     "Positive Examples": [
         {
             "input": "10/05/1847, input_format=dd/mm/yyyy", 

tasks/task1405_find_median.json

@@ -6,9 +6,16 @@
         "synthetic"
     ],
     "Categories": [
-        "Answer Generation"
+        "Text Generation -> Command Execution -> Code Execution",
+        "Text Span Selection",
+        "Reasoning -> Numerical Reasoning",
+        "Mathematics -> Statistics"
     ],
     "Definition": "In this task, you are given a list of integers. You need to find the median of the list of integers and return that as the output. The median is the middle value in the list of numbers such that half of the elements are less than the median and the other half of elements are greater than the median.",
+    "Domains": [      
+        "Mathematics",
+        "Statistics"
+    ],
     "Positive Examples": [
         {
             "input": "[149, 403, 272, 453, 472, 499, 419, 277, 411, 252, 48, 359, 351, 147, 298, 61, 114, 178, 250, 34, 400, 417, 184, 326, 96]",

tasks/task1406_kth_smallest_element.json

@@ -6,9 +6,16 @@
         "synthetic"
     ], 
     "Categories": [
-        "Answer Generation"
-    ], 
+        "Text Generation -> Command Execution -> Code Execution",
+        "Text Span Selection",
+        "Reasoning -> Numerical Reasoning",
+        "Mathematics -> Statistics"
+    ],
     "Definition": "In this task, you are given a list of integers and an integer k. You need to find the kth smallest element in the input list and return that.", 
+    "Domains": [      
+        "Mathematics",
+        "Statistics"
+    ],
     "Positive Examples": [
         {
             "input": "[163, 290, 90, 51, 96, 86, 282, 265, 10, 85, 91, 201, 68, 126, 26, 228, 229, 236, 31, 71], k=14", 

tasks/task1407_dart_question_generation.json

@@ -3,13 +3,16 @@
         "dart (https://huggingface.co/datasets/dart)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Question Generation"],
-    "Domains": ["Wikipedia"],
-
+    "Categories": [
+        "Question Generation -> Fill in the Blank",
+        "Structured Text Processing -> Operation on Primitives -> List -> String"
+    ],
+    "Domains": [
+        "Wikipedia" 
+    ],
     "Input_language": ["English"],
     "Output_language": ["English"],
-    "Instruction_language": ["English"],
-
+    "Instruction_language": ["English"],    
     "Definition": "This task is about using the specified Resource Description Framework (RDF) triplets consisting of [subject, predicate, object] and converting them to a valid fill-the-slots question that utilizes one or more of these triplets. Triplet values encompassed in [*] are special tokens useful that can be replaced with synonyms. The objective is to construct a question in a manner that (a) captures the facts specified in at least one of the RDF triplets, and (b) ideally contains a limited number of slots such that it is a well-formed question that is easy to answer. A slot can represent a single word or a phrase. The input is a list of triplets of the form [subject, predicate, object] and the output is a question based on the triplets but with the subject and/or object replaced with slots (represented using two or more consecutive underscores).",
     "Positive Examples": [
         {

tasks/task1408_dart_similarity_classification.json

@@ -3,12 +3,15 @@
         "dart (https://huggingface.co/datasets/dart)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Relevance Verification"],
+    "Categories": [
+        "Classification -> Verification -> Relevance Verification",
+        "Text Comparison -> Relevancy Comparison",
+        "Structured Text Processing"
+    ],
     "Domains": ["Wikipedia"],
     "Input_language": ["English"],
     "Output_language": ["English"],
     "Instruction_language": ["English"],
-
     "Definition": "This task is about classifying the similarity of two sentences. The sentences can be classified as (a) SIMILAR - similar to each other, and (b) DISSIMILAR - not similar to each other. Sentences that have the same RDF relationship in terms of [subject, predicate, object] are similar to each other. The input is a list of two sentences and the output is either SIMILAR or DISSIMILAR.",
     "Positive Examples": [
         {

tasks/task1410_dart_relationship_extraction.json

@@ -3,12 +3,14 @@
         "dart (https://huggingface.co/datasets/dart)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Relationship Extraction"],
+    "Categories": [
+       "Text Generation -> Structured Text Generation -> List", 
+       "Relation Prediction"
+    ],
     "Domains": ["Wikipedia"],
     "Input_language": ["English"],
     "Output_language": ["English"],
     "Instruction_language": ["English"],
-
     "Definition": "This task is about using the specified sentence and converting the sentence to Resource Description Framework (RDF) triplets of the form (subject, predicate object). The RDF triplets generated must be such that the triplets accurately capture the structure and semantics of the input sentence. The input is a sentence and the output is a list of triplets of the form [subject, predicate, object] that capture the relationships present in the sentence. When a sentence has more than 1 RDF triplet possible, the output must contain all of them.",
     "Positive Examples": [
         {

tasks/task1411_dart_subject_identification.json

@@ -3,7 +3,10 @@
         "dart (https://huggingface.co/datasets/dart)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Token Classification"],
+    "Categories": [
+        "Identification -> Subject Identification",
+        "Text Span Selection"
+    ],
     "Domains": ["Wikipedia"],
     "Input_language": ["English"],
     "Output_language": ["English"],

tasks/task1413_dart_object_identification.json

@@ -3,12 +3,14 @@
         "dart (https://huggingface.co/datasets/dart)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Token Classification"],
+    "Categories": [
+        "Identification -> Object Identification",
+        "Text Span Selection"
+    ],
     "Domains": ["Wikipedia"],
     "Input_language": ["English"],
     "Output_language": ["English"],
-    "Instruction_language": ["English"],
-
+    "Instruction_language": ["English"],    
     "Definition": "This task is about identifying the object of a given sentence. The object of a sentence is the person or thing that receives the action of the verb. It is the who or what that the subject does something to. Each sentence in this task is guaranteed to contain a single object. The input is a sentence and the output is the identified object of that sentence.",
     "Positive Examples": [
         {

tasks/task1414_ajgt_twitter_ar_classification.json

@@ -3,13 +3,19 @@
         "ajgt_twitter_ar (https://huggingface.co/datasets/ajgt_twitter_ar)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Sentiment Analysis"],
+    "Categories": [
+        "Sentiment Analysis -> Sentiment Classification", 
+        "Reasoning -> Reasoning on Social Interactions",
+        "Reasoning -> Commonsense Reasoning -> Social Situations",
+        "Classification -> Verification"
+    ],
     "Input_language": ["Arabic"],
     "Output_language": ["English"],
-    "Instruction_language": ["English"],
-
+    "Instruction_language": ["English"],        
     "Definition": "This task is about classifying the sentiment of tweets in the Arabic language as POSITIVE or NEGATIVE. A positive (negative) sentiment indicates that the expressed opinion in the sentence is positive (negative). The input is a sentence is a sentence in Arabic and the output is the classified sentiment.",
-
+    "Domains": [
+        "Social Media -> Twitter"
+    ],
     "Positive Examples": [
         {
             "input": " اربد فيها جامعات اكثر من عمان ... وفيها قد عمان ونص لعيبه المنتخب منها ... و 80 % من مطربين الاردن منها",

tasks/task1415_youtube_caption_corrections_grammar_correction.json

@@ -3,14 +3,17 @@
         "youtube_caption_corrections (https://huggingface.co/datasets/youtube_caption_corrections)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Grammar Error Correction"],
-    "Domains": ["Machine Learning"],
+    "Categories": [
+        "Proofreading -> Grammar Error -> Grammar Error Correction"
+    ],
     "Input_language": ["English"],
     "Output_language": ["English"],
     "Instruction_language": ["English"],
-
-    "Definition": "Given an input stream, the objective of this task is to produce a grammatically correct version of the input sentence. The sentence can contain words that have case, punctuation, stem-based, intra-word, and digit puncutation errors that need to be corrected. For example, the text should be uppercased/lowercased as per necessity (like the beginning of a sentence, proper nouns etc.), digits need to be normalized (6 as six and so on). The input to this task is a stream of words, possibly from captions generated by a speech-to-text engine, and the output is a grammatically correct version of the input stream.",
-
+    "Definition": "In this task, you are given an input stream, and your task is to produce a grammatically correct version of the input sentence. The input to this task is a stream of words, possibly from captions generated by a speech-to-text engine, and the output is a grammatically correct version of the input stream. Stream words are in [] and have been separated with a comma. The sentence can contain words that have case, punctuation, stem-based, intra-word, and digit punctuation errors that need to be corrected. For example, the text should be uppercased/lowercased as per necessity (like the beginning of a sentence, proper nouns, etc.), digits need to be normalized (6 as six and so on). ",    
+    "Domains": [
+        "Computer Science -> Machine Learning",
+		"Captions -> Video Captions"
+    ],
     "Positive Examples": [
         {
             "input": "['hey', 'everybody', 'ivan', 'from', 'weights', 'and', 'biases', 'here', 'in', 'this', 'video', \"i'd\"]",

tasks/task1416_youtube_caption_corrections_incorrect_grammar_classification.json

@@ -3,14 +3,17 @@
         "youtube_caption_corrections (https://huggingface.co/datasets/youtube_caption_corrections)"
     ],
     "Contributors": ["Rushang Karia"],
-    "Categories": ["Grammar Error Detection"],
-    "Domains": ["Machine Learning"],
+    "Categories": [
+        "Proofreading -> Grammar Error -> Grammar Error Correction"
+    ],
     "Input_language": ["English"],
     "Output_language": ["English"],
     "Instruction_language": ["English"],
-
-
     "Definition": "Given an input stream, the objective of this task is to classify whether words in the stream are grammatically correct or not. The input to this task is a stream of words, possibly from captions generated by a speech-to-text engine, and the output is a classification of each word from the labels (reason) = [NO_DIFF (correct), CASE_DIFF (case error), PUNCUATION_DIFF (punctuation error), CASE_AND_PUNCUATION_DIFF (both case and punctuation error), STEM_BASED_DIFF (stem word error), DIGIT_DIFF (digit error), INTRAWORD_PUNC_DIFF (intra-word punctuation error), and UNKNOWN_TYPE_DIFF (an error that does not corrrespond to the previous categories)].",
+    "Domains": [
+        "Computer Science -> Machine Learning",
+		"Captions -> Video Captions"
+    ],
     "Positive Examples": [
         {
             "input": "['hey', 'everybody', 'ivan', 'from', 'weights', 'and', 'biases', 'here', 'in', 'this', 'video', \"i'd\"]",

tasks/task1418_bless_semantic_relation_classification.json

@@ -6,7 +6,10 @@
         "BLESS dataset from \"How we BLESSed distributional semantic evaluation\""
     ],
     "Categories": [
-        "Classification"
+        "Classification",
+        "Detection -> Relation Detection",
+        "Reasoning -> Commonsense Reasoning",
+        "Reasoning -> Relational Reasoning"
     ],
     "Input_language": [
         "English"
@@ -17,7 +20,11 @@
     "Instruction_language": [
         "English"
     ],
-    "Definition": "Given a pair of words, generate the relation between them. The first word is called the 'concept' and the second word is called the 'relatum' The relation must be one of the following: co-hyponym (coordinate), hypernym, meronym, attribute, event, or random. A coordinate relation indicates that the concept and relatum belong to the same semantic class. A hypernym relation indicates that the relatum is a category of which the concept is a specific instance. A meronym relation implies that relatum is a part/component/organ/member of the concept. An attribute relation is when the relatum is an adjective expressing an attribute of the concept. An event relation holds when the relatum is a verb referring to an action/activity/happening/event that is performed by or with the concept. If the relatum and concept are unrelated, the relation between them is 'random'\",",
+    "Definition": "Given a pair of words, generate the relation between them. The first word is called the 'concept' and the second word is called the 'relatum' The relation must be one of the following: co-hyponym (coordinate), hypernym, meronym, attribute, event, or random. A coordinate relation indicates that the concept and relatum belong to the same semantic class. A hypernym relation indicates that the relatum is a category of which the concept is a specific instance. A meronym relation implies that relatum is a part/component/organ/member of the concept. An attribute relation is when the relatum is an adjective expressing an attribute of the concept. An event relation holds when the relatum is a verb referring to an action/activity/happening/event that is performed by or with the concept. If the relatum and concept are unrelated, the relation between them is 'random'. Classify your answers into coord, hyper, mero, attri, event, and random.",
+    "Domains": [
+    		"Animals",
+        "Commonsense"
+    ],
     "Positive Examples": [
         {
             "input": "Concept: alligator, Relatum: lizard.",

tasks/task1419_mathqa_gain.json

@@ -6,12 +6,16 @@
         "MathQA ( https://math-qa.github.io/math-QA/ )"
     ],
     "Categories": [
-        "Answer Generation -> Multiple Choice Question Answering"
+        "Answer Generation -> Multiple Choice Question Answering",
+        "Question Answering -> Numerical Question Answering",
+        "Reasoning -> Quantitative Reasoning",
+        "Reasoning -> Numerical Reasoning",
+        "Mathematics -> Arithmetic"
     ],
     "Domains": [
         "Mathematics"
     ],
-    "Definition": "In this task, you need to provide the correct option for a given problem on gain from the provided options.",
+    "Definition": "In this task, you need to answer the given multiple-choice question on the gain. Gain is the value by which to multiply the input. Classify your answers into 'a', 'b', 'c', 'd', and 'e'.",
     "Input_language": [
         "English"
     ],