Update metadata documentation #1602

pabloarosado · 2023-09-13T16:27:30Z

Update metadata documentation in schemas.

…umentation

lucasrodes

Thanks, Pablo! LGTM in general, added minor comments.

lucasrodes · 2023-09-13T21:32:48Z

schemas/dataset-schema.json

-          "description": "Title of the dataset (mostly for internal purposes, or for users of our data catalog) which is a one-line description of the dataset (follow the guidelines of the origin's `data_product_title`). NOTE: Dataset titles should be propagated automatically from snapshots. By default, the title of the dataset will be the title of the containing table. But, if there are multiple tables, or if the user wants to manually edit the dataset title, it can be done by editing `dataset.title`.",
-          "requirement_level": "required (often automatic)"
+          "title": "Title of the ETL dataset",
+          "description": "Title of the dataset (mostly for internal purposes, or for users of our data catalog) which is a one-line description of the dataset. NOTE: Dataset titles should be propagated automatically from snapshots. By default, the title of the dataset will be the title of the containing table. But, if there are multiple tables, or if the user wants to manually edit the dataset title, it can be done by editing `dataset.title`.",


to manually edit the dataset title, it can be done by editing dataset.title.

I find it strange to refer to the field itself with the explicit name. I'd go for

to manually edit the dataset title, it can be done by editing this field.

Also, the description seems long enough that it would benefit from some line breaks?

Title of the dataset (mostly for internal purposes, or for users of our data catalog) which is a one-line description of the dataset.\n\nNote: Dataset titles should be propagated automatically from snapshots. By default, the title of the dataset will be the title of the containing table. But, if there are multiple tables, or if the user wants to manually edit the dataset title, it can be done by editing this field.

I have rewritten it a bit.

lucasrodes · 2023-09-13T21:35:14Z

schemas/dataset-schema.json

-          "description": "Description of the dataset (mostly for internal purposes, or for users of our data catalog) which is a one- (or a few) paragraph description of the dataset (follow the guidelines of the origin's `description`). NOTE: Dataset descriptions should be propagated automatically from snapshots. By default, the description of the dataset will be the description of the containing table. But, if there are multiple tables, or if the user wants to manually edit the dataset description, it can be done by editing `dataset.description`.",
-          "requirement_level": "recommended (often automatic)"
+          "title": "Description of the ETL dataset",
+          "description": "Description of the dataset (mostly for internal purposes, or for users of our data catalog) which is a one- (or a few) paragraph description of the content of the tables. NOTE: Dataset descriptions should be propagated automatically from snapshots. By default, the description of the dataset will be the description of the containing table. But, if there are multiple tables, or if the user wants to manually edit the dataset description, it can be done by editing `dataset.description`.",


Idem comments as in dataset.title. Would probably add a line break and make it dataset.description → this field.

lucasrodes · 2023-09-13T21:37:40Z

schemas/dataset-schema.json

+          "guidelines": [
+            ["Must start with a capital letter."],
+            ["Must end with a period."],
+            ["Must not mention other metadata fields like `producer`.", {"type": "exceptions", "value": ["Other metadata fields are crucial in the description of the data product."]}],


I would phrase it like

"Must not mention other metadata fields (e.g. producer or ...)." (add another metadata field)

and then

"The other metadata fields are crucial in describing the data product."

lucasrodes · 2023-09-13T21:39:08Z

schemas/dataset-schema.json

-            "description": "Title of the table (mostly for internal purposes, or for users of our data catalog) which is a few words description of the table (follow the guidelines as origin's `title`). NOTE: Table titles should be propagated automatically from snapshots (from origin's `title`). But, if there are multiple tables, or if the user wants to manually edit the title, it can be done by editing `table.title`.",
-            "requirement_level": "required (often automatic)"
+            "title": "Title of the ETL table",
+            "description": "Title of the table (mostly for internal purposes, or for users of our data catalog) which is a few words description of the table. NOTE: Table titles should be propagated automatically from snapshots (from origin's `title`). But, if there are multiple tables, or if the user wants to manually edit the title, it can be done by editing `table.title`.",


same comments as in dataset.title and dataset.description (e.g. breakline and referring to the field as this field).

lucasrodes · 2023-09-13T21:39:56Z

schemas/dataset-schema.json

          },
          "description": {
            "type": "string",
            "title": "Description of this ETL table",
-            "description": "Description of the table (mostly for internal purposes, or for users of our data catalog) which is a one- (or a few) paragraph description of the table (follow the guidelines of origin's `description`). NOTE: Table descriptions should be propagated automatically from snapshots (from origin's `description`). But, if there are multiple tables, or if the user wants to manually edit the description, it can be done by editing `table.description`.",
-            "requirement_level": "recommended (often automatic)"
+            "description": "Description of the table (mostly for internal purposes, or for users of our data catalog) which is a one- (or a few) paragraph description of the table. NOTE: Table descriptions should be propagated automatically from snapshots (from origin's `description`). But, if there are multiple tables, or if the user wants to manually edit the description, it can be done by editing `table.description`.",


same comments as in tables.title (e.g. breakline and referring to the field as this field).

lucasrodes · 2023-09-13T21:44:55Z

schemas/dataset-schema.json

@@ -462,8 +490,16 @@
                  "properties": {
                    "title_public": {
                      "type": "string",
-                      "description": "Indicator title to be shown in data pages (follow the guidelines of indicator's `title`). NOTE: This may be needed, for example, when the indicator comes from a big dataset where titles can't easily be curated manually. For those cases, `title_public` will override the indicator's `title`.",
-                      "requirement_level": "optional"
+                      "description": "Indicator title to be shown in data pages. NOTE: This may be needed, for example, when the indicator comes from a big dataset where titles can't easily be curated manually. For those cases, `title_public` will override the indicator's `title`.",


Same as in dataset.title (linebreaks)

lucasrodes · 2023-09-13T21:46:28Z

schemas/dataset-schema.json

@@ -488,22 +524,29 @@
                    "attribution": {
                      "type": "string",
                      "title": "Indicator's attribution",
-                      "description": "",
+                      "description": "Citation of the indicator's origins, to be used when the automatic format `producer1 (year1); producer2 (year2)` needs to be overridden.",


I'd rephrase it as

Citation of the indicator's origins. Use it only when you want to overwrite the automatic format producer1 (year1); producer2 (year2).

but not really important

lucasrodes · 2023-09-13T21:49:31Z

schemas/definitions.json

@@ -269,43 +272,52 @@
        "guidelines": [
          ["Must start with a capital letter."],
          ["Must end with a period."],
-          ["Should describe the data product, not the snapshot (i.e. the subset of data we extract from the data product)."]
+          ["Must not mention other metadata fields like `producer` or `version_producer`.", {"type": "exceptions", "value": ["Other metadata fields are crucial in the description of the data product."]}],


would modify

"Other metadata fields are crucial in the description of the data product."

to

"These other metadata fields are crucial in the description of the data product."

lucasrodes · 2023-09-13T21:53:28Z

schemas/definitions.json

        "examples": [
+          "Global Carbon Budget - Fossil fuels",
          "Neutron star mergers"
        ],
        "examples_bad": [
          [
+            "Global Carbon Budget",
            "Neutron star mergers (NASA, 2023)",
            "Data on neutron star mergers",
            "Neutron star mergers dataset"
          ]
        ],


To correctly render the list of examples in the docs or wizard we should either:

not have any examples or examples_bad

just have examples

have the same number of examples and examples_bad.

Here I'd do sth like

"examples": [ "Global Carbon Budget - Fossil fuels", "Neutron star mergers" ], "examples_bad": [ [ "Global Carbon Budget", ], [ "Neutron star mergers (NASA, 2023)", "Data on neutron star mergers", "Neutron star mergers dataset" ] ],

(see how it is done for attribution_short)

Right, I may have other examples that don't fulfil this rule.

lucasrodes · 2023-09-13T21:57:03Z

schemas/definitions.json

-          ["Should only be used if the automatic attribution format `producer (year)`is incomplete. For example, when the dataset title is well known and should be cited along with the producer, or when the original dataset version should also be mentioned."],
-          ["Should be constructed using placeholders, e.g. `{title}, {producer} ({year})`."]
+          ["Should only be used if the automatic attribution format `producer (year)` is considered uninformative. For example, when the title of the data product is well known and should be cited along with the producer, or when the original version of the data product should also be mentioned."],
+          ["Should follow the format `{producer} - {title} {version_producer} ({year})`."],


I am a bit confused with this one. Thought attribution was free, but from this guideline it seems that there is a specific style we should follow?

I've rephrased it. This is a "should" not a "must". My point here is that, if you are simply including the data product, then that should be the preferred way to do so. If you want to do something completely different, you are free.

…umentation

pabloarosado added 2 commits September 13, 2023 18:26

Update metadata documentation in schemas

badfa78

Merge branch 'master' of github.com:owid/etl into update-metadata-doc…

2b9947d

…umentation

pabloarosado requested a review from lucasrodes September 13, 2023 16:27

pabloarosado marked this pull request as ready for review September 13, 2023 16:27

github-actions bot assigned pabloarosado Sep 13, 2023

lucasrodes reviewed Sep 13, 2023

View reviewed changes

lucasrodes mentioned this pull request Sep 13, 2023

Write guidelines on how the new metadata should be used #1474

Closed

pabloarosado added 6 commits September 14, 2023 09:31

Merge branch 'master' of github.com:owid/etl into update-metadata-doc…

e26dae5

…umentation

Improve metadata documentation

21352a4

style: Improve format of json schemas

219635b

Revert prettify, as it causes CI errors, for some reason

cbf9d6a

Fix bad formatting of examples

00f8235

style: Improve format of json schemas

18cb9b2

pabloarosado merged commit c7a8ae8 into master Sep 14, 2023

pabloarosado deleted the update-metadata-documentation branch September 14, 2023 08:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update metadata documentation #1602

Update metadata documentation #1602

pabloarosado commented Sep 13, 2023

lucasrodes left a comment

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

pabloarosado Sep 14, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

lucasrodes Sep 13, 2023

pabloarosado Sep 14, 2023

lucasrodes Sep 13, 2023

pabloarosado Sep 14, 2023

Update metadata documentation #1602

Update metadata documentation #1602

Conversation

pabloarosado commented Sep 13, 2023

lucasrodes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment