Cosmograph Documentation Extras

[thorwhalen]

To put extra documentation and dev notes which we can then link in comments and docs of the code.

[thorwhalen]

_data_utils/_resources.py

This module provides a framework for synchronizing the configuration parameters of a
JavaScript visualization library, cosmograph, with Python’s cosmograph_widget.
Although the JS side is currently missing a formal single source of truth (SSOT),
the Python side endeavors to unify the multiple JS/TS files, markdown documentation,
and Python traitlets into a coherent SSOT (see Single Source Of Truth) that drives all Python-facing parameters
(name, type, default, description).

In practice, the module parses and reconciles TS code and markdown documentation
(optionally retrieved from GitHub). It then merges these details with Python-side
traitlets defined in the cosmograph_widget.Cosmograph class. The result is a robust
“SSOT” JSON file, so that the Python interface stays in sync with the latest or staged
TS changes.

flowchart LR
    subgraph Data Sources
        TS["TypeScript & GitHub URLs"]
        MD["Markdown Docs"]
        TRT["Python traitlets \(CosmographWidget\)"]
        JSONC["Local JSON Caches"]
    end
    
    TS --> C(ConfigsDacc)
    MD --> C
    TRT --> C
    JSONC --> C

    C --> MIDF[matched_info_df]
    C --> SIG[cosmograph_base_signature]
    C --> DOCS[cosmograph_base_docs]
    C --> PARAMS[cosmograph_base_params_json]

    subgraph Additional Resources
        R(ResourcesDacc)
        R --> ColorSet[color_names_set]
        R --> ColorTable[_color_table]
    end

    subgraph Legacy Approach
        LSD(ConfigSourceDicts)
        LAC(ConfigAnalysisDacc)
        LSD --> LAC
        LAC --> C
    end

Loading

Key Components

ConfigsDacc

A central class that creates, inspects, and updates the comprehensive SSOT data for
cosmograph’s configuration parameters. Typical functionalities include:

1. Parse and store source texts.
   Via methods like source_strings, the class fetches TypeScript and markdown files—
   either from local paths or GitHub—and caches the text locally
   (e.g., JSON and parquet caching via decorators like cache_to_json_files).

2. Extract and unify details.
   Once the raw content is fetched, parsed_defaults, parsed_types, and
   parsed_descriptions parse them into structured dictionaries or DataFrame objects.
   The module compares these results against Python’s traitlets in
   cosmograph_widget.Cosmograph.

3. Generate SSOT artifacts.

   • matched_info_df: A merged DataFrame that shows parameter alignment across
     defaults, types, descriptions, and actual traitlets.
   • cosmograph_base_signature and cosmograph_base_docs: Automatic generation
     of Python function signatures and docstrings that match the discovered TS/markdown
     content.
   • cosmograph_base_params_json: A JSON-serialized listing of parameter info
     (names, types, defaults, descriptions), forming the final SSOT the Python side
     relies on.

4. Diagnosis and staging.
   The module includes utilities for diagnosing alignment issues (print_diagnosis,
   signature_diffs, info_dfs). To handle updates carefully, developers can make a
   staged instance of ConfigsDacc (pointing to a temporary or “staging” directory)
   and compare it to the “current” instance that points to the active SSOT. Once
   verified, staged files can replace the existing ones.

ResourcesDacc

A smaller, focused class that manages non-configuration resources (like color tables or
other references) using the same caching mechanisms. It is less about parameter
alignment and more about storing prepared data used elsewhere in cosmograph.

Older Classes: ConfigSourceDicts and ConfigAnalysisDacc

These belong to a legacy approach for handling configuration data, mostly loading from
older JSON references:

• ConfigSourceDicts: Staticmethods returning older key–value data from local JSON
  files.
• ConfigAnalysisDacc: A more ad-hoc aggregator for analyzing those older
  configurations, now largely superseded by ConfigsDacc.

Typical Workflow

1. Instantiate
   Simply create a ConfigsDacc instance, which reads or creates caching files in a
   standard data directory.

2. Fetch & parse sources
   The call c.source_strings triggers loading from GitHub. Then c.parsed_defaults,
   c.parsed_types, and c.parsed_descriptions become available, each containing
   structured Python data for defaults, interfaces, or doc descriptions.

3. Inspect merges
   Access c.matched_info_df for an aligned table of parameter names across all
   sources. This reveals any mismatches in defaults or missing doc strings.

4. Generate Python interface

   • c.cosmograph_base_signature() returns a Python signature object with matching
     defaults and type annotations, all from the SSOT.
   • c.cosmograph_base_docs() yields docstring text that can directly drop into a
     Python class or function.

5. Staging updates

   • Create a second ConfigsDacc with its config_files_dir set to a staging path.
   • Re-compute or fetch from updated TS/markdown sources.
   • Compare the staged data to the current SSOT (check matched_info_df, parameter
     differences, or docstring changes).
   • If correct, copy the staged JSON files over to the main config directory, or
     update _params_ssot.json as needed.

With these tools, Python developers can confidently maintain a local single source of
truth for cosmograph’s parameter definitions—anticipating future changes on the JS side
without breaking alignment in Python.

[thorwhalen]

Q: What are these jsons of the config_prep folder? (parsed_defaults.json, parsed_descriptions.json etc.) How do I get/create those?

They are generated on-demand when the corresponding ConfigsDacc properties are
accessed. Within the module, each property—parsed_defaults, parsed_descriptions,
parsed_types—is decorated with something like:

@cache_to_prep_jsons
def parsed_defaults(...):
	  ...

That @cache_to_prep_jsons decorator tells the code to:

1. Parse the raw data (from the GitHub TypeScript, markdown files, etc.)
2. Generate a Python dictionary of the results
3. Save that dictionary as a JSON file in the config_prep folder
   (e.g., parsed_defaults.json, parsed_descriptions.json, etc.)

Then, whenever you do:

from cosmograph.dev_utils._resources import ConfigsDacc

c = ConfigsDacc()
defaults = c.parsed_defaults      # triggers read/generation of parsed_defaults.json
descriptions = c.parsed_descriptions  # triggers read/generation of parsed_descriptions.json
types = c.parsed_types           # triggers read/generation of parsed_types.json

the code looks in config_prep first. If a matching JSON file already exists, it
gets loaded immediately; if not, the source data is fetched, parsed, and saved to a
new file in config_prep.

[thorwhalen]

Q: What about _widget_config.json file -- it doesn't seem like that one is automatically created!

The short answer is that _widget_config.json is a holdover from an older workflow that merged all the Cosmograph TypeScript interface information into a single JSON artifact, then saved it under that filename. In the current code, there is no active function that writes _widget_config.json directly. Instead, the module now parses and caches data under names like parsed_types.json and parsed_descriptions.json in the prep_jsons folder, or under other user-specified paths.

The old pattern was something like this:
1. Fetch and parse multiple TS files (the ones in source_url_groups["types"]).
2. Merge these files’ interface data into one aggregated JSON object.
3. Write that JSON to the “config” cache folder with the key "_widget_config", producing a file named _widget_config.json.

That logic was captured in parts of ConfigAnalysisDacc and ConfigSourceDicts.widget_config_from_ts, which do something like:

widget_config = json_files["_widget_config.json"]

and then proceed to read that merged data.

Today, ConfigsDacc typically handles all the new code parsing and caching routines (for instance, storing its outputs in parsed_types.json). If you actually need _widget_config.json, the simplest approach is to replicate the older pattern by taking all the parsed TypeScript data (c.parsed_types), merging it yourself, and manually saving the result to _widget_config.json:

from cosmograph.dev_utils._resources import ConfigsDacc, json_files

c = ConfigsDacc()
all_types_data = c.parsed_types  # a dict keyed by each TS file
# merge or transform as needed...
json_files["_widget_config.json"] = all_types_data

This will give you a fresh _widget_config.json in the data folder, but it is no longer the main code path in the module. The new approach uses files like parsed_types.json and parsed_descriptions.json instead.