Enable running OGE pipeline with Early Release data and PUDL nightly builds #390
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
grgmiller
changed the title
rouille
reviewed
Sep 6, 2024
src/oge/data_pipeline.py
Outdated
| @@ -142,6 +142,7 @@ def main(args): | |||
| logger.info("1. Downloading data") | |||
| # PUDL | |||
| download_data.download_pudl_data(source="aws") | |||
| logger.info(f"Using {os.getenv('PUDL_BUILD')} PUDL build") | |||
There was a problem hiding this comment.
I think you want to handle the case where PUDL_BUILD is not set:
logger.info(f"Using {os.getenv('PUDL_BUILD', 'stable')} PUDL build")
rouille
reviewed
Sep 6, 2024
| @@ -107,56 +108,77 @@ def download_pudl_data(source: str = "aws"): | |||
| Args: | |||
| source (str, optional): where to download pudl from, either 'aws' or 'zenodo'. | |||
| Defaults to 'aws'. | |||
| build (str): whether to download the "stable" or "nightly" build | |||
|
|
|||
| Raises: | |||
| ValueError: if `source` is neither 'aws' or 'zenodo'. | |||
There was a problem hiding this comment.
You can add the UserWarning to the docstring.
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
In order to access hourly OGE data for 2023, this PR updates the pipeline to access and run EIA early release data.
NOTE: this PR does nothing to check or address data quality warnings raised for early release data, but just makes it available. Any data produced for an early release year should be considered "use at your own risk" for now.
This also updates the pipeline to work with the most recent stable pudl release: v2024.8.0 (see https://catalystcoop-pudl.readthedocs.io/en/stable/release_notes.html#v2024-8-0-2024-08-19). This release mostly just adds new data, but also fixes some of the issues with missing/inconsistent generator operating dates, and changes the pudl.sqlite file from a gzip to a zip.
What the code is doing
We add a new constant
current_early_release_yearwhich defines the year that early release data is available for.When I wrote this code last week, the early release data was only available through the nightly build data release of pudl, rather than the stable build, so I have updated the data download function to download the nightly build data if specified by a new environment variable
PUDL_BUILD, and also access the pudl database from the correct location (pudl databases are now either saved inpudl/stableorpudl/nightly.Testing
Running the pipeline for 2023 worked without any hard errors. There were a lot more warnings about incomplete data, which is expected.
Review estimate
10-15 min
Future work
Actually validate the early release outputs to ensure they are as complete as possible.
Checklist
black