-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Parthenon
frontend
#4323
Add Parthenon
frontend
#4323
Conversation
This is awesome! Thank you for contributing it. I'll provide a review ASAP. |
Congratz @pgrete ! I'll make a first pass for QA and will try to focus on stuff that isn't already reported by linters (see pre-commit.ci's report). |
I think we have one example of a framework-as-a-dataformat already: the At this point, and I'm not saying that to discourage this pull request, you may want to consider distributing the frontend as an extension package, which can help if frequent contributions are expected and you want to be free from our relatively slow release cycle.
yes that would be a necessary step for us to merge this PR. |
gamma_attrs = { | ||
key: val | ||
for key, val in self._handle["Params"].attrs.items() | ||
if key.endswith("AdiabaticIndex") | ||
} | ||
if "gamma" in self.specified_parameters: | ||
self.gamma = float(self.specified_parameters["gamma"]) | ||
elif len(gamma_attrs) >= 1: | ||
self.gamma = next(iter(gamma_attrs.values())) | ||
else: | ||
self.gamma = 5.0 / 3.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@forrestglines what's the intention behind this logic, i.e., why creating the gamma_attrs
rathern than checking if AdiabaticIndex
is already in Params
?
(same below for He_mass_fraction
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's due to the actual parameter being Hydro/AdiabaticIndex
in AthenaPK which will be different for the other Parthenon codes.
If this logic is too opaque, we should instead look for Hydro/AdiabaticIndex
for AthenaPK and implement the adiabatic index as needed for the other codes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As it's supposed to work right now, an adiabatic index passed in via gamma=
will take priority over an adiabatic index defined in the parameters of the Parthenon simulation, then 5./3.
if nothing else is found.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The He_mass_fraction
piece isn't working since at some point we stopped outputing He_mass_fraction
and started including mbar_over_kb
instead. We could back out He_mass_fraction
from mbar_over_kb
instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm trying to fix reading in He_mass_fraction
from mbar_over_kb
but I'm getting nonsensical results. I'll have to look at it later with fresh eyes. I'll push what I'm working with in case someone can see what I'm doing wrong.
Regarding the dataset, am I assuming correctly that the hdf5 file should not use (default) compression, i.e., |
@pgrete I think as long as it doesn't require |
I think that the reason CI is now failing is that #4344 was merged in between runs. Add |
I'm currently figuring out some useful tests to add (with the appropriate data file). |
Looks like I'm stuck. I first thought that the test are answer tests so that I should call them using a class (like Then I tried
so what am I missing? |
I'm a little stuck here. So I installed a clean env with python 3.8 and the "full" version of yt Trying the test now results in: $ pytest yt/frontends/parthenon/tests/test_outputs.py
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.8.16, pytest-7.2.2, pluggy-1.0.0 -- /home/pgrete/miniconda3/envs/py3.8/bin/python
cachedir: .pytest_cache
rootdir: /home/pgrete/src/yt, configfile: pyproject.toml
collected 0 items / 1 error
=========================================================================================== ERRORS ===========================================================================================
_______________________________________________________________ ERROR collecting yt/frontends/parthenon/tests/test_outputs.py ________________________________________________________________
yt/frontends/parthenon/tests/test_outputs.py:11: in <module>
from yt.utilities.answer_testing.framework import (
yt/utilities/answer_testing/framework.py:24: in <module>
from nose.plugins import Plugin
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/__init__.py:1: in <module>
from nose.core import collector, main, run, run_exit, runmodule
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/core.py:12: in <module>
from nose.loader import defaultTestLoader
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/loader.py:21: in <module>
from nose.importer import Importer, add_path, remove_path
../../miniconda3/envs/py3.8/lib/python3.8/site-packages/nose/importer.py:12: in <module>
from imp import find_module, load_module, acquire_lock, release_lock
../../miniconda3/envs/py3.8/lib/python3.8/imp.py:31: in <module>
warnings.warn("the imp module is deprecated in favour of importlib; "
E DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
================================================================================== short test summary info ===================================================================================
ERROR yt/frontends/parthenon/tests/test_outputs.py - DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 1 error in 0.29s ======================================================================================
What am I missing? |
Sorry for the late reply. |
So I'd call |
so clearly I'm not the expert here. Does the documentation help ? if not, maybe @Xarthisius is would be of better guidance. |
I figured it out (for reference I just added a plain Parthenon 2D test (which of course will fail given that the data is not available online). What do the next steps look like in general (in terms of when to upload/open a PR for the reference files) and the combined review? With respect to this PR, I plan to make the following two additions:
Do you recommend any other additional tests? |
AFAICT it seems mature enough that submitting datasets for upload now wouldn't be inappropriate.
I think that frontend tests should be as focused as possible on things that might go wrong specifically with the data format at hand. Tests exposing problems encountered during development are also very welcome, but don't feel like you need to be 100% exhaustive.
Eventually we'll want to have 2 approvals from maintainers, but you're close to getting mine already, since my initial comments were all addressed ! I'll need to review again, and I'm ready to do so as soon as needed. Feel free to ping me if you'd rather get my review before dataset are uploaded, otherwise I'll just wait for tests to go green ! |
In the process of adding data files of two other codes I'm currently wondering |
@pgrete For a), this should be feasible (@chrishavlin has done some on-the-same-line-of-thought work with integrating pressure to get spatial coordinates) but might need some collaborative thinking, which I'm up for. For b) this should be feasible by even in a worst case using multiple "field types". Either way, both are high priorities, so let's try! |
for mom_field_name in ["MomentumDensity", "cons_momentum_density_"]: | ||
mom_field = ("parthenon", f"{mom_field_name}{i+1}") | ||
if mom_field in self.field_list: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated quite a bit of the field logic (moving things to known_other_fields
above) and am now wondering if the registered/resolved fields are potentially already available here.
This would simplify logic here (and at several places below) wrt to just checking if ("gas", "momentum_density_x")
is known rather than lookups for our own names in self.field_list
.
Just pushed some quick changes so that coordinate ordering for cylindrical and spherical coordinates follows Athena++'s conventions, which AthenaPK will inherit. (This is the last functional addition we had planned for this PR) |
@@ -17,7 +17,9 @@ | |||
from .fields import ParthenonFieldInfo | |||
|
|||
geom_map = { | |||
"UniformCartesian": Geometry.CARTESIAN, | |||
"UniformCartesian": (Geometry.CARTESIAN,("x","y","z")), | |||
"UniformCylindrical": (Geometry.CYLINDRICAL,("r","theta","z")), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@forrestglines I suspect this line is the only reason for your latest commit, so I should point out that you should just use Geometry.POLAR
here, so you don't need to specify axis_order at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @neutrinoceros that would certainly allow a bit of code cleanup. Unfortunately changing to Geometry.POLAR
is breaking my scripts which use fixed resolution buffers. For example, this code to make a slice plot works with both Geometry.POLAR
and the current solution with Geometry.CYLINDRICAL
:
ds = yt.load(filename)
slc = yt.SlicePlot(ds, "z", ("gas", "density"))
but this code below only works with my current solution with Geometry.CYLINDRICAL
:
ds = yt.load(filename)
slc = yt.SlicePlot(ds, "z", ("gas", "density"))
frb = slc.data_source.to_frb(width=(4,"cm"),resolution=512,center=(0,0))
frb["r"]
with an error being thrown by pixelize_cylinder
File ~/installs-chicoma/miniconda3/envs/py39/lib/python3.9/site-packages/yt/visualization/fixed_resolution.py:600, in CylindricalFixedResolutionBuffer._generate_image_and_mask(self, item)
597 @override
598 def _generate_image_and_mask(self, item) -> None:
599 buff = np.zeros(self.buff_size, dtype="f8")
--> 600 mask = pixelize_cylinder(
601 buff,
602 self.data_source["r"],
603 self.data_source["dr"],
604 self.data_source["theta"],
605 self.data_source["dtheta"],
606 self.data_source[item].astype("float64"),
607 self.radius,
608 return_mask=True,
609 )
610 self.data[item] = ImageArray(
611 buff, units=self.data_source[item].units, info=self._get_info(item)
612 )
613 self.mask[item] = mask
File ~/installs-chicoma/miniconda3/envs/py39/lib/python3.9/site-packages/yt/utilities/lib/pixelization_routines.pyx:552, in yt.utilities.lib.pixelization_routines.pixelize_cylinder()
ValueError: Buffer has wrong number of dimensions (expected 2, got 1)
Suggestions? Is this something to be fixed in the frontend or in yt? I can provide the data file for this cylindrical dataset. The dataset will be included with our tests
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hard to tell for sure without the data file, but I can have a look if you can share it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I've pushed the small changes to switch from Geometry.CYLINDRICAL
to Geometry.POLAR
.
Here's the file I was testing on (with an .txt extension due to github's limitations)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thank you ! I can reproduce locally, which is a great start. I'll try to debug this today.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually it's a yt bug, I can also reproduce it with AMRVAC data
import yt
ds = yt.load_sample("bw_polar_2d")
slc = yt.SlicePlot(ds, "z", ("gas", "density"))
frb = slc.data_source.to_frb(width=(4, "cm"), resolution=512, center=(0, 0))
frb["r"]
I'll open a dedicated issue.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I saw issue #4790, the bug is not related to 2D datasets. I also explored this with a 3D dataset and got the same error.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fantastic that we have a bug fix started with this so soon. While this did eliminate the error with FRB, the polar coordinates are still not behaving like the cylindrical coordinates. In some comparison plots between Athena++ and AthenaPK using FRB's of a slice generated with the same code, I get plots looking like this:
My Parthenon frontend (using Geometry.POLAR
with your new bugfix) is only getting one quadrant while the Athena++ frontend (using Geometry.CYLINDRICAL
) is getting the full disk as expected.
Let me work on a better reproducer and perhaps one using an existing frontend in YT. I'm not convinced either way that this is now the Parthenon frontend's bug or Polar's bug but it's still a problem that changing from cylindrical to polar changes my plots.
just FYI, i hit the button for the tests to run, but you'll likely get unrelated failures after your merge with main (e.g., #4785 ). |
I've pushed @neutrinoceros 's bug fix to this PR although I'm still working on Polar. You can advise on how you'd like me to fix the git history later on if the merges get messy. |
Don't worry about it. It's not blocking this PR. I re-triggered docs build and it will likely pass this time. |
Not sure that's a deterministic error. Let me try again: |
Docs built alright but this time regular tests failed (also flaky) |
All green now ! I hope I'll find time to review this week, but with the imminent release of a major numpy release candidate I may not. Thank you for your patience ! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally looks good to me! Left a number of questions and comments, but nothing too critical.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for taking so long to get my review in. I note that there's also another, mostly unaddressed (from what I can see) review from @chrishavlin. Both need to be addressed before we can merge this.
Thank you for your patience !
Alright, I think I've addressed all review comments (@neutrinoceros @chrishavlin ) pending potential replies by you to the unresolved ones. From my point of view this should be ready to go (assuming all tests pass). What are the chances this PR will make it to the 4.4 release? |
@pgrete I will also take a look at this tomorrow and I think that we should make this a priority for the 4.4 release. |
I would normally oppose stacking more PRs (especially large ones) to a release's milestone at the last minute, but this one has been waiting for so long that it makes sense to try and get it in. |
That'd be awesome. We've been using this branch for quite some time now without issues and given that it doesn't change anything in the yt core, I think it should not pose any potential harm to the release code base as a whole. |
Indeed. Though, if for any reason we cannot make it, you guys could also consider packaging this frontend as an extension |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great from my perspective, thanks for addressing my comments @pgrete (edit: apologies, tagged the wrong person at first!!)
also +1 for making this a priority for 4.4. |
"@pgrete dismissed @chrishavlin 's stale review via dd42468" <- I don't know where/how that came from. I just pushed a one line change removing a "Todo comment" I already addressed yesterday. |
This can be caused by the branch protection settings: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#require-pull-request-reviews-before-merging.
|
@pgrete ya, no worries, that's cause of yt's repo config. I'll hit approve again, and you can consider this my blanket approval if it gets dismissed again :) |
Alright, I think all my comments have been addressed, and with 2 other approvals I feel like there's no reason to hold this back any longer, so let's get it in ! Thank you @pgrete and @forrestglines for your dedication and patience with the process. We'll be trying to get yt 4.4 out the door very shortly, so stay tuned ! |
Thanks for pulling this in on short notice! I'm very happy to update our downstream notes to pull the main (and future yt 4.4) version rather than a custom branch. |
PR Summary
Add
yt
frontend for codes based on theParthenon
AMR framework (/~https://github.com/parthenon-hpc-lab/parthenon) including AthenaPK (/~https://github.com/parthenon-hpc-lab/athenapk), Phoebus (/~https://github.com/lanl/phoebus) or KHARMA (/~https://github.com/AFD-Illinois/kharma)Note that the mesh structure in
Parthenon
is heavily derived fromAthena++
, so that's why we used theAthena++
frontend as baseline.We've been using this frontend for quite some time now (without any recent major changes) so we consider it stable enough to open it for review and merge to
yt
proper.At this point I'm looking for
PR Checklist