The coding work for this issue is complete. I am waiting for the rstprod group to be added to Niagara before updating the branch, completing testing, and submitting a PR.

@arunchawla-NOAA Can we submit a request to get rstprod on Niagara? It would help with the lack of direct HPSS access from Orion (and Hercules). Thanks!

This will remain blocked until rstprod is added to Niagara.

Curious about the status of this as we evaluate whether we can use Orion/Hercules for future GFSv17 prototypes/retros. Was there ever a request made to get rstprod on Niagara?

Tagging @JacobCarley-NOAA on this as acting EIB chief.

@KateFriedman-NOAA Do you know if Arun ever made the request for rstprod on Niagara?

Do you know if Arun ever made the request for rstprod on Niagara?

To my knowledge he never submitted the request. @JacobCarley-NOAA is this something you could submit?

@KateFriedman-NOAA @DavidHuber-NOAA @JessicaMeixner-NOAA @JacobCarley-NOAA

Good news everyone: Rstprod has now been added to Niagara.

Hooray! Thanks for the update @CatherineThomas-NOAA !

I am working with Georgy Fekete to establish a workflow for these transfers (RDHPCS ticket number 2024061454000228).

I met with George Fekete today who showed off a generic tool for pushing and pulling from HPSS via Niagara (or any HPSS-connected machine). I will be testing the tool out then start implementing it into the workflow.

Hi @DavidHuber-NOAA. I'm curious as to why this has moved from Todo to Blocked. Is there anything we can help with/escalate?

Hi @CatherineThomas-NOAA I am waiting on code delivery from George for a generic transfer tool. Once I have that, I should be able to move forward. I will ping him today and see where that is at.

@KateFriedman-NOAA @WalterKolczynski-NOAA For the gfs_globus, etc, jobs, I would like to store the dictionaries created by the associated gfs_arch, etc, tasks as YAMLs. These dictionaries contain the name of the tarball and whether they have rstprod data or not. What COM location makes sense for this? I was thinking about COM_CONF, but I'm open to suggestions.

COM_CONF seems good to me. The dictionaries will be removed at the end of the job or when?

OK, sounds good.

At the end of the job would be fine, I think. If globus needs to be run again, the arch tasks would need to be run again to generate the YAMLs, but those are lightweight tasks.