stantargets: reproducible Stan pipelines at scale

[wlandau]

Submitting Author: Will Landau (@wlandau)
Repository: /~https://github.com/wlandau/stantargets
Version submitted: 0.0.0.9000
Editor: @melvidoni
Reviewers: @sakrejda @mattwarkentin

Due date for @sakrejda: 2021-03-31

Due date for @mattwarkentin: 2021-03-31
Archive: TBD
Version accepted: TBD

---

• Paste the full DESCRIPTION file inside a code block below:

Package: stantargets
Title: Targets for Stan Workflows
Description: Bayesian data analysis usually incurs long runtimes
  and cumbersome custom code. A specialized pipeline toolkit for
  Bayesians, the 'stantargets' R package leverages
  'targets' and 'cmdstanr' to ease these burdens.
  'stantargets' makes it super easy to set up useful scalable
  Stan pipelines that automatically parallelize the computation
  and skip expensive steps when the results are already up to date.
  Minimal custom code is required, and there is no need to manually
  configure branching, so usage is much easier than 'targets' alone.
  'stantargets' can access all of 'cmdstanr''s major algorithms
  (MCMC, variational Bayes, and optimization) and it supports
  both single-fit workflows and multi-rep simulation studies.
  For the statistical methodology, please refer to 'Stan' documentation
  (Stan Development Team 2020) <https://mc-stan.org/>.
Version: 0.0.0.9000
License: MIT + file LICENSE
URL: https://wlandau.github.io/stantargets/, /~https://github.com/wlandau/stantargets
BugReports: /~https://github.com/wlandau/stantargets/issues
Authors@R: c(
  person(
    given = c("William", "Michael"),
    family = "Landau",
    role = c("aut", "cre"),
    email = "will.landau@gmail.com",
    comment = c(ORCID = "0000-0003-1878-3253")
  ),
  person(
    family = "Eli Lilly and Company",
    role = "cph"
  ))
Depends:
  R (>= 3.5.0)
Imports:
  cmdstanr (>= 0.2.0),
  digest (>= 0.6.21),
  fst (>= 0.9.4),
  posterior (>= 0.1.2),
  purrr (>= 0.3.4),
  qs (>= 0.14.1),
  rlang (>= 0.4.8),
  stats,
  targets (>= 0.0.1),
  tarchetypes (>= 0.0.1),
  tibble (>= 3.0.4),
  tools
Suggests:
  dplyr (>= 1.0.2),
  fs (>= 1.5.0),
  knitr (>= 1.28),
  R.utils (>= 2.10.1),
  rmarkdown (>= 2.1),
  testthat (>= 3.0.0),
  visNetwork (>= 2.0.9),
  withr (>= 2.1.2)
Remotes:
  stan-dev/cmdstanr,
  stan-dev/posterior
SystemRequirements: CmdStan >= 2.25.0
Encoding: UTF-8
Language: en-US
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.1.1
VignetteBuilder: knitr
Config/testthat/edition: 3

Scope

• Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):

  • data retrieval
  • data extraction
  • data munging
  • data deposition
  • workflow automation
  • version control
  • citation management and bibliometrics
  • scientific software wrappers
  • field and lab reproducibility tools
  • database software bindings
  • geospatial data
  • text analysis

• Explain how and why the package falls under these categories (briefly, 1-2 sentences):

stantargets is very similar to jagstargets (#425). stantargets leverages the existing workflow automation capabilities of targets to orchestrate computation and skip up-to-date tasks in Bayesian data analysis pipelines. stantargets reduces the burden of user-side custom code that targets would otherwise require, which helps free statisticians to focus more on the models and less on the software engineering.

• Who is the target audience and what are scientific applications of this package?

stantargets is for Bayesian statisticians who develop and run Stan models. Example workflows range from individual analyses of clinical data to large-scale simulation-based calibration studies for validation.

• Are there other R packages that accomplish the same thing? If so, how does yours differ or meet our criteria for best-in-category?

targets already provides the same kind of workflow automation, but it requires more custom code to set up a workflow. stantargets uses specialized domain knowledge to make this process easier. Packages rstan and cmdstanr interface with Stan but do not provide the same kind of workflow automation. In light of the recent preprint by Gelman et al. (2020), I believe the Stan Development Team would be very interested in this kind of workflow automation.

• (If applicable) Does your package comply with our guidance around Ethics, Data Privacy and Human Subjects Research?

N/A

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

N/A

Technical checks

Confirm each of the following by checking the box.

• I have read the guide for authors and rOpenSci packaging guide.

This package:

• does not violate the Terms of Service of any service it interacts with.
• has a CRAN and OSI accepted license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions, created with roxygen2.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, including reporting of test coverage using services such as Travis CI, Coveralls and/or CodeCov.

Publication options

• Do you intend for this package to go on CRAN?

• Do you intend for this package to go on Bioconductor?

• Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

If JOSS is still an option, I would like to publish there. I have prepared a manuscript at /~https://github.com/wlandau/stantargets/blob/main/inst/paper.md.

MEE Options

• The package is novel and will be of interest to the broad readership of the journal.
• The manuscript describing the package is no longer than 3000 words.
• You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
• (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
• (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
• (Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

• I agree to abide by rOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

[melvidoni]

Editor checks:

• Documentation: The package has sufficient documentation available online (README, pkgdown docs) to allow for an assessment of functionality and scope without installing the package.
• Fit: The package meets criteria for fit and overlap
• Automated tests: Package has a testing suite and is tested via a CI service.
• License: The package has a CRAN or OSI accepted license
• Repository: The repository link resolves correctly

---

Editor comments

Thank you for submitting it! I'll be your handling editor this time. Goodpractice is very happy about it:

♥ Aha! Fantastic package! Keep up the prime work!

There are no spelling checks, and covr is also happy. Good job!

I will be searching for reviewers then.

---

Reviewers: @sakrejda @mattwarkentin
Due date: Wednesday, March 31st

[wlandau]

Thanks, @melvidoni! Looking forward to the review process. Please let me know if you would like me to suggest potential reviewers.

[melvidoni]

Quick update. I have contacted about 15 people, and received no acceptance to review. Please bear with me while I find someone. And yes, I have your master list of suggestions.

[wlandau]

Thanks for the update, @melvidoni. I do understand this may take some time.

And yes, I have your master list of suggestions.

Do you mean the folks I mentioned at #401 (comment) for targets? I actually think about stantargets a bit differently because of its focus on Bayesian Statistics. (Likewise for jagstargets in #425.) For my own learning, it would be amazing to include someone from the Stan Development Team. If not, then I think practicing Bayesian statisticians, particularly Stan users, would have a lot to add.

[melvidoni]

Hello @wlandau. Yes, that list. We curated it with some editors, and also have our own list of people. However, given the current world circumstances, I am afraid that finding reviewers may take longer than expected.

[wlandau]

Yes, that list.

Oh good, glad we are on the same page. Sorry, I may have misremembered the earlier parts of this thread.

However, given the current world circumstances, I am afraid that finding reviewers may take longer than expected.

I completely understand.

For what it's worth, dependency packages cmdstanr and posterior are not yet on CRAN, so this review process is extremely unlikely to bottleneck any of my plans. And unlike targets, I am not developing any packages that depend on stantargets.

[mdscheuerell]

Perhaps Krzysztof Sakrejda would be available?

[melvidoni]

Perhaps Krzysztof Sakrejda would be available?

Hello @sakrejda are you interested?

[sakrejda]

Hi Melina, I am interested and the three-week time frame is reasonable for me. I'm not the right person to say much about whether the package aligns well with how targets itself is commonly used but should be able to provide feedback otherwise. If you haven't asked @tjmahr yet he might be interested in reviewing.

[melvidoni]

Hello @sakrejda thank you kindly! @mattwarkentin will also be reviewing this package.

Please remember to follow the guidelines for the review: https://devguide.ropensci.org/reviewerguide.html

The review deadline will be Wednesday 31st.

[tjmahr]

If you haven't asked @tjmahr yet he might be interested in reviewing.

Thanks for the mention, but I just did a targets/tarchetypes review so a fresher set of eyes would be better. Please do consider me for brmstargets 😉 though.

[sakrejda]

@wlandau to make my life easier reviewing, is there a particular overall approach to the API, which functions you are exposing to users and which ones are considered "internal"? I don't immediately recognize the structure.

To be more specific, in the stan compilation functions there are quite a few layers of functions before you actually get down to calling the stan compilation function and it would be helpful to have an idea of why each layer is there. Just to be clear, I don't disagree and I sometimes take a similar approach, but I'm not sure what the goals are there. I assume this is something that happens consistently across the package. Can you reply here pointing out the reasons for the different layers?

[wlandau]

Most stantargets interface functions create target objects. Each target object has a name, some settings, and an R command that will run later (during tar_make()). Putting these things together involves some layers of setup and metaprogramming. These are the layers for standalone compilation of models:

1. tar_stan_compile(): diffuse/quote any user-supplied language objects and move on to the next layer. In other words, users should be able to write tar_stan_compile(target_name, "model_file.stan") without risk of immediately evaluating target_name as a variable. This may seem trivial in this particular example, but for other target factories like tar_stan_mcmc(), we also need capture the user-defined expression that generates the Stan data (and support tidy evaluation, which is best done in the first call frame).
2. tar_stan_compile_raw(): Gather all the settings and expressions from (1) into a target object. This layer takes a 10000-foot view of the whole target and cannot afford to go down any rabbit holes. For the sake of keeping a clean codebase and breaking down the problem into manageable pieces, it delegates the more specialized tasks to subsequent deeper layers. In targets and packages I build on top of targets, the "_raw" suffix indicates this intent.
3. tar_stan_compile_command(): In stantargets, not all the R expressions come directly from user input. Target factories in stantargets automatically supply many of these expressions to reduce the amount of custom code the user needs to write. The metaprogramming to create an expression from non-language input is complicated enough to need its own layer.
4. tar_stan_compile_run(): actually compile a Stan model when called. The "*_run()" functions significantly reduce burden of metaprogramming in layers like (3). Rather than bootstrapping a large elaborate code chunk in (3), (3) can simply create a function call to invoke (4). (4) usually has no metaprogramming, so it is a much better place to manage the complicated details of how to properly invoke cmdstanr.

(1) is the only layer the user needs to see. (4) still needs to be exported because functions like tar_stan_compile_run() are part of the commands in the target objects, but I hide these functions from the user using @keywords internal.

[sakrejda]

When running devtools::test tests fail unless cmdstanr::install_cmdstan is run first, I think it would be helpful to check the cmdstan installation first. There is the cmdstanr::check_cmdstan_toolchain function that could be run in an initial test, with the fix=TRUE option. This is really a question: are you thinking you'd like to avoid doing this type of check at all because it's too fragile or do you think it would be a good idea to include it. I would lean towards including it since the functionality in the underlying key package is there and could be improved in cmdstanr if needed.

[wlandau]

Good point. keras handles a similar situation with skip_if_not_tensorflow_version (), and testthat::skip_if_not_installed() for Suggests: packages is a community-accepted pattern. Personally, I think this makes sense. Do you agree?

[sakrejda]

That's a great solution, also thanks for the prompt replies!

[wlandau]

Great. Should be fixed now.

[sakrejda]

Aside that's really for the base targets pacakge, but mentioning it here since it affects running the README.md example on one of my systems:

> targets::tar_visnetwork()
Error in viewer(index_html) : 
  'browser' must be a non-empty character string

From the error it's unclear that the answer is option(browser = "<path/to/broswer>"). It would be pretty straightforward to catch the lack of a default browser so that the message could specifically say "A default browser is required to view the network visualization, set a default browser with 'options(browser = "</path/to/browser>")`" or something similar. From the current message I wasn't sure if I need to pass a path to the function or set an option and if I was setting an option I wasn't sure what the option was called.

[sakrejda]

In the README.md I initially got an error b/c x.stan was not found so since I was running from the package root I changed it to inst/example.stan and that made tar_make() complete successfully. For a slightly intimidating package it would be helpful to have the default example in the README.md just work. I think you could retrieve the .stan file through a call to system.file so that the example runs without any changes. It would give a new user some confidence that they could figure the package out.

[sakrejda]

The README.md example seems to reference an internal data generating function for the example rather than the function included in the example (not sure I got that right but I think I did). Here's what I modified it to in order to have it working:

library(targets)
library(stantargets)

generate_data <- function(n = 10) {
  true_beta <- stats::rnorm(n = 1, mean = 0, sd = 1)
  x <- seq(from = -1, to = 1, length.out = n)
  y <- stats::rnorm(n, x * true_beta, 1)
  list(n = n, x = x, y = y, true_beta = true_beta)
}

list(
  tar_stan_mcmc(test, "inst/example.stan", generate_data())
)

Also I wasn't sure exactly how stantargets relates to targets and it would be helpful in the README.md to have a short paragraph (maybe before the "prerequisites" section?) that states that the tar_stan_mcmc target defines sub-targets following a specific naming convention which allow you to schedule chains and access different components of the results. I don't want to nit-pick but taking the perspective of a naive user (I know Stan but I'm new-ish to targets itself) the auto-generation of a bunch of new targets is not obvious and I wasn't initially understanding that I could use tar_read on the auto-generated targets.

[wlandau]

The README.md example seems to reference an internal data generating function for the example rather than the function included in the example

Thanks, just updated the README to use generate_data(). (tar_stan_example_data() is exported, but generate_data() is still better since most users will write their own data functions.)

Also I wasn't sure exactly how stantargets relates to targets and it would be helpful in the README.md to have a short paragraph (maybe before the "prerequisites" section?) that states that the tar_stan_mcmc target defines sub-targets following a specific naming convention which allow you to schedule chains and access different components of the results.

I try to address these points early in the first vignette. I would prefer to keep the README at a high level in order to keep it digestible for new users and to avoid repeating documentation.

[sakrejda]

There's no way to pass additional arguments here: /~https://github.com/wlandau/stantargets/blob/522a94c3b0941aba10d49bfc756b2a6def1a5577/R/tar_stan_compile.R#L155

That means the pedantic and threads arguments currently can't be passed down. What do you think about adding a compile_dot_args to the tar_stan_compile function that would get passed down to the cmdstanr::cmdstan_model function?

[sakrejda]

I guess that's just a maintenance issue that will come up repeatedly with the way that arguments are currently structured (explicitly passed down and enumerated at each level). In some ways it's helpful, but it also means a user can end up stuck. What do you think about this issue? Is it an explicit tradeoff you're making?

[wlandau]

Thanks again for reviewing, @mattwarkentin! The spelling mistake is fixed in ropensci/stantargets@296fcac.

[melvidoni]

Hello @wlandau. I see you've made great work addressing all comments.

I see that @mattwarkentin has approved the package, what about @sakrejda? What do you think?

[sakrejda]

@melvidoni thanks for the reminder, I wasn't sure how to signal approval, is there something specific? @wlandau addressed all of my comments more than adequately.

[melvidoni]

Approved! Thanks @wlandau for submitting and @mattwarkentin @sakrejda for your reviews!

To-dos for @wlandau:

• Transfer the repo to rOpenSci's "ropensci" GitHub organization under "Settings" in your repo. I have invited you to a team that should allow you to do so. You'll be made admin once you do.
• Fix all links to the GitHub repo to point to the repo under the ropensci organization.
• Delete your current code of conduct file if you had one since rOpenSci's default one will apply, see https://devguide.ropensci.org/collaboration.html#coc-file
• If you already had a pkgdown website and are ok relying only on rOpenSci central docs building and branding,
  • deactivate the automatic deployment you might have set up
  • remove styling tweaks from your pkgdown config but keep that config file
  • replace the whole current pkgdown website with a redirecting page
  • replace your package docs URL with https://docs.ropensci.org/package_name
  • In addition, in your DESCRIPTION file, include the docs link in the URL field alongside the link to the GitHub repository, e.g.: URL: https://docs.ropensci.org/foobar (website) /~https://github.com/ropensci/foobar
• Fix any links in badges for CI and coverage to point to the ropensci URL. We no longer transfer Appveyor projects to ropensci Appveyor account so after transfer of your repo to rOpenSci's "ropensci" GitHub organization the badge should be [![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/ropensci/pkgname?branch=master&svg=true)](https://ci.appveyor.com/project/individualaccount/pkgname). If Appveyor does not pick up new commits after transfer, you might need to delete and re-create the Appveyor project. (Repo transfers are smoother with GitHub Actions)
• Please check you updated the package version post-review version updated and that you documented all changes in NEWS.md
• We're starting to roll out software metadata files to all ropensci packages via the Codemeta initiative, see /~https://github.com/ropensci/codemetar/#codemetar for how to include it in your package, after installing the package - should be easy as running codemetar::write_codemeta() in the root of your package.

Should you want to acknowledge your reviewers in your package DESCRIPTION, you can do so by making them "rev"-type contributors in the Authors@R field (with their consent). More info on this here.

Welcome aboard! We'd love to host a post about your package - either a short introduction to it with an example for a technical audience or a longer post with some narrative about its development or something you learned, and an example of its use for a broader readership. If you are interested, consult the blog guide, and tag @stefaniebutland in your reply. She will get in touch about timing and can answer any questions.

We've put together an online book with our best practice and tips, this chapter starts the 3d section that's about guidance for after onboarding. Please tell us what could be improved, the corresponding repo is here.

[wlandau]

Wonderful, @melvidoni! Thank you for serving as editor, and thanks again @mattwarkentin and @sakrejda for reviewing! I will work on these to-dos.

[wlandau]

@melvidoni, I believe I have implemented most of the items you identified. If you would add me as an admin of /~https://github.com/ropensci/stantargets, I can deactivate the current GitHub Pages setup and migrate to https://docs.ropensci.org.

[melvidoni]

Done, please check now!

[wlandau]

Thanks, @melvidoni. I deactivated GitHub Pages and removed the gh-pages branch. Does anything else need to be done for https://docs.ropensci.org/stantargets/ to appear?

[melvidoni]

Thanks, @melvidoni. I deactivated GitHub Pages and removed the gh-pages branch. Does anything else need to be done for https://docs.ropensci.org/stantargets/ to appear?

Hello, apologies for the delay in coming back to you. There was a wrong GitHub token somewhere so things were stalled. I spoke with the Editors and believe the website should appear today or tomorrow.

[jeroen]

See: https://dev.ropensci.org/job/stantargets/1/console

The docs didn't appear because the fails with:

-- Building articles -----------------------------------------------------------
Writing 'articles/index.html'
Reading 'vignettes/mcmc.Rmd'
-- RMarkdown error -------------------------------------------------------------
Quitting from lines 120-121 (mcmc.Rmd) 
Error : callr subprocess failed: CmdStan path has not been set yet. See ?set_cmdstan_path.
Visit https://books.ropensci.org/targets/debugging.html for debugging advice.
--------------------------------------------------------------------------------
Error : Failed to render RMarkdown

Is this something we should install on the docs server? Or maybe you could do something like this in the vignette:

if(identical(Sys.getenv("IN_PKGDOWN"), "true")){
  cmdstandr::install_cmdstan()
}

[wlandau]

Thanks, that explains it. I will try the IN_PKGDOWN workaround for now.

More generally, do package developers have control over custom system dependencies in ropensci.org builds like we do in GHA? Not sure how I feel about a system-wide CmdStan installation (it is developing rapidly) and the IN_PKGDOWN workaround assumes the vignettes always build first.

[wlandau]

Looks to have worked. The website is up now.

@melvidoni, have I missed anything?

[jeroen]

The image that we use for building the docs has all the same system dependencies pre-installed as on the CRAN servers. So if your package is intended to work on CRAN, it should also work on our build servers.

You could also move this line to your package onLoad function so that it runs first.

[wlandau]

The image that we use for building the docs has all the same system dependencies pre-installed as on the CRAN servers. So if your package is intended to work on CRAN, it should also work on our build servers.

That's reasonable.

You could also move this line to your package onLoad function so that it runs first.

I will consider it. So far, I have been trying to follow what cmdstanr does. I am not sure when cmdstanr will reach CRAN, and I do wonder how @jgabry and @rok-cesnovar et al. plan to handle the dependency on CmdStan when the time comes for a release.

[jgabry]

and I do wonder how @jgabry and @rok-cesnovar et al. plan to handle the dependency on CmdStan when the time comes for a release.

Probably similarly to how RStudio handles (or handled) it's tensorflow package. I haven't checked recently so maybe they've changed it, but it used to be that no tests involving tensorflow were run on CRAN, which is what we'd have to do with CmdStan since CRAN won't have a CmdStan installation. Is that what you were referring to?

[wlandau]

Yeah, and I also noticed tensorflow has no vignettes/ folder. cmdstanr might be able to host its vignettes elsewhere, but built-in vignettes are a requirement for rOpenSci packages like stantargets.

Actually, I just remembered that stantargets tries to suppress code in the vignettes if it detects CRAN. Hopefully that will work.

if (identical(Sys.getenv("NOT_CRAN", unset = "false"), "false")) {
  knitr::opts_chunk$set(eval = FALSE)
}

[jgabry]

Yeah we also have some logic in the vignettes to prevent them from running on CRAN. We've used similar logic with no issues before in packages already on CRAN like rstanarm and brms where the vignettes could in theory be run on CRAN but take too long.

[melvidoni]

Looks to have worked. The website is up now.

@melvidoni, have I missed anything?

Hello @wlandau it looks good! Great work! @stefaniebutland we can maybe make a blogpost about this package?

[stefaniebutland]

we can maybe make a blogpost about this package?

absolutely. blog post or tech note according to Will's preference 🙂
https://blogguide.ropensci.org/

[wlandau]

Thanks for inviting me to write again! It may take time for me to free up enough bandwidth, but I am interested.

[wlandau]

A couple updates:

1. I got permission to release /~https://github.com/wlandau/stantargets-example-validation, the example stantargets project. (@mattwarkentin mentioned the idea in his review.) It has the example code from the simulation study vignette and an RStudio Cloud workspace to let people run it in a web browser. /~https://github.com/wlandau/stantargets-example-validation is just like /~https://github.com/wlandau/targets-stan, but with a much shorter and simpler _targets.R file because of the abstractions in stantargets.
2. The JOSS manuscript was just approved!

[wlandau]

On reflection, I think it might be nice to broaden the tech note (or blog post) to Bayesian data analysis pipelines in general and also include jagstargets when it completes its review.

[stefaniebutland]

blog post!