Move options to pandoc-defaults.yaml #290

dhimmel · 2019-12-19T16:39:17Z

Pandoc has introduced a --defaults option to "specify a set of default option settings". The benefits for us are a few-fold:

reduce duplication of pandoc options between various build commands
move to more of a configuration file model with the eventual goal of being able to build repositories with only content files and nothing else.

AppVeyorBot · 2019-12-19T16:41:36Z

AppVeyor build 1.0.62 for commit 6486c3b by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

dhimmel · 2019-12-19T16:45:59Z

Here's a local WeasyPrint build of 6486c3b: manuscript.pdf

agitter · 2019-12-19T16:51:43Z

The simplification and elimination of redundancy in the build script is very nice. Could you please test a CI or local DOCX build as well?

dhimmel · 2019-12-19T17:25:11Z

Here's the output from a local BUILD_DOCX=true build: manuscript.docx. Some formatting issues, but I don't think those are new.

stderr output below

Exporting Word Docx manuscript
[INFO] Not rendering RawInline (Format "html") "<small>"
[INFO] Not rendering RawInline (Format "html") "<em>"
[INFO] Not rendering RawInline (Format "html") "</em>"
[INFO] Not rendering RawInline (Format "html") "</small>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<small>"
[INFO] Not rendering RawInline (Format "html") "</small>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<small>"
[INFO] Not rendering RawInline (Format "html") "</small>"
[INFO] Not rendering RawInline (Format "html") "<sup>"
[INFO] Not rendering RawInline (Format "html") "</sup>"
[INFO] Not rendering RawInline (Format "html") "<sub>"
[INFO] Not rendering RawInline (Format "html") "</sub>"
[INFO] Not rendering RawBlock (Format "html") "<!-- an arbitrary comment. visible in input, but not visible in output. -->"
[INFO] Fetching /~https://github.com/manubot/resources/raw/15493970f8882fce22bef829619d3fb37a613ba5/test/square.png...
[INFO] Fetching /~https://github.com/manubot/resources/raw/15493970f8882fce22bef829619d3fb37a613ba5/test/wide.png...
[INFO] Fetching /~https://github.com/manubot/resources/raw/15493970f8882fce22bef829619d3fb37a613ba5/test/tall.png...
[INFO] Fetching https://raw.githubusercontent.com/manubot/resources/master/test/vector.svg?sanitize=true...
[INFO] Not rendering RawInline (Format "html") "<!-- $colspan=\"2\" -->"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-exclamation-triangle\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<!-- $id=\"element_id\" class=\"some_class\" $style=\"color: #ad1457; margin-left: 40px;\" $disabled=\"true\" $title=\"a paragraph of text\" $data-color=\"red\" -->"
[INFO] Not rendering RawBlock (Format "html") "<!-- include the Font Awesome library, per: https://fontawesome.com/start -->"
[INFO] Not rendering RawInline (Format "html") "<link rel=\"stylesheet\" href=\"https://use.fontawesome.com/releases/v5.7.2/css/all.css\">"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-check\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-question\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-star\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-bell\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-times-circle\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-ellipsis-h\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-scroll fa-lg\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-info-circle fa-lg\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawInline (Format "html") "<i class=\"fas fa-ban fa-lg\">"
[INFO] Not rendering RawInline (Format "html") "</i>"
[INFO] Not rendering RawInline (Format "html") "<br>"
[INFO] Not rendering RawBlock (Format "html") "<!-- Explicitly insert bibliography here -->"

dhimmel · 2019-12-19T17:34:04Z

One question is whether we should have additional default files for specific outputs.

For example, we could have pandoc-defaults-html.yaml, pandoc-defaults-pdf-athena.yaml, pandoc-defaults-pdf-weasyprint.yaml, and pandoc-defaults-docx.yaml. This would make it so the pandoc calls in build.sh only include --defaults (specified multiple times). This would make it easier to migrate the pandoc calls to our python package with the end goal of getting rid of build.sh and moving to a single command to process and build the manuscript. CC @vincerubinetti

Nebucatnetzer · 2019-12-19T17:37:06Z

Looks nice. Some mention in the README, USAGE might be good?
Another thing that came to mind, can one include or reference a file which overwrites the defaults?
In Ansible you can provide sane default values which the user can easily overwrite and if he removes his customisation the default values get applied again.
I haven't looked at the pandoc documentation yet.

Btw. please excuse that I'm a bit slow with responding.
I haven't been on my computer since Wednesday and doing code stuff on the phone or iPad is not that much fun :)

Edit: had a quick look and saw this:

Default files can be placed in the defaults subdirectory of the user data directory and used from any directory. For example, one could create a file specifying defaults for writing letters, save it as letter.yaml in the defaults subdirectory of the user data directory, and then invoke these defaults from any directory using pandoc --defaults letter or pandoc -dletter.

When multiple defaults are used, their contents will be combined.

Which is nice, it would allow, as with the thoughts about the theme to have a common.yml file and the ln maybe a docx.yaml, pdf.yaml, etc.

In addition that values don't get overwritten but get applied multiple times.

For reference:
https://pandoc.org/MANUAL#default-files

AppVeyorBot · 2019-12-19T17:57:02Z

AppVeyor build 1.0.63 for commit 7d9224b by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

AppVeyorBot · 2019-12-19T18:17:58Z

AppVeyor build 1.0.65 for commit c1636e4 by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

Added by @dhimmel from local builds:

weasyprint manuscript.pdf
manuscript.docx

AppVeyorBot · 2019-12-19T20:07:47Z

AppVeyor build 1.0.66 for commit b068b9e by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

dhimmel · 2019-12-20T16:03:27Z

can one include or reference a file which overwrites the defaults

I think there are two things a user may want to do:

discard an entire defaults file in favor of their own file. For example, do not use html.yaml at all and instead use custom-html.yaml
apply additional defaults files on top of the ones already specified by Manubot. This equates to specifying --defaults additional times on top of the Manubot builtin --defaults files. Pandoc has a behavior for merging defaults, but this is not entirely consistent and well documented yet as per combination of several defaults files? jgm/pandoc#5988 (will likely improve).

I'd like to look into whether we can accomplish 2 without requiring any changes to build.sh. For example, whether we could do something equivalent to --defaults=html*.yaml (or do this in our Python tooling eventually)

please excuse that I'm a bit slow with responding

@Nebucatnetzer you're under no obligation to respond at all... so nothing to excuse 😺

AppVeyorBot · 2019-12-20T17:02:46Z

AppVeyor build 1.0.67 for commit 18b78fd by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

Nebucatnetzer · 2019-12-21T09:10:37Z

I think there are two things a user may want to do:
1. discard an entire defaults file in favor of their own file. For example, do not use `html.yaml` at all and instead use `custom-html.yaml`

2. apply additional defaults files on top of the ones already specified by Manubot. This equates to specifying `--defaults` additional times on top of the Manubot builtin `--defaults` files. Pandoc has a behavior for merging defaults, but this is not entirely consistent and well documented yet as per [jgm/pandoc#5988](/~https://github.com/jgm/pandoc/issues/5988) (will likely improve).
I'd like to look into whether we can accomplish 2 without requiring any changes to build.sh. For example, whether we could do something equivalent to --defaults=html*.yaml (or do this in our Python tooling eventually)

please excuse that I'm a bit slow with responding

@Nebucatnetzer you're under no obligation to respond at all... so nothing to excuse smiley_cat

Aye something like --defaults=pandoc-custom/*.yaml might work, I reckon a sub directory would be a bit cleaner.
Maybe we would've to include an empty YAML file so that it doesn't trip when there aren't any customisations present.

As for 1. maybe we could make the build.sh a bit more complex so that if it finds a YAML with the exact same name as a defaults file (e.g. html.yaml) it would replace the defaults and if it has a different name (e.g. custom-html.yml) it would be added on top?

dhimmel · 2020-01-02T17:34:17Z

As for 1. maybe we could make the build.sh a bit more complex

I looked briefly into using bash / shell commands to detect files matching a collection of patterns and then transforming each file into a pandoc --defaults argument. There didn't seem to be an easy / readable solution.

This makes me lean towards implementing this logic in Python. So the proposal would be:

merge this PR with the current design where all --defaults files must be manually specified when calling pandoc.
work on python code to detect default files (either additional or replacement files) in the manubot package.

Nebucatnetzer · 2020-01-03T10:12:01Z

Sounds good to me :)

agitter

I have one suggestion about making the file paths more flexible. Everything else looks good to me.

agitter · 2020-01-03T13:32:38Z

build/build.sh

-  --include-after-body=build/plugins/analytics.html \
-  --output=output/manuscript.html \
-  "$INPUT_PATH"
+  --defaults=build/pandoc-defaults/common.yaml \


The proposal in #261 (comment) would give more flexibility for specifying where the defaults files reside.

@agitter are you suggesting that we allow setting the directory like:

PANDOC_DEFAULTS_DIR="${PANDOC_DEFAULTS_DIR:-build/pandoc-defaults}"

This means there's a single environment variable and that users could do:

cp build/pandoc-defaults build/custom-pandoc-defaults # edit files in build/custom-pandoc-defaults export PANDOC_DEFAULTS_DIR=build/custom-pandoc-defaults

Or do you think each file should have it's own variable like PANDOC_DEFAULTS_COMMON_PATH?

I propose that initially we do:

PANDOC_DEFAULTS_DIR="${PANDOC_DEFAULTS_DIR:-build/pandoc-defaults}"

Your example customization above looks pretty straightforward.

That may end up being a placeholder depending what changes as part of

work on python code to detect default files (either additional or replacement files) in the manubot package.

AppVeyorBot · 2020-01-04T21:40:40Z

AppVeyor build 1.0.68 for commit 220998d by @dhimmel is now complete. The rendered manuscript from this build is temporarily available for download at:

This build is based on 06fb01d. This commit was created by the following Travis CI build and job: https://travis-ci.com/manubot/rootstock/builds/143118608 https://travis-ci.com/manubot/rootstock/jobs/272428366 The full commit message that triggered this build is copied below: build: configure pandoc using --defaults files merges #290 The options for the pandoc commands in build/build.sh are now configured in YAML files that are passed to Pandoc using --defaults. export PANDOC_DEFAULTS_DIR to override the directory with pandoc --defaults files

merges manubot/rootstock#290 The options for the pandoc commands in build/build.sh are now configured in YAML files that are passed to Pandoc using --defaults. export PANDOC_DEFAULTS_DIR to override the directory with pandoc --defaults files

Move options to pandoc-defaults.yaml

6486c3b

dhimmel mentioned this pull request Dec 19, 2019

. env file for options #287

Open

dhimmel mentioned this pull request Dec 19, 2019

Support --defaults=URL to load default options via HTTP(s) locations jgm/pandoc#5999

Closed

Move defaults to directory

7d9224b

Multiple default files: common & 1 per format

c1636e4

dhimmel force-pushed the pandoc-defaults branch from e573d9d to c1636e4 Compare December 19, 2019 18:14

manubot deleted a comment from AppVeyorBot Dec 19, 2019

Update build/README.md

b068b9e

Add comments to --defaults files

18b78fd

dhimmel mentioned this pull request Dec 20, 2019

[stub] support alternate HTML theme through environment variable #261

Closed

agitter reviewed Jan 3, 2020

View reviewed changes

Support setting PANDOC_DEFAULTS_DIR

220998d

dhimmel merged commit 06fb01d into manubot:master Jan 4, 2020

dhimmel deleted the pandoc-defaults branch January 4, 2020 22:38

agitter mentioned this pull request Apr 27, 2020

Themes #334

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move options to pandoc-defaults.yaml #290

Move options to pandoc-defaults.yaml #290

dhimmel commented Dec 19, 2019

AppVeyorBot commented Dec 19, 2019

dhimmel commented Dec 19, 2019

agitter commented Dec 19, 2019

dhimmel commented Dec 19, 2019 •

edited

Loading

dhimmel commented Dec 19, 2019 •

edited

Loading

Nebucatnetzer commented Dec 19, 2019 •

edited by dhimmel

Loading

AppVeyorBot commented Dec 19, 2019

AppVeyorBot commented Dec 19, 2019 •

edited by dhimmel

Loading

AppVeyorBot commented Dec 19, 2019

dhimmel commented Dec 20, 2019 •

edited

Loading

AppVeyorBot commented Dec 20, 2019

Nebucatnetzer commented Dec 21, 2019

dhimmel commented Jan 2, 2020

Nebucatnetzer commented Jan 3, 2020

agitter left a comment

agitter Jan 3, 2020

dhimmel Jan 3, 2020 •

edited

Loading

agitter Jan 3, 2020

AppVeyorBot commented Jan 4, 2020

Move options to pandoc-defaults.yaml #290

Move options to pandoc-defaults.yaml #290

Conversation

dhimmel commented Dec 19, 2019

AppVeyorBot commented Dec 19, 2019

dhimmel commented Dec 19, 2019

agitter commented Dec 19, 2019

dhimmel commented Dec 19, 2019 • edited Loading

dhimmel commented Dec 19, 2019 • edited Loading

Nebucatnetzer commented Dec 19, 2019 • edited by dhimmel Loading

AppVeyorBot commented Dec 19, 2019

AppVeyorBot commented Dec 19, 2019 • edited by dhimmel Loading

AppVeyorBot commented Dec 19, 2019

dhimmel commented Dec 20, 2019 • edited Loading

AppVeyorBot commented Dec 20, 2019

Nebucatnetzer commented Dec 21, 2019

dhimmel commented Jan 2, 2020

Nebucatnetzer commented Jan 3, 2020

agitter left a comment

Choose a reason for hiding this comment

agitter Jan 3, 2020

Choose a reason for hiding this comment

dhimmel Jan 3, 2020 • edited Loading

Choose a reason for hiding this comment

agitter Jan 3, 2020

Choose a reason for hiding this comment

AppVeyorBot commented Jan 4, 2020

dhimmel commented Dec 19, 2019 •

edited

Loading

dhimmel commented Dec 19, 2019 •

edited

Loading

Nebucatnetzer commented Dec 19, 2019 •

edited by dhimmel

Loading

AppVeyorBot commented Dec 19, 2019 •

edited by dhimmel

Loading

dhimmel commented Dec 20, 2019 •

edited

Loading

dhimmel Jan 3, 2020 •

edited

Loading