Tracking issue for hoedown -> pulldown regressions

[steveklabnik]

In #40338, we landed pulldown-cmark. 🎊

But, given that it's a different renderer, there are bound to be differences. This comment pointed out some obvious problems. These bugs are going to be much easier to clean up than the initial PR was to land, though 👍

To help with this work, I generated docs for both the commit before and the commit of, ran all the HTML files through tidy-html5, and put it up here: steveklabnik/docdiff@ddda1fe the ' -> " changes are expected, but that tool doesn't have the ability to re-write those, so I left them in for now, which, frankly, makes the diff kinda large. working on it.

Here's the stuff we've found so far:

• horizontal rules
• footnotes (waiting for pulldown update End footnote definition with one blank line. pulldown-cmark/pulldown-cmark#21)
• images
• <br /> for HardBreaks
• the title part of links being rendered as visible text rather than as an attribute on the <a> tag (imperio update: I'll need an example to be able to fix this one)
• MarkdownHtml now doing the same as Markdown when it should be escaping raw HTML.
• some strange rendering around methods at times? rendered: https://doc.rust-lang.org/nightly/std/collections/struct.LinkedList.html#method.front_place source /~https://github.com/rust-lang/rust/blob/master/src/libcollections/linked_list.rs#L700 fix: Fix links #41066
• In src/libstd/collections/hash/map.rs under "Relevant papers/articles:", it appears the ordered list has been rendered as an unordered list instead. PR r+'d
• ^ now needs to be an html <sup>. (this is not a regression in the code, but something that needs to be adapted in the docs) Replace ^ with <sup> html balise #41043
• Panic caused by Markdown in latest nightly #40918 (this one was due to markdown spec switch)
• Footnotes should have a link back to their referent, see Add support for image, rules and footnotes #40919 (comment)
• words getting smooshed together (I think the hard/soft break fixed this, I'm not observing it now)
• Nonfatal rustdoc error on cssparser-macros #40978 (this one was due to markdown spec switch)

This change will land in tonight (3/29)'s nightly, so we can also poke at them then. I plan on making a users post tomorrow to advertise this bug.

Tagging as a regression so we make sure to take care of it. Marking as P-high and assigning @GuillaumeGomez and @frewsxcv who are both already working at knocking some of this out.

[ScottAbbey]

In core/intrinsics/fn.overflowing_sub.html.txt (and elsewhere?), hoedown was rendering 2^N as 2<sup>N</sup>, the new version just leaves it as 2^N.

This isn't recognized in the CommonMark spec, probably need to change the source to 2<sup>N</sup> ?

[steveklabnik]

@ScottAbbey agree 100%; we'll have to send in a PR to fix. I wonder what the easiest way of finding all of them is.....

[ScottAbbey]

It looks like markdown.rs is squishing things together when a list item spans multiple lines. In pulldown-cmark's html.rs, this would have been a SoftBreak that got turned into a \n, it appears this rendering is just throwing those away. Maybe not all of them? But some?

[steveklabnik]

For the ^ issue; rg -g !*.cpp -g !*.h \^ | rg "///" is giving me pretty reasonable results. Still lots of false positives, but manageable.

[ScottAbbey]

In src/libstd/collections/hash/map.rs under "Relevant papers/articles:", it appears the ordered list has been rendered as an unordered list instead.

[SergioBenitez]

I'm getting an unexpected panic from rustc that appears to be caused by the transition:

---- src/request/form/from_form_value.rs - request::form::from_form_value::FromFormValue (line 59) stdout ----
	error: unknown start of token: `
 --> <anon>:2:42
  |
2 | A value is validated successfully if the `from_str` method for the given
  |                                          ^

thread 'rustc' panicked at 'Box<Any>', /checkout/src/libsyntax/parse/lexer/mod.rs:81
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread 'rustc' panicked at 'couldn't compile the test', /checkout/src/librustdoc/test.rs:270

The markdown being referenced in the error looks like this:

/// Rocket implements `FromFormValue` for many standard library types. Their
/// behavior is documented here.
///
///   * **f32, f64, isize, i8, i16, i32, i64, usize, u8, u16, u32, u64**
///
///   **IpAddr, Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6, SocketAddr**
///
///     A value is validated successfully if the `from_str` method for the given
///     type returns successfully. Otherwise, the raw form value is returned as
///     the `Err` value.

Edit: I'll open a separate issue for this.