EmbedAPI: support "pretty URLs"

[humitos]

Tru for the URL ending with index.html if the original URL doesn't exist in the storage.

This makes Sphinx HTMLDir build, Docusaurus and other documentation tools with pretty URLs to work properly.

[humitos]

Failing test readthedocs/rtd_tests/tests/test_domains.py::FormTests::test_valid_domains is unrelated to this PR.

[stsewd]

You should use build_media_storage.join, otherwise this is likely vulnerable to path traversal. Also, I think we already have some logic for handling index files somewhere, we just check if the file ends with / and append index.html, otherwise just treat it as a regular file.

[humitos]

I found this code:

readthedocs.org/readthedocs/proxito/views/mixins.py

Lines 66 to 76 in 7195740

	# Handle our backend storage not supporting directory indexes,
	# so we need to append index.html when appropriate.
	if not filename or filename.endswith("/"):
	filename += "index.html"
	# If the filename starts with `/`, the join will fail,
	# so we strip it before joining it.
	try:
	storage_path = build_media_storage.join(
	base_storage_path, filename.lstrip("/")
	)

[humitos]

I updated this PR following that code 👍🏼

[humitos]

we just check if the file ends with / and append index.html

We can't do that in this case. The filename I'm receiving when using Docusaurus doesn't have an ending /:

(Pdb++) filename
'/docs/tutorial-basics/create-a-page'

I'm not finding where in the code we manage this case from El Proxito. These type of URLs are definitely being served correctly.

[humitos]

I found why it works on El Proxito, we are performing a redirect when the URL doesn't ends with /

▶ curl -sIL http://test-builds.devthedocs.org/en/docusaurus/docs/tutorial-basics/create-a-page | head
HTTP/1.1 302 Found
Server: nginx/1.22.0
Date: Tue, 03 Dec 2024 07:41:12 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Location: /en/docusaurus/docs/tutorial-basics/create-a-page/
Access-Control-Allow-Origin: *
Content-Language: en
Vary: Accept-Language, Cookie
...

This redirect is done at

readthedocs.org/readthedocs/proxito/views/serve.py

Line 653 in bef0dbe

log.info("Redirecting to index file.", tryfile=tryfile)

[stsewd]

yeah, we always redirect to the correct path instead of trying to serve the file from two different URLs

[stsewd]

We can't do that in this case. The filename I'm receiving when using Docusaurus doesn't have an ending /:

Do you mean that Docusaurus creates links without /?

[humitos]

Do you mean that Docusaurus creates links without /?

Yes.

There are a few documentation tools that don't use ending /, in particular those that are SPA for some reason.

If you go to https://test-builds.readthedocs.io/en/docusaurus/docs/intro we will redirect you to https://test-builds.readthedocs.io/en/docusaurus/docs/intro/. However, if you click any link, the resulting URL in your browser won't have that ending / -- that's the URL we are sending to the backend, and since we don't find the file in the storage we are returning 404.

Checking for the index.html solves this issue.