Allow translating from javascript files

[cofiem]

This PR enables translating strings in javascript.

It demonstrates the js translation in use by applying it to the timeago.js js util.

Edit: added overview of the two mutually-exclusive approaches.

Approach: webpack plugin the generates js message bundles

This is the current state of this PR.

I attempted an approach similar to https://www.npmjs.com/package/@zainulbr/i18n-webpack-plugin
i.e. embed the translations into separate js bundles, one for each language.

Unfortunately, the existing webpack plugins were not suitable (issues with webpack 5 and/or did not work for the warehouse setup).
However, I did manage to create a webpack plugin that does the job (in webpack.plugin.localize.js).
The plugin checks the plural forms and embeds them into the generated js bundles.

So, the development process with these changes:

• Use gettext or ngettext as appropriate in js files
• Run make translations: pybabel extracts messages from js files, then a python script generates messages.json files for only the KNOWN_LOCALES and only includes translations that are needed in js
• Run make static_pipeline: the webpack plugin embeds the translated data from the messages.json files
• At runtime, the matching js bundle for the current locale is set in base.html, and gettext.js is initialised with the translation data. It can determine plurals and do string templating.

These are the benefits of this approach:

• Uses the existing translation and js tooling - make, pybabel, locale files, webpack
• Enables plurals and template placeholders
• Limits or does not use the hard to cache routes nor additional http requests
• Limits the number of unmaintained packages required, and works with webpack 5
• Straightforward to maintain and use as part of contributions

Approach: request from client to server to get translation for a given message code

A js util fetch-gettext.js that requests the translation code returns a promise, which contains the translation text from the server, using the locale set on the server-side.

view the changes for this approach.

Changes:

• enables extracting strings from javascript using pybabel,
• adds a route that renders the appropriate translation using the existing machinery, and
• adds js util for obtaining the translated string.

Considerations:

• may make many http requests for translations

Notes

• Pybabel seems to require particular js function names and the config to change them does not work, similar to this pybabel issue. This is ok, I've used the names expected by pybabel.

Relates to issues:

• Make it easier to see the available wheels per package release #15087
• Localise JS strings #6673
• Content originating from JavaScript not marked for translation #7955
• Missing translations #7455

[di]

Very cool! I like this approach a lot, I think this will definitely work.

One thing I'm wondering: since we probably won't have a ton of JS translations, would it make sense to load them all for a given locale in a bundle via a single request, as opposed to making N different requests?

[cofiem]

One thing I'm wondering: since we probably won't have a ton of JS translations, would it make sense to load them all for a given locale in a bundle via a single request, as opposed to making N different requests?

I'm not sure I understand this.

I found a way to use the server-side rendering of translations because:

• I can't see that there's a way to get just the js translation strings to make a bundle of translation strings
• each translation can have up to 6 variants, which are chosen based on the num parameter
• some translations have placeholders in the string, and I didn't want to have to find or make a way to fill the placeholders in js

If you're concerned about the server load or frequency of requests, the browser cache seems to work well, and there's likely some additional caching to be done in the server-side view that could reduce the need to execute the localiser.pluralize or localizer.translate.

Please let me know if I've misunderstood what you're asking.

[cofiem]

I managed to get the plurals extracted.

It looks like python-babel expects particular function names.

TODO:

• I'll include some screenshots of the js translations in action.
• I may also have time to go through and make the rest of the text that's been identified in the other issues as translatable.
• fix the couple of failing tests

Question:

• Is there an existing way to cache the output of a view that depends on the request.params? That would work well for caching the translation view on the server side.
  Maybe origin_cache?

[di]

If you're concerned about the server load or frequency of requests, the browser cache seems to work well, and there's likely some additional caching to be done in the server-side view that could reduce the need to execute the localiser.pluralize or localizer.translate.

Yes, I was thinking about balancing the number of requests with the size of the requests. It's probably an over-optimization though, our CDN will cache these responses by default so it's not the load on our backend that I was concerned with, just the number of requests the user's browser would have to make after the page has loaded. I think it's fine to move forward as-is, just wanted to think through it a bit.

Is there an existing way to cache the output of a view that depends on the request.params? That would work well for caching the translation view on the server side.

By default, we cache all requests, but strip all URL parameters when caching a given view (with some exceptions). We can add an exception for this view. This happens here:

/~https://github.com/pypi/infra/blob/9568346c6781a6d9e754e9edc3883dd34febd97b/terraform/warehouse/vcl/main.vcl#L103-L126

[cofiem]

The only remaining question is what @di was asking - is the number of requests made by the browser a problem?

Some example screenshots of the js translations in use.

[cofiem]

Test errors:

> msg += " " + " ".join(results["feedback"]["suggestions"])
E TypeError: unsupported operand type(s) for +=: 'LazyString' and 'str'

This one could be fixed by adding str(msg), but there might be a better way?

- msg += " " + " ".join(results["feedback"]["suggestions"])
+ msg = str(msg) + " " + " ".join(results["feedback"]["suggestions"])

> return request.localizer.translate(_translation_factory(message, **kwargs))
E AttributeError: 'NoneType' object has no attribute 'localizer'

I'm not sure how to fix this error. Other form tests seem to be able to set request=pretend.stub(), but the Validator directly uses from warehouse.i18n import localize as _, which errors at request.localizer.

[di]

LGTM

[miketheman]

I'm +1 to these changes, and will add a few commits that fix minor nits that have crept in since. I plan to merge tomorrow morning.

[di]

Looks like this is causing build failures: /~https://github.com/pypi/warehouse/actions/runs/12811292443/job/35720280486?pr=17373

[di]

I think this should resolve it: #17424