ci: cleanup lerna modifications, writing docs

[ElysaSrc]

Motivation

Lerna should only be used on the CI to manage the releasing, and should never be used locally. The version locally should remains stuck at an identifiable development version.

We do not want Lerna to be used locally to release, since we want to have reproducible releases and prevent releasing a version from the workstation of a developer (and for all the security issues it brings with it).

The way of deploying proposed in this PR is opinionated, you can find the rationale in the README of this pull request.

Note: it's the same pattern for the main osrd repository. We do not want to build docker images locally and push them on GHCR: the build must be done in the CI in order to be auditable and reproducible easily.

The goal here is to standardize and document the way we release on this repository.

What is not covered here

This PR does not solve the "local dependency" issue of the project: when working on a project that depends on another one we still have an issue.

Those changes (like the proposal to integrate preconstruct) should be integrated in another pull request.

If adding preconstruct requires to add an extra step while building packages, we should modify .github/scripts/publish.sh accordingly.

Changes

• Removed local Lerna modifications to keep them solely on the CI
• Writing a README documenting the whole process of deployment
• Upgrading flake dependencies for those using Nix

[clarani]

LGTM ! Missing new lines at the end of package.json files, is it on purpose ?

[jacomyal]

Nice work thanks 👏
One question though: The local versions will always remain 0.0.1-dev in the local package.json files? I think it's quite standard nowadays in JavaScript / TypeScript projects to carry the last published version numbers in the package.json.

[jacomyal]

I have a concern, also, about what we discussed yesterday together with @Math-R: How will you handle the dependency versions? Like this one, for instance? Does you script increments it as well? If it doesn't, then it might be a problem when we install @osrd-project/ui-manchette in OSRD. This is something that was pretty well handled with Lerna, also.

[ElysaSrc]

Nice work thanks 👏 One question though: The local versions will always remain 0.0.1-dev in the local package.json files? I think it's quite standard nowadays in JavaScript / TypeScript projects to carry the last published version numbers in the package.json.

Since the version number is always managed externally of the code (git tags), it's not really required to have them inside the code. Furthermore, from experience (and from what we've noticed on this repository) it seems that its generally forgotten by developers and reviewers. Not having the mental burden of thinking of version number outside of release management seems a sane default.

[ElysaSrc]

I have a concern, also, about what we discussed yesterday together with @Math-R: How will you handle the dependency versions? Like this one, for instance? Does you script increments it as well? If it doesn't, then it might be a problem when we install @osrd-project/ui-manchette in OSRD. This is something that was pretty well handled with Lerna, also.

You can see it's handled here: /~https://github.com/OpenRailAssociation/osrd-ui/pull/169/files#diff-2b02f63afb7c58df806c69a5361a5b7eeb72463a351cdee777202fc7ca184f1dR41.

I think using Lerna to modify a json to change some values inside a basic JSON object here and there is quite over-engineered. Furthermore, getting rid of Lerna gives us the flexibility to change the flow as we want in the future. (And remove yet another tool from our dependencies).

[jacomyal]

I have a concern, also, about what we discussed yesterday together with @Math-R: How will you handle the dependency versions? Like this one, for instance? Does you script increments it as well? If it doesn't, then it might be a problem when we install @osrd-project/ui-manchette in OSRD. This is something that was pretty well handled with Lerna, also.

You can see it's handled here: /~https://github.com/OpenRailAssociation/osrd-ui/pull/169/files#diff-2b02f63afb7c58df806c69a5361a5b7eeb72463a351cdee777202fc7ca184f1dR41.

OK, good then. I do think it is very uncommon to handle things that way in JavaScript / TypeScript project, but it seems to work, and has much better documentation than before. LGTM

[jacomyal]

Well, I was discussing with @sim51, and your script seens not that easy to maintain (it breaks if we start using peerDependencies, rename or move the NPM namespace, start publishing packages outside the namespace...).

Why didn't you just add a lerna version (that handles all those updates, is battle tested etc...) as I showed yesterday, but in the CI script, and without pushing? Is there any good reason to write a custom bash script instead?

[ElysaSrc]

Well, I was discussing with @sim51, and your script seens not that easy to maintain (it breaks if we start using peerDependencies, rename or move the NPM namespace, start publishing packages outside the namespace...).

I do not agree with the statement of maintainability. We should focus primarily on the workflow we want to have for releasing, then see if a tool can match it or if we want to have a script.

A shell script of a few line like this one is not complicated to maintain.

We do handle way more complicated scripts for the osrd repository deployments. If the issue is peerDependencies, it can be handled quite easily by adding it inside the jq command. Same goes with the other concerns : even if we add a new namespace, then we can adapt the script to match this behavior.

Why didn't you just add a lerna version (that handles all those updates, is battle tested etc...) as I showed yesterday, but in the CI script, and without pushing? Is there any good reason to write a custom bash script instead?

Because Lerna is basically made to create commit, releases, tags and stuff that occurs before the release on npmjs in our workflow.

Using lerna to just have "lerna version" is litteraly the workflow that was present before, if you check the previous command in the diff.

The flow you showed us yesterday was basically you creating a local tag updating package.json file in the repository using lerna.

With this workflow you must use Lerna locally and remotely, meaning the burden of handling the revisions falls on the developer and can easily be forgotten.

All in all, I'd rather have a bit of complexity hidden in the .github folder than asking developers more actions to remember when they are working on the feature. If developers of osrd-ui mainly disagree with this statement, then I guess we can switch back to the old process.

[jacomyal]

OK, all good for me 👍

[sim51]

Just my 2 cents :

Lerna is made for monorepo to simplify their management, and is a standard in the JS industry (turf is using it for example). If you prefer to make a custom shell script that reproduces what we use today in lerna, deal!

But IMHO it's a technical debt: a tool already exists, does what we want (and well, with more possibilities for the futur) , it is maintained and follows the evolution of npm (monorepo in JS has many improvements each year).

A shell script of a few line like this one is not complicated to maintain.

The maintenance is not only a question of code, but also of knowledge with the evolution of the tech.

[Khoyo]

Otherwise LGTM

[ElysaSrc]

Just my 2 cents :

Lerna is made for monorepo to simplify their management, and is a standard in the JS industry (turf is using it for example). If you prefer to make a custom shell script that reproduces what we use today in lerna, deal!

I do not see how it is what is used today since the release flow included action on the developer side.

From what I've saw yesterday, that was definitely not the flow we had for releasing, that @jacomyal shown. If the flow shown by @jacomyal was not the intended flow, then I'm opened to suggestions.

If you feel there is a way to leverage lerna directly and solely within the CI so that we can release from within the CI using only the flow where:

1. The developer tags a version on the repo, and push it (meaning using git tag, git push --tags).
2. The developer goes to GitHub then create a release using that version number.
3. All packages are built and released on npmjs

I'm not against Lerna per se, I'm basically against "developer has to do something locally that can be easily (and will be) forgotten".

Feel free to open a PR that matches the mentioned flow. I'm not here in order to make things harder with people working on the project, that's the exact opposite.

But IMHO it's a technical debt: a tool already exists, does what we want (and well, with more possibilities for the futur) , it is maintained and follows the evolution of npm (monorepo in JS has many improvements each year).

A shell script of a few line like this one is not complicated to maintain.

The maintenance is not only a question of code, but also of knowledge with the evolution of the tech.

Since we go from LTS to LTS releases and rely solely on npm, we're quite safe about this point IMHO.

What works on nodejs 20 and npm will still be working as long as we stay on v20. When we upgrade, then we can also upgrade our pattern if some better way are available.

[jacomyal]

From what I've saw yesterday, that was definitely not the flow we had for releasing, that @jacomyal shown. If the flow shown by @jacomyal was not the intended flow, then I'm opened to suggestions.
If you feel there is a way to leverage lerna directly and solely within the CI so that we can release from within the CI using only the flow where:

Well I do, that's what I was trying to describe earlier. So, yesterday, I ran lerna version, that did multiple things:

• It properly edited the version number in all the package.json and package-lock.json files
• Committed the updates and tagged the commit (we don't care here if we are in the CI)

Then, at this point, you have everything that needs to be handled in the package.json and package-lock.json files.

You want it to skip the commit and tag part that you don't care about from the CI? Add the --no-git-tag-version option. Also, you can skip the questions part with --yes I think, and give a version directly to Lerna.

So, it would look like lerna version --no-git-tag-version --yes X.Y.Z I think. From the CI.

[ElysaSrc]

Force-pushed.

Removed the custom script in favor of Lerna. Note that I haven't been able to test the result locally.

We should not have code files/json or whatever text file in the repository that refers to the commit hash or to the version since it is managed through git tags and github releases. Those two elements should remain the only source of truth when it comes to this.