twine: command not found

[albertz]

It has all worked for a long time, but now I get:

/app/twine-upload.sh: line 35: twine: command not found

I assume this is because I use ubuntu-latest which has now been upgraded to Ubuntu 22?

My GitHub actions file:

name: Publish

on:
  workflow_run:
    workflows: ["CI"]
    branches: [master]
    types:
      - completed

jobs:
  publish:
    if: >-
      github.event.workflow_run.conclusion == 'success' &&
      github.event.workflow_run.head_branch == 'master' &&
      github.event.workflow_run.event == 'push' &&
      github.repository == 'rwth-i6/returnn'
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Checkout submodules
      run: |
        git submodule sync
        git submodule update --init --recursive

    - uses: actions/setup-python@v2
      with:
        python-version: 3.7

    - name: Install Python deps
      run: |
        echo "PATH=$PATH:~/.local/bin" >> $GITHUB_ENV
        pip3 install --user --upgrade pip setuptools wheel
        pip3 install --user typing  # needed for Python 2 in some cases
        pip3 install --user -r requirements.txt

    - run: python3 setup.py sdist

    # /~https://github.com/marketplace/actions/pypi-publish
    - name: Publish to PyPI
      uses: pypa/gh-action-pypi-publish@release/v1
      with:
        user: __token__
        password: ${{ secrets.pypi_password }}

[albertz]

I assume this is because I use ubuntu-latest which has now been upgraded to Ubuntu 22?

Ok, this does not seem to be the problem. I get the same problem with ubuntu-20.04.

[webknjaz]

The action runtime is a docker container. Your runner VM setting does not influence what's inside.
Is your workflow run a few days old? We saw bugs that are fixed now.

[webknjaz]

Side note: calling setup.py directly is deprecated. Use build instead.

[webknjaz]

Do you have a workflow run link so I could take a look at the logs?

[albertz]

Is your workflow run a few days old? We saw bugs that are fixed now.

No, just a few minutes ago.

[albertz]

Do you have a workflow run link so I could take a look at the logs?

/~https://github.com/rwth-i6/returnn/actions/runs/3629006064/jobs/6120696860
/~https://github.com/rwth-i6/returnn/actions/runs/3628833422/jobs/6120316700
/~https://github.com/rwth-i6/returnn/actions/runs/3628666104/jobs/6119948952

[webknjaz]

I wonder if the parent image has changed somehow... Although, it is weird since the PATH var is being set in ours regardless.

[webknjaz]

There shouldn't be any significant difference but could you try unstable/v1?

[albertz]

There shouldn't be any significant difference but could you try unstable/v1?

I also tried v1.5 but got the same error.
/~https://github.com/rwth-i6/returnn/actions/runs/3629446217/jobs/6121659923

[albertz]

There shouldn't be any significant difference but could you try unstable/v1?

Same error. /~https://github.com/rwth-i6/returnn/actions/runs/3629576346/jobs/6121942005

[albertz]

Is there anything else I can try? Some workaround?

Do you have an idea what's wrong?

[albertz]

I now tried release/v1.4 and that seems to work fine.

uses: pypa/gh-action-pypi-publish@release/v1.4

[webknjaz]

Do you have an idea what's wrong?

Kinda. It seems like the $PATH env var is not set correctly but /~https://github.com/pypa/gh-action-pypi-publish/blob/unstable/v1/Dockerfile#L12 is supposed to handle that which is confusing. Plus I think I saw the current version working just fine which makes it even weirder.

[webknjaz]

Oh wait. I see what's happening. GitHub executes the container with -e "PATH" which effectively replaces the default value that we set.
And you modify the $PATH value in your workflow. The container is Debian-based so there may be differences with Ubuntu. But the main problem is replacing the variable so that it no longer includes the necessary path.

P.S. You are setting your var wrong anyway — ~ does not work there, you must expand it or interpolate "$HOME".

[albertz]

Thanks!

[albertz]

Hm now I tried:

      uses: pypa/gh-action-pypi-publish@fedca4a9a83c83baf9a426a681a2aab0d5c97e30

(fedca4a)
But I still get the same error: /~https://github.com/rwth-i6/returnn/actions/runs/3633756573/jobs/6131129233

[webknjaz]

Thanks for reporting the new log. I've realized that the problem is even deeper. Moreover, it'd affect any container-based action and is not technically our bug.
I'll try making the situation better, but you should really avoid setting the $PATH env var. When you set the env var in a job, GHA injects it into every step, including passing it inside container-based actions — it has a lot of potential to break stuff.
One thing that can be done to limit the damage is to separate the build job from the publish one. The former would save an artifact and the latter would download it and publish.

[albertz]

I'm not too familiar with GHA but aren't there other actions which also modify PATHor PYTHONPATH like setup-python?

If you don't want to inherit PATHin a container, maybe it makes sense for a feature request for GitHub or whoever is responsible for that?

[webknjaz]

I'm not too familiar with GHA but aren't there other actions which also modify PATHor PYTHONPATH like setup-python?

I'm not entirely sure that it uses the same mechanism for setting up the env.

If you don't want to inherit PATHin a container, maybe it makes sense for a feature request for GitHub or whoever is responsible for that?

It's not specific to this var, any var that you set will be passed as if you've set the env: value explicitly.

[webknjaz]

v1.6.3 should work even with your broken path.

[albertz]

I thought that release/v1 should also work now but it seems it does not? /~https://github.com/rwth-i6/returnn/actions/runs/3634342125/jobs/6132328966

[webknjaz]

@albertz it does now: #115