Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smoke tests - automatically test against stable build #127799

Merged
merged 9 commits into from
Jul 12, 2021
Merged

Conversation

bpasero
Copy link
Member

@bpasero bpasero commented Jul 1, 2021

This PR attempts to include the smoke tests that are usually only running when you provide --stable-build in our CI to fully automate smoke tests and not having to run them in endgame.

By using our existing vscode-test module it is relatively easy to download latest stable bits.

However, when I run the following command for the first time in a fresh checkout:

yarn smoketest --build /Users/bpasero/Desktop/Visual\ Studio\ Code\ -\ Insiders.app

I am greeted with this error:

  1) VSCode Smoke Tests (Electron)
       Stable vs Insiders Smoke Tests: This test MUST run before releasing
         Datamigration
           verifies opened editors are restored:
     Error: connect ENOENT /var/folders/fv/p__j7gw9097crr40g7sf2pxw0000gn/T/tmp-8451c917E3g5M20a
      at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)

  2) VSCode Smoke Tests (Electron)
       Stable vs Insiders Smoke Tests: This test MUST run before releasing
         Datamigration
           verifies that 'hot exit' works for dirty files:
     Error: connect ENOENT /var/folders/fv/p__j7gw9097crr40g7sf2pxw0000gn/T/tmp-8451dPWOrpoUbcmm
      at PipeConnectWrap.afterConnect [as oncomplete] (node:net:1133:16)

When I run again, it works fine. vscode-test is storing the downloaded bits into a .vscode-test folder in test/smoke and will not download it again if it is found there.

I tried all things like adding a timeout to ensure the build is there but couldn't figure it out. Need your help @joaomoreno if you maybe got further ideas what this could be.

@bpasero bpasero added this to the July 2021 milestone Jul 1, 2021
@bpasero bpasero requested a review from joaomoreno July 1, 2021 16:35
@bpasero bpasero self-assigned this Jul 1, 2021
@bpasero bpasero added the engineering VS Code - Build / issue tracking / etc. label Jul 1, 2021
@bpasero
Copy link
Member Author

bpasero commented Jul 2, 2021

@joaomoreno joaomoreno self-assigned this Jul 6, 2021
@joaomoreno joaomoreno merged commit d85bc64 into main Jul 12, 2021
@joaomoreno joaomoreno deleted the ben/smoke-stable branch July 12, 2021 12:44
@github-actions github-actions bot locked and limited conversation to collaborators Aug 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
engineering VS Code - Build / issue tracking / etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants