Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sandbox e2e test #206

Merged
merged 35 commits into from
Sep 7, 2023
Merged

Sandbox e2e test #206

merged 35 commits into from
Sep 7, 2023

Conversation

edwardfoyle
Copy link
Contributor

@edwardfoyle edwardfoyle commented Sep 5, 2023

Issue #, if available:
Fixes #46
Description of changes:
Adds a lightweight e2e framework for controlling the CLI. Adds a sandbox e2e test that checks that sandbox deploy of a basic project works and produces expected output.

There are several snapshot updates that were triggered by changing the file structure of the test projects to match the defined file convention of <projectRoot>/amplify/backend.ts.

The bulk of the functional changes are in the integration-tests package. I decided to put e2e tests in this package because they are a form of integration test and it allows us to easily reuse the same test project templates in both integration and e2e tests.

Out of scope:

  1. there's a bug in the data construct preventing sandbox delete from cleaning up all resources. This should be fixed within the week. For now, we can manually clean up resources if/when we hit limits
  2. Once we have multiple e2e tests we'll need a way to identify interleaved cli console output. We'll probably want to amend the process controller to prefix each line with the test name so we can grep out lines pertaining to individual test when multiple are running in parallel.
  3. There's also definitely still work to be done to create some nice abstractions around setup and cleanup of e2e tests but I want to wait until we have a few tests under our belt to see what good abstractions look like there

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@changeset-bot
Copy link

changeset-bot bot commented Sep 5, 2023

🦋 Changeset detected

Latest commit: 9718ab6

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@edwardfoyle edwardfoyle added the run-e2e Label that will include e2e tests in PR checks workflow label Sep 5, 2023
@edwardfoyle edwardfoyle marked this pull request as ready for review September 5, 2023 23:10
sobolk
sobolk previously approved these changes Sep 5, 2023
Copy link
Member

@sobolk sobolk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As for the data bug.

Regardless of that we need some kind of sweeper that runs on schedule and nukes orphaned resources (i.e. older than 2 hours). This can be GH action that runs on schedule or lambda (I like Action more I think).
We can't rely on tests.

package.json Outdated
@@ -12,6 +12,7 @@
"clean:npm-proxy": "npm run stop:npm-proxy && rimraf verdaccio-cache verdaccio-logs.txt",
"diff:check": "tsx scripts/check_pr_size.ts",
"docs": "typedoc",
"e2e": "tsx --test --test-reporter spec --test-name-pattern \"/^\\[E2E\\]/\" packages/integration-tests/lib/e2e",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to use path convention instead of test name convention ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it doesn't look like the node test runner supports glob file patterns which is why I went with this approach. I can do a little more digging though because there is this issue which is closed but it still doesn't look like the feature is documented

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other option is
describe('foo', { skip: /** evaluate some logic to determine test category **/}, ()=> {...
but that would mean being verbose in unit tests...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated to using a combination of glob and grep in the test command to run only the paths we care about without relying on specific test name prefixes

packages/integration-tests/src/e2e/sandbox.test.ts Outdated Show resolved Hide resolved
packages/integration-tests/src/e2e/sandbox.test.ts Outdated Show resolved Hide resolved
Comment on lines +72 to +73
# we have to build and install in this job rather than restore from cache because the
# symlinks in the node_modules folder are broken if we just restore the build from the cache
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know why?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because restoring from cache is a different file. So even though it's restoring a symlink, the link is now broken

packages/integration-tests/src/e2e/sandbox.test.ts Outdated Show resolved Hide resolved
sobolk
sobolk previously approved these changes Sep 7, 2023
@edwardfoyle
Copy link
Contributor Author

@sobolk tracking the e2e resource sweeper here: #218

@edwardfoyle edwardfoyle merged commit e842e30 into main Sep 7, 2023
@edwardfoyle edwardfoyle deleted the e2e-tests branch September 7, 2023 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run-e2e Label that will include e2e tests in PR checks workflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Strategy for E2E tests to get AWS credentials for making service calls
3 participants