Cypress Component Test Performance Issue with Angular

[edbzn]

Description

We have recently noticed that running Cypress component tests with Angular in our CI pipeline is taking a significant amount of time. This is due to the overhead from Cypress itself and the fact that a large shared library was recently split into smaller libraries, resulting in multiple projects running component tests. We are not currently using nx-cloud and are unable to parallelize component tests due to the port already in use issue.

Motivation

In the current state, it's almost impossible to scale component tests as they drastically slow down the CI pipeline, especially in a workspace with many libraries.

Suggested Implementation

@Coly010 suggested batching all tests from the affected projects into one Cypress run, which would reduce the overhead of running Cypress multiple times and only test what needs to be tested. A similar approach is already in place for Jest and eslint.

As a workaround, we are forced to disable component tests in the CI pipeline and run them manually.

Alternate Implementations

• Run tests in parallel
• Replace webpack with vite (experimental and may introduce issues with Angular features not yet supported by the vite builder)

[Coly010]

Tagging @barbados-clemens as I think he has an idea around this potentially.

[barbados-clemens]

Basically it's a tricky spot. across multiple libs the cost of spinning up cypress starts to cost more but ideally that allows you to have higher cache rate hits.

You could look into using using the cypress parallel option if you can't turn DTE on today. https://docs.cypress.io/guides/guides/parallelization#CI-parallelization-interactions

One of the issues with trying to run in parallel with Nx is the running multiple Cypress instances on a CI machine is a pretty heavy task so doing more than 1 on a single agent might cause things to get even slower or become flaky or cause CI to kill the agent for high resource utilizations.

Even if you could run on multiple component testing tasks in CI (say you got a really big spec'd out machine) there is an issue of port conflict, So you'll need to set all ports to be different and I'm not too sure if that's controllable with component testing (I'll ask).

Which leads us into a strange corner were it's almost recommended to make a larger app in certain cases were it's faster to test more than we want than it is to spin up cypress over and over again (which includes starting the dev server)

First I'd make sure CI caching is in place and working. caching the cypress binary and the node_modules so we aren't waiting for cypress to download/unzip/launch the first time.

Next thing to question is what part of task is taking how much amount of time.
i.e. if the CT task takes 5 minutes and 2 minutes of that is waiting for the app to boot then maybe there is a way to slim down the app that is being used i.e. the devServerTarget executor option that's being used.

In which case maybe try using a freshly generated app with no components in it and only has the extras you know are needed for CT testing like styles/assets/scripts. spinning up an app with no components is faster than with components even when those components aren't being used.

The "pulling all the stops out" approach would be to make 1 large CT app and section it all off by folders and then figure out what files are affected across your projects and then map that back to the single CT project folder layout and dynamically pass in a glob to grab the respective files. (this assumes they all use the same app for devServerTarget).
this way you only pay the cypress and devserver start up time once.

But you should not have to do that (unless you really want to). I'm working with the Cypress CT team to see how we can better integrate and will keep you posted.

[flobacher]

Hey!
May I ask if there is any updates regarding this issue?
Thank you very much in advance!

[barbados-clemens]

@flobacher it's on my list to look at, we just released the feature based testing with Cypress which made it possible to run cypress e2e tests in parallel. I have plans to help use that work to help parallelize the CT test running as well.

[flobacher]

@barbados-clemens Great news! Thank you very much!

[flobacher]

@barbados-clemens asking politely if you have any news on this topic? Thank you very much!

[barbados-clemens]

Non so far. haven't had a chance to look into this as other feature work as taken priority.

[flobacher]

@barbados-clemens thx for the update!

[joewIST]

This is also an issue for us. Is there a way for the dev-server specified in devServerOutput to be shared between libraries?

[barbados-clemens]

Hi all, just a quick update.
So I experimented with running in parallel, and basically vite/webpack config based projects actually can run in parallel just by changing the different port the vite/webpack configuration will use for a dev server.

The exception to this is angular, since cypress uses a build configuration to run the project, that doesn't have a port value. Even manually passing a port to the build options it's still not utilized. So unable to run angular CT projects in parallel currently.

cc @jordanpowell88, maybe this is something we can look into for enabling running CT in parallel for angular projects.

[barbados-clemens]

@joewIST I'm not 100% sure what options you're talking about could you give an example?

[jordanpowell88]

cc @jordanpowell88, maybe this is something we can look into for enabling running CT in parallel for angular projects.

Lets do it!

[mirobo]

Hi all, just a quick update. So I experimented with running in parallel, and basically vite/webpack config based projects actually can run in parallel just by changing the different port the vite/webpack configuration will use for a dev server.

The exception to this is angular, since cypress uses a build configuration to run the project, that doesn't have a port value. Even manually passing a port to the build options it's still not utilized. So unable to run angular CT projects in parallel currently.

cc @jordanpowell88, maybe this is something we can look into for enabling running CT in parallel for angular projects.

With Cypress 13 the port configuration is now apparently usable :-)

Dev Servers will now respect and use the port configuration option if present. Fixes #27675.

[joewIST]

@joewIST I'm not 100% sure what options you're talking about could you give an example?

I believe this may no longer be an issue given the rest of the conversation. Can I assume that with Cypress v.13 we can run parallel tests in Angular/Nx libraries by setting a different port value in each library's cypress.config.ts?

[jordanpowell88]

Cypress v 13 does in fact fix the port issue preventing CT from running in parallel. We did discover a small change blocking it from working in Angular. I will have a PR landing soon

[mirobo]

But does Cypress still need to start/setup the dev server for each spec? I miss a NX/Angular devServer startup script that starts the devServer once for a run and every component test can reuse that very devServer (most of the time they rely on the same ng application). Now we have a overhead of about 45 seconds to before the component test even starts which makes them slower than e2e tests and frankly quite unpractical in development. Also the ng application is watching component test files within any nx library and does a rebuild (and says that there were 0 changes). When editing component tests this is also affecting "developer experience performance". Same happens if you have e2e tests within a NX library to take advantage of NX affected (once you change an e2e test, nx/ng rebuilds the library for nothing)..

[joewIST]

@barbados-clemens ^ This is kind of what I meant before

[barbados-clemens]

@mirobo @joewIST Yes, right now the dev server will still need to spin up for each new instance of Cypress. Re-using a running dev server will require quick a bit of re-architecting on both the Nx and Cypress side (depending on how it's tackled).

Ideally Nx would be able to provide cypress a custom dev server implementation that just runs a cached build based on static files, so while the dev server will have to restart multiple times, the start time would take hardly any time. But this most likely will introduce other behaviors that have to be handled for Cypress to be able to still instrument the running app.

Other idea would be for Cypress to pass the dev server around to other instances of Cypress which might not really be possible from that side. These are things we still have to explore and understand. the parallel running on different ports while not a perfect solution was an easy win and enables something for people to better improve their CT times in CI at least.

[jordanpowell88]

@mirobo , @joewIST I believe that this is the plan for Nx to support this behavior but I don't want to speak on an official capacity around this.