Chore/SK-1468 | Add new CI tests with integration with studio #833
+246
−17
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Introduces new CI tests with integration with Studio using the new features in fedn cli.
Obs that the test scripts can be used by anyone with an studio account. Good for quick testing of examples. However, the user can NOT have any other projects in studio when running this test. A fix will hopefully come soon.
Obs that all clients will be killed after any EXIT. The clients are running as background processes with id $PID, client outputs are logged into a file test_.log in FEDN_EXAMPLE folder. Suffix is a number [0,FEDN_NR_CLIENTS -1]
Configurations for the test is taken from .ci/tests/studio/.env and can be changed:
"FEDN_NR_ROUNDS": 5
"FEDN_ROUND_TIMEOUT": 180
"FEDN_BUFFER_SIZE": -1
"FEDN_NR_CLIENTS": 2 # this can be scaled but will run as subprocesses from test client (your machine if running there)
"FEDN_CLIENT_TIMEOUT": 60 # Max time to wait for clients "online" status before giving up
"FEDN_FL_ALG": "fedavg" # agg settings will hopefully come in the future
"FEDN_NR_EXPECTED_AGG": 2 # Number of expected aggregated models per combiner, you might for example expect nr_agg < nr_clients if you want to simulate stragglers
"FEDN_SESSION_TIMEOUT":300 # Session timeout in seconds, all rounds must be finished within this time
"FEDN_SESSION_NAME": test
"FEDN_EXAMPLE": mnist-pytorch # the test will do a pushd/popd into /examples/$FEDN_EXAMPLE
"STUDIO_HOST": api.studio.scaleoutsystems.com # host adress to API
"STUDIO_USER": github@scaleoutsystems.com
The only ENV which is not in the .env is "STUDIO_PASSWORD" and needs to be exported by user/workflow:
export STUDIO_PASSWORD=
To run the test it's recommended that there are "precreated" projects before the test. From the root folder of fedn source:
.ci/tests/studio/studio.sh