-
Notifications
You must be signed in to change notification settings - Fork 23
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(validation)!: validation testing framework (#667)
* feat: inject internal package * feat: updated inject functionality for complex filters * feat: inject testing * feat(validation): add validation testing skeleton * feat(validation-tests): refactor of inject->transform, tests, docs * fix: updated deps, fixes per lint checks * feat(validation-tests): additional testing * feat(dev): updated dev validate * docs: updated test docs, small cli print mods * fix: handle path indicies, docs updates * fix: rm unused test files * Update docs/reference/testing.md Co-authored-by: Brandt Keller <43887158+brandtkeller@users.noreply.github.com> * feat(dev): updated tests printing * fix: run tests test * fix: some refactor * fix: some reorg, names * feat(validation-tests): updated tests * fix: unit tests * fix: reinit validation * dev(validate): updated runtest logic and tests --------- Co-authored-by: Brandt Keller <43887158+brandtkeller@users.noreply.github.com>
- Loading branch information
1 parent
7a56e8c
commit 57aae78
Showing
28 changed files
with
2,445 additions
and
265 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| # Testing | ||
|
|
||
| Testing is a key part of Lula Validation development. Since the results of the Lula Validations are determined by the policy set by the `provider`, those policies must be tested to ensure they are working as expected. | ||
|
|
||
| ## Validation Testing | ||
|
|
||
| In the Lula Validation, a `tests` property is used to specify each test that should be performed against the validation. Each test is a map of the following properties: | ||
|
|
||
| - `name`: The name of the test | ||
| - `changes`: An array of changes or transformations to be applied to the resources used in the test validation | ||
| - `expected-result`: The expected result of the test - satisfied or not-satisfied | ||
|
|
||
| A change is a map of the following properties: | ||
|
|
||
| - `path`: The path to the resource to be modified. The path syntax is described below. | ||
| - `type`: The type of operation to be performed on the resource | ||
| - `update`: (default) updates the resource with the specified value | ||
| - `delete`: deletes the field specified | ||
| - `add`: adds the specified value | ||
| - `value`: The value to be used for the operation (string) | ||
| - `value-map`: The value to be used for the operation (map[string]interface{}) | ||
|
|
||
| An example of a test added to a validation is: | ||
|
|
||
| ```yaml | ||
| domain: | ||
| type: kubernetes | ||
| kubernetes-spec: | ||
| resources: | ||
| - name: podsvt | ||
| resource-rule: | ||
| version: v1 | ||
| resource: pods | ||
| namespaces: [validation-test] | ||
| provider: | ||
| type: opa | ||
| opa-spec: | ||
| rego: | | ||
| package validate | ||
| import future.keywords.every | ||
| validate { | ||
| every pod in input.podsvt { | ||
| podLabel := pod.metadata.labels.foo | ||
| podLabel == "bar" | ||
| } | ||
| } | ||
| tests: | ||
| - name: modify-pod-label-not-satisfied | ||
| expected-result: not-satisfied | ||
| changes: | ||
| - path: podsvt.[metadata.namespace=validation-test].metadata.labels.foo | ||
| type: update | ||
| value: baz | ||
| - name: delete-pod-label-not-satisfied | ||
| expected-result: not-satisfied | ||
| changes: | ||
| - path: podsvt.[metadata.namespace=validation-test].metadata.labels.foo | ||
| type: delete | ||
| ``` | ||
| There are two tests here: | ||
| * The first test will locate the first pod in the `validation-test` namespace and update the label `foo` to `baz`. Then a `validate` will be executed against the modified resources. The expected result of this is that the validation will fail, i.e., will be `not-satisfied`, which would result in a successful test. | ||
| * The second test will locate the first pod in the `validation-test` namespace and delete the label `foo`, then proceed to validate the modified resources and compare to the expected result. | ||
|
|
||
| ### Path Syntax | ||
|
|
||
| This feature uses the kyaml library to inject data into the resources, so the path syntax is based on this library. | ||
|
|
||
| The path should be a "." delimited string that specifies the keys along the path to the resource seeking to be modified. In addition to keys, a list item can be specified by using the “[some-key=value]” syntax. For example, the following path: | ||
|
|
||
| ``` | ||
| pods.[metadata.namespace=grafana].spec.containers.[name=istio-proxy] | ||
| ``` | ||
|
|
||
| Will start at the pods key, then since the next item is a [*=*] it assumes pods is a list, and will iterate over each item in the list to find where the key `metadata.namespace` is equal to `grafana`. It will then find the `containers` list item in `spec`, and iterate over each item in the list to find where the key `name` is equal to `istio-proxy`. | ||
|
|
||
| Multiple filters can be added for a list, for example the above example could be modified to filter both by namespace and pod name: | ||
|
|
||
| ``` | ||
| pods.[metadata.namespace=grafana,metadata.name=operator].spec.containers.[name=istio-proxy] | ||
| ``` | ||
|
|
||
| To support map keys containing ".", [] syntax will also be used, e.g., | ||
|
|
||
| ``` | ||
| namespaces.[metadata.namespace=grafana].metadata.labels.["some.key/label"] | ||
| ``` | ||
| Additionally, individual list items can be found via their index, e.g., | ||
| ``` | ||
| namespaces.[0].metadata.labels | ||
| ``` | ||
| Which will point to the labels key of the first namespace. Additionally, a `[-]` can be used to specify the last item in the list. | ||
| >[!IMPORTANT] | ||
| > The path will return only one item, the first item that matches the filters along the path. If no items match the filters, the path will return an empty map. | ||
| ### Change Type Behavior | ||
| **Add** | ||
| * All keys in the path must exist, except for the last key. If you are trying to add a map, then use `value-map` and specify the existing root key. | ||
| * If a sequence is "added" to, then the value items will be appended to the sequence. | ||
| **Update** | ||
| * If a sequence is "updated", then the entire sequence will be replaced. | ||
| **Delete** | ||
| * Currently only supports deleting a key, error will be returned if the last item in the path resolves to a sequence. | ||
| * No values should be specified for delete. | ||
| A note about replacing a key with an empty map - due to the way the `kyaml` library works, simply trying to overwrite an existing key with an empty map will not yield a removal of all the existing data of the map, it will just try and merge the differences, which is possibly not the desired outcome. To replace a map with an empty map, you must combine `delete` a change type and `add` a change type, e.g., | ||
| ```yaml | ||
| changes: | ||
| - path: pods.[metadata.namespace=grafana].metadata.labels | ||
| type: delete | ||
| - path: pods.[metadata.namespace=grafana].metadata | ||
| type: add | ||
| value-map: | ||
| labels: {} | ||
| ``` | ||
|
|
||
| Which will delete the existing labels map and then add an empty map, such that the "labels" key will still exist but will be an empty map. | ||
|
|
||
| ## Executing Tests | ||
|
|
||
| Tests can be executed by specifying the `--run-tests` flag when running `lula dev validate`. E.g., | ||
|
|
||
| ```sh | ||
| lula dev validate -f ./validation.yaml --run-tests | ||
| ``` | ||
|
|
||
| This will execute the tests and print the test results to the console. | ||
|
|
||
| To aid in debugging, the `--print-test-resources` flag can be used to print the resources used for each test to the validation directory, the filenames will be `<test-name>.json`.. E.g., | ||
|
|
||
| ```sh | ||
| lula dev validate -f ./validation.yaml --run-tests --print-test-resources | ||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.