META - Extension System Testing

[costrouc]

Context

#1833 is a huge PR. It need significant testing. Here is an exhaustive checklist

Value and/or benefit

Ensure this PR is successful and all previous behavior is preserved.

Providers

Preview Give feedback

1. AWS basic deployment #1951
   1 of 7
   project: JATIC +1
2. GCP deployment (custom vpc, gpu, preemptible) #1952
   project: JATIC +1
   
3. Azure basic deployment (custom vpc) #1953
   status: blocked ⛔️ +1
4. Digital Ocean basic deployment #1954
   project: JATIC +1
   

Extra

Preview Give feedback

1. META - Extension System Testing CLI Commands #1955
   10 of 12
   project: JATIC +1
2. [META] - Nebari testing suite enhancements #1958
   3 of 11
   area: testing ✅ good first issue type: enhancement 💅🏼 +3
   

When testing out a checklist item follow /~https://github.com/nebari-dev/nebari/issues/new?assignees=&labels=type%3A+release+%F0%9F%8F%B7&template=testing-checklist.md&title=Testing+checklist+for+%3Cversion%3E.

[costrouc]

Breaking up tasks:

• stream for cli testing "framework" @costrouc
• stream for cloud testing "framework" @aktech
• configuration "fixture" for specifying things like ("aws" + "gpu" + "github"), documentation or some easy way to allow more configuration matrix tests @iameskild
• specifying expectations of minimal "working check" e.g. for gpu with nvidia-smi working does that count?
• tear-downs for all clouds (load-balancers and storage cleaned up)
• schema fields all having descriptions and additional testing where needed. /~https://github.com/nebari-dev/nebari/pull/1833/files#diff-f7cda9f473c67dd06224a3ab445bb68065976dc6593511fab331179d60a50013R69 (stretch goal) not needed for this issue.

[aktech]

One note the cleaning of resources, it doesn't cleans quite well by the way:

2023-08-01 13:27:23      INFO   49 utils: destroying Nebari took 3220.010 [s]
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/07-kubernetes-services failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/05-kubernetes-keycloak failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/04-kubernetes-ingress failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/03-kubernetes-initialize failed to fully destroy
2023-08-01 13:27:23     ERROR  186 destroy: Stage=stages/02-infrastructure failed to fully destroy
2023-08-01 13:27:23     ERROR  189 destroy: ERROR: not all nebari stages were destroyed properly. For cloud deployments of Nebari typically only stages 01 and 02 need to succeed to properly destroy everything