doc: Update doc on switchback experiment (caraml-dev#78)

* Update doc on switchback experiment

* Update treatment validation test case

* Fix formatting

* Update PyYaml library

* Update Python version in Github workflows

* Fix lint error

.github/workflows/xp.yml

@@ -39,7 +39,7 @@ jobs:
         id: setup-python
         uses: actions/setup-python@v2
         with:
-          python-version: "3.7"
+          python-version: "3.10"
           architecture: x64
       - name: Install dependencies
         run: make install-python-ci-dependencies
@@ -185,7 +185,7 @@ jobs:
         id: setup-python
         uses: actions/setup-python@v2
         with:
-          python-version: "3.7"
+          python-version: "3.10"
           architecture: x64
 
       - name: Install dependencies

docs/concepts.md

@@ -56,7 +56,7 @@ XP support various experiment types.
 
 - **A/B Experiments** - Treatment assignment is randomized on the unit supplied in the request and one of the treatments in the experiment will be chosen at random, accounting for the traffic allocation for each treatment.
 - **Switchback Experiments** - The main idea behind switchback experiments is that the experiment engine switches back and forth between the control and treatment configurations, per configured time interval. In XP, switchback experiments can have one or more treatments and the engine cycles through them, selecting one treatment for all requests in every time interval.
-- **Randomized Switchback Experiments** - This is a hybrid between the A/B experiments and Switchbacks. These experiments are Switchbacks by nature (they have a time interval). In addition, they can have a traffic allocation on each of the treatments. Thus, at every new interval, the selection of the treatment is not cyclical, but randomized. In the default mode, all requests in a given time interval will still receive the same treatment. However, it is also possible to vary this.
+- **Randomized Switchback Experiments** - This is a hybrid between the A/B experiments and Switchbacks. These experiments are Switchbacks by nature (they have a time interval). In addition, they can have a traffic allocation on each of the treatments. Thus, at every new interval, the selection of the treatment is not cyclical, but randomized. All requests in a given time interval will receive the same treatment.
 
 #### Experiment Hierarchy
 

tests/e2e/test_validate_treatment.py

@@ -1,9 +1,12 @@
 from xp_client import XPClient
 
 
-def test_validate_valid_treatment_with_url(xp_client: XPClient):
+def test_validate_valid_treatment_with_url(xp_client: XPClient, httpserver):
+    # Start test server for external validation endpoint
+    httpserver.expect_request("/external", method="POST").respond_with_json({})
+
     data = {
-        "validation_url": "http://www.mocky.io/v2/5e4cacd4310000e1cad8c073",
+        "validation_url": httpserver.url_for("/external"),
         "data": {"field1": "abc", "field2": "def", "field3": {"field4": 0.1}},
     }
 

tests/requirements.txt

@@ -1,6 +1,7 @@
-PyYAML==5.4.1
+PyYAML==6.0.1
 google-cloud-pubsub==2.7.0
 requests==2.26.0
+pytest-httpserver==1.0.10
 python-dotenv==0.19.0
 s2cell==1.4.0
 pytz==2021.1

treatment-service/go.mod

@@ -8,7 +8,7 @@ require (
 	github.com/caraml-dev/mlp v1.7.7-0.20230428104022-779530aec912
 	github.com/caraml-dev/xp/clients v0.0.0-00010101000000-000000000000
 	github.com/caraml-dev/xp/common v0.0.0
-	github.com/confluentinc/confluent-kafka-go v1.8.2
+	github.com/confluentinc/confluent-kafka-go v1.9.2
 	github.com/deepmap/oapi-codegen v1.11.0
 	github.com/getkin/kin-openapi v0.94.0
 	github.com/go-chi/chi/v5 v5.0.7