Merge pull request #1 from wolfhoundgelert/master

Release 0.2.0 candidate

README.md

@@ -1,11 +1,62 @@
 # xMan
 
-`xMan` (E**x**periment **Man**ager) is a `Python` library for effectively managing and organising experiments in `Jupyter Notebooks` on thе `Google Colab` platform or local machines, or right in the Python `REPL` (Read-Eval-Print Loop) interactive mode.
+`xMan` (E**x**periment **Man**ager) is a `Python` library for effective management and organising experiments in `Jupyter Notebooks` on thе `Google Colab` platform or local machines, or right in the Python `REPL` (Read-Eval-Print Loop) interactive mode.
 
 It offers the following key features:
 - ⭐ Keeping track of your experiments, providing a clear overview of your progress.
 - ⭐ Saving results in a centralised location, such as a shared Google Drive folder, with the option to use intermediate checkpoints.
-- ⭐ Running experiments in parallel across multiple Google Colab accounts, making it a powerful tool for AI, ML, NN, and DL students, learners, enthusiasts, researchers and their teams. 😍
+- ⭐ Rapid deployment and running experiments in parallel across multiple Google Colab accounts, making it a powerful tool for AI, ML, NN, and DL students, learners, enthusiasts, researchers and their teams. 😍
+
+In particular, provided:
+
+- Create, delete, rename, and move directories, and print directory tree with a depth limitation.
+
+- Create a new project, load existing projects specified by their directory paths, update and reload projects.
+
+- Print information about the project, specific groups or experiments, and retrieve various details such as location directory, directory tree, number, name, description, status, manual or automatic flag, etc.
+
+- All the information is saved in storage.
+
+- Retrieve groups and experiments by their numbers or names.
+
+- Manage groups and experiments - create, delete, edit, move, retrieve children and obtain various information about them.
+
+- Filter experiments and groups based on various parameters in different modes, with support for custom filters.
+
+- Retrieve experiments that are ready to start, initiate such experiments without manual checking, and support automatic start of the next experiment.
+
+- Support manual experiments and experiments with running pipelines, and view their results.
+
+- Set manual statuses and manual results for manual experiments.
+
+- Automatically set statuses for pipeline experiments.
+
+- Protection for currently running experiments.
+
+- Set a custom method for stringifying experiment results at the experiment, group, and project levels.
+
+- Set a custom result viewer for viewing experiment results at the experiment, group, and project levels.
+
+- Finalize experiments, groups, and projects after review, marking them with SUCCESS or FAIL statuses.
+
+- Add notes in txt, json, and pickle formats to experiments, groups, and project structures.
+
+- Save errors and error stacks in case of issues during pipeline execution.
+
+- Save checkpoints during pipeline execution and load them later or start the pipeline from a specific saved point.
+
+- Set markers on experiments to highlight specific experiments.
+
+And so on...
+
+## What for and why
+
+You are probably familiar with the following situations:
+
+1. You are looking for the optimal solution, just sorting through a lot of different network parameters and their combinations. After launching a dozen experiments, you find yourself losing track of what you've done and the corresponding results. Most importantly, you feel uncertain about which direction to pursue. It's a nightmare you'd rather forget. However, there's a way to salvage the situation. By structuring your experiments, conducting evaluative reviews, and keeping the results within easy reach, this tool will help you conserve effort and spare your nerves. You'll experience fewer chaotic launches, while significantly increasing the number of experiments. With a conscious approach, you can achieve exceptional model quality and bid farewell to the frustrations of the past. Explore [First glance](#first-glance) and [Performing experiments](#performing-experiments).
+
+
+2. You are hard at work on a project in Google Colab. Your network training requires substantial time and computational resources, relying on the GPU (a heartfelt thanks and appreciation to Google and the Colab team for making their infrastructure available to anyone ❤️). The training process has been going for the third hour, but, suddenly, disaster strikes – you've exhausted your daily GPU limit. Now, you face the daunting task of migrating your experiment to a new Colab account and redeploying everything from scratch. This process consumes considerable time and frays your nerves. You've not deployed under a second (let alone a third) account, making the prospect of working in parallel and avoiding idle time, because it seems overly complicated and bewildering. And it remains unclear how to piece everything together later in different Jupyter Notebooks. It's a nightmare best forgotten. But fear not! Now you can swiftly deploy under any account, even starting from a completely empty Jupyter Notebook. And don't forget to use checkpoints saving feature during your train execution. Simply refer to the [Pipelines with checkpoints](#pipelines-with-checkpoints) and [Running in parallel under different Google Colab accounts](#running-in-parallel-under-different-google-colab-accounts) section for seamless execution.
 
 
 
@@ -22,7 +73,8 @@ The `xMan` library can be downloaded from the `GitHub` [repository](https://gith
 </details>
 
 
-## First Glance
+
+## First glance
 <details><summary>Click to expand/collapse</summary>
 
 Let's assume that we work in Google Colab and want our experiments to be saved in Google Drive folder, which we can share lately among other Google Colab (and related Google Drive) accounts - it will be useful if we work in a team or/and want to execute experiments in parallel under different Google Colab accounts.
@@ -43,7 +95,7 @@ We can check our Google Drive directory:
 xman.dir_tree('./drive/MyDrive')
 ```
 
-Create a folder for our experiments specifying the path, name and description:
+Make a new project for our experiments specifying the path, name and description:
 ```python
 xman.make_proj('./drive/MyDrive/xman_exps', "My Proj Name", "My proj descr")
 ```
@@ -128,6 +180,7 @@ As you probably already noticed:
 </details>
 
 
+
 ## Sharing across Google Colab accounts
 <details><summary>Click to expand/collapse</summary>
 
@@ -159,7 +212,8 @@ xman.info()
 </details>
 
 
-## Performing Experiments
+
+## Performing experiments
 <details><summary>Click to expand/collapse</summary>
 
 A life cycle of any experiment, either it is `ML` experiment written in code or it is some live experiment performing on a physics lesson, can be described by statuses from the next workflow:
@@ -362,8 +416,11 @@ if cp_paths is not None:
 You can delete checkpoints with `xman.exp(2, 2).delete_checkpoints()`.
 
 </details>
+</details>
 
-### Running in parallel under different Google Colab accounts
+
+
+## Running in parallel under different Google Colab accounts
 <details><summary>Click to expand/collapse</summary>
 
 First, read the information about `Sharing across Google Colab accounts` in this document above. After you share your project, there are 2 ways of running experiments in parallel under different `Google Colab` accounts:
@@ -381,18 +438,30 @@ First, read the information about `Sharing across Google Colab accounts` in this
 
 All your results will be saved into the project shared `Google Drive` folder, so you can use as many accounts as you want. Regardless of which approach you'll choose, you can create any mess with or in your notebooks without worrying about it - all the results will be saved and organised in your project in one place. Sounds amazing, yeah? 😀 Just don't forget to duplicate your notebooks (not only share them) under different accounts to avoid conflicting between cell's output in the notebook's history.
 
-</details>
+> **Note:** `xMan` uses `cloudpickle` library for saving pipeline information for running exp in other runtimes. Sometimes `cloudpickle` can't resolve dependencies for saving and further extraction, e.g. some information from `datasets` or `nltk` libraries is placed to the storage. You have to perform such imports and initialisations in other runtime, or you can save them just into the `run_func` of your pipeline:
+   ```python
+   !pip install datasets
+   import datasets
+   dataset = datasets.load_dataset('ag_news')
+
+   import nltk
+   nltk.download('punkt')
+   ```
 
 </details>
 
+
+
 ## What's next
 
-- Tests coverage
-- API Documentation coverage
-- CI/CD
+- Tests coverage.
+- API Documentation coverage.
+- CI/CD for the library releases.
 - `grid-search` implementation (there are some issues with storage space consumption).
 - `xman.guide_me()` will advise you what to do next (load proj, create group, create exp, finalise completed exps and so on). 
 - Project management control modes, e.g. you can activate `STRICT` mode, in this case you're not allowed to make a new exp group until you finalise with `success` or `fail` all exps from the previous one - it will help you to keep things from getting messed up.
+- UI forms.
+
 
 
 ## Conclusion

TODO.md

@@ -4,6 +4,10 @@
 
 ### BUGS:
 
+- Colab cash doesn't allow to get actual info about experiments "/content/drive/.shortcut-targets-by-id/1G9R8MCoshlzIuEOrmbNzHnDSWEZi-4-b"
+  - There also "/content/drive/.Trash-0"
+  - There also "/content/drive/.file-revisions-by-id"
+
 - Can't upload `xman` on pypi.org 
   +  Write a letter about `xman` repo to `support@pypi.org`
   - /~https://github.com/pypi/support/issues/2738
@@ -21,12 +25,12 @@
   - Each protected method with some logic should be covered by tests
   - Each private method with some logic should be covered by tests
 
-- Add `__str__` and `_repr_pretty_` to XMan, show there the credits, version, quick help and links to documentation.
+- Add `__str__` and `_repr_pretty_` to xMan, show there the credits, version, quick help and links to documentation.
 
 
 ### BACKLOG:
 
-- [LOW] Add `__str__` and `_repr_pretty_` to XMan, show there the credits, version, quick help and links to documentation.
+- [LOW] Add `__str__` and `_repr_pretty_` to xMan, show there the credits, version, quick help and links to documentation.
 
 - [LOW] How to solve the issue of multiple `update`-s in a chain `a.foo().bar().biz()`. I need to register each call in some `UpdateManager`, if it's the first call - set `is_chain` status to `True`, this status activates some other thread (e.g. by minimal timer) for setting `is_chain` back to `False`. Each of methods (foo, bar, biz) check `is_chain` flag, and if it's `True`, they skip `update`. In this case, the flag will be cleared right after the chain and the next chain starts again with `update`. I can add this logic to the root `super().update()` and it will return `False` if the chain were marked:
   ```

requirements.txt

@@ -1,3 +1,4 @@
 cloudpickle~=2.2.1
 pytest~=7.3.1
-setuptools~=65.5.1
+setuptools~=65.5.1
+# dill~=0.3.6

setup.py

@@ -1,42 +1,33 @@
 from setuptools import setup
+import re
+
+
+with open('src/xman/__init__.py') as f:
+    version_match = re.search(r'^__version__\s*=\s*[\'"]([^\'"]*)[\'"]', f.read(), re.MULTILINE)
+    if version_match:
+        version = version_match.group(1)
+    else:
+        raise RuntimeError("Version not found!")
+
+
+with open('src/requirements.txt') as f:
+    requirements = f.read().splitlines()
+
 
 setup(
-    name='xman',
-    version='0.1.0',
+    name='xMan',
+    version=version,
     packages=['xman'],
     package_dir={'': 'src'},
     url='/~https://github.com/wolfhoundgelert/xman',
     license='BSD 3-clause',
     author='Andrei Polkanov',
     author_email='andrey.polkanoff@gmail.com',
-    description='Experiment Manager (xman) [IN_PROGRESS]'
+    description='xMan - Experiment Manager, python library for effective management and organising '
+                'experiments on thе Google Colab platform with sharing and running in parallel '
+                'across different Colab accounts.',
+    install_requires=requirements,
 )
 
-# TODO /~https://github.com/pypi/support/issues/2738
-
 
-
-# TODO >>>>> support setting from setup.py
-    # import re
-    #
-    # def get_current_version():
-    #     with open('xman/__init__.py', 'r') as f:
-    #         version_file = f.read()
-    #     version_match = re.search(r"^__version__ = ['\"]([^'\"]*)['\"]", version_file, re.M)
-    #     if version_match:
-    #         return version_match.group(1)
-    #     raise RuntimeError("Unable to find version string in __init__.py")
-    #
-    # def get_updated_version():
-    #     current_version = get_current_version()
-    #     major, minor, patch = current_version.split('.')
-    #     patch = int(patch) + 1
-    #     return f"{major}.{minor}.{patch}"
-    #
-    # from setuptools import setup
-    #
-    # setup(
-    #     name='xman',
-    #     version=get_updated_version(),
-    #     ...
-    # )
+# TODO /~https://github.com/pypi/support/issues/2738

src/requirements.txt

@@ -0,0 +1,2 @@
+# dill~=0.3.6
+cloudpickle~=2.2.1

src/tmptest.py

@@ -3,62 +3,19 @@
 
 from xman import xman
 
-proj_dir = '../gitignore/experiments'
 
+PROJ_DIR = '../gitignore/experiments'
 
-from xman import filesystem
-# filesystem.__save_json('test string\another string', os.path.join(proj_dir, 'test.json'))
-print("test string\u0007nother string")
 
-xman.exp(1, 1).note.json = {"param1": 123, "param2": 'dsfas'}
-xman.exp(1, 1).note.text = "Some text here"
-xman.exp(1, 1).note.pickle = some_object
+create_new = True
 
-xman.exp(1, 1).note.json
-xman.exp(1, 1).note.txt
-xman.exp(1, 1).note.pickle
+if create_new:
+    if os.path.exists(PROJ_DIR):
+        shutil.rmtree(PROJ_DIR)
+    xman.make_proj(PROJ_DIR, 'Test Project', "Test project descr")
+    xman.make_group("Test Group", "Test group descr")
+    xman.make_exp(1, "Test Exp", "Test exp descr")
+else:
+    xman.load_proj(PROJ_DIR)
 
-# xman.dir_tree(proj_dir)
-#
-# create_new = True
-# if create_new:
-#     if os.path.exists(proj_dir):
-#         shutil.rmtree(proj_dir)
-#     xman.make_proj(proj_dir, 'Test Project', "This is a test experimental project")
-#     xman.proj.make_group("Test Group", "Test group descr")
-#     xman.proj.group(1).make_exp("Test Exp", "Test Exp Descr")
-# else:
-#     xman.load_proj(proj_dir)
-#
-# xman.dir_tree(proj_dir)
-# xman.proj.info()
-#
-#
-#
-#
-# def train(pulse, p1, p2, np1=None, np2=None):
-#     print("train started")
-#     import time
-#     time.sleep(10)
-#     print("train is still in progress...")
-#     result = f"p1={p1}, p2={p2}, np1={np1}, np2={np2}"
-#     time.sleep(10)
-#     print(f"train finished with result `{result}`")
-#     return result
-#
-# params = {
-#     'p1': 1,
-#     'p2': 2,
-#     'np1': 3,
-#     'np2': 4,
-# }
-
-# xman.proj.exp('1.1').destroy_pipeline().make_pipeline(train, params)
-# xman.proj.info()
-# xman.proj.exp('1.1').start()
-# xman.proj.info()
-# print(xman.proj.exp('1.1').result)
-
-# xman.proj.exp(1.1).set_manual_result('DCG@   1: 0.250 | Hits@   1: 0.250\nDCG@   5: 0.316 | Hits@   5: 0.376\nDCG@  10: 0.339 | Hits@  10: 0.448\nDCG@ 100: 0.389 | Hits@ 100: 0.691\nDCG@ 500: 0.419 | Hits@ 500: 0.927\nDCG@1000: 0.427 | Hits@1000: 1.000\n')
-# xman.proj.exp('1.1').info()
-# xman.proj.info()
+xman.exp(1, 1).info()

src/xman/__init__.py

@@ -1,4 +1,4 @@
-__version__ = '0.1.0'  # TODO support auto setting from setup.py
+__version__ = '0.2.0'
 
 
 from .api import XManAPI