Rebuilding the Ray version of data-processing-lib and transforms with an alternative to Ray

[ritvikrao]

I am working on running the Ray versions of data-prep-kit pipelines, with one change: I'm using an experimental Charm4Py (/~https://github.com/charmplusplus/charm4py)-based version of the Ray core API in lieu of the official Ray module. The idea is to replace the Ray scheduler with the Charm4Py scheduler, but using the same API. Essentially, this requires me to change

import ray

to

from charm4py import ray

in the data processing library and in code transforms. Then, when rebuilding the data processing library and the transform libraries, I build the data_prep_toolkit wheels (pip install data_prep_toolkit-0.2.4.dev0-py3-none-any.whl) to install the libraries into my virtual environment. I can then use the 'data_processing_ray' and 'data_processing' modules successfully, but I get a ModuleNotFoundError when trying to access a module such as 'dpk_ededup'. How do I properly build and install the preprocessing libraries?

[agoyal26]

@matouma @shivdeep-singh-ibm Please see ^^

[shivdeep-singh-ibm]

instead of building and installing dpk_ededup wheels, you can set up PYTHONPATH to transforms/universal/ededup and replace import ray by from charm4py import ray in your environment. This way you would be able to test your implementation.

[ritvikrao]

Thank you, this seems to work. If I wanted to test a pipeline with multiple modules, however, would I need to change the PYTHONPATH each time? For example, if I wanted to use the fdedup module and dpk_fdedup, would I need to change the PYTHONPATH in my .py code or in a Jupyter notebook?

[shivdeep-singh-ibm]

There are two ways.

a) you can append all paths of modules to your PYTHONPATH and run jupyter notebook
$ export PYTHONPATH=:
$ . ./venv/bin/activate && jupyter lab

b) In your jupyter notebook, you can use sys.path.append() and append all paths you want, this way you can dynamicaly append paaths and achive same thing as a jupyter notebook.