Merge pull request #436 from juripetersen/wayang-python-api

Add wayang-api-json and wayang-api-python to create WayangPlans from python

build/pyplangenerator.sh

@@ -22,5 +22,5 @@ dir=$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)
 cd ${dir}/..
 
 protoc -I=./wayang-commons/wayang-serializable/src/main/proto \
---python_out=./python/src/pywayang/protobuf/ \
-pywayangplan.proto
+--python_out=./python/src/pywy/platforms/jvm/serializable \
+pywayangplan.proto

conf/spark/default.properties

@@ -18,4 +18,5 @@
 spark.master = local[1]
 spark.app.name = Wayang App
 spark.ui.showConsoleProgress = false
-spark.driver.allowMultipleContexts=true
+spark.driver.allowMultipleContexts=true
+

pom.xml

@@ -849,7 +849,6 @@
 
         <!-- Coverage -->
         <!-- TODO: add the coverage elements -->
-
     </dependencies>
 
     <build>
@@ -1141,7 +1140,7 @@
                     </includedLicenses>
                     <failOnBlacklist>true</failOnBlacklist>
                     <excludedGroups>
-                        org.apache.spark.*|org.apache.hadoop.*|org.apache.giraph.*|org.antlr.*|junit.*|org.graphchi.*
+                        org.apache.spark.*|org.apache.hadoop.*|org.apache.giraph.*|org.antlr.*|junit.*|org.graphchi.*|org.springframework.*
                     </excludedGroups>
                     <excludeTransitiveDependencies>true</excludeTransitiveDependencies>
                 </configuration>

python/.gitignore

@@ -1,277 +1,3 @@
+# Build files
+dist
 
-# Created by https://www.toptal.com/developers/gitignore/api/intellij+all,python,macos
-# Edit at https://www.toptal.com/developers/gitignore?templates=intellij+all,python,macos
-
-### Intellij+all ###
-# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
-# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
-
-# User-specific stuff
-.idea/**/workspace.xml
-.idea/**/tasks.xml
-.idea/**/usage.statistics.xml
-.idea/**/dictionaries
-.idea/**/shelf
-.idea/*
-
-# AWS User-specific
-.idea/**/aws.xml
-
-# Generated files
-.idea/**/contentModel.xml
-
-# Sensitive or high-churn files
-.idea/**/dataSources/
-.idea/**/dataSources.ids
-.idea/**/dataSources.local.xml
-.idea/**/sqlDataSources.xml
-.idea/**/dynamic.xml
-.idea/**/uiDesigner.xml
-.idea/**/dbnavigator.xml
-
-# Gradle
-.idea/**/gradle.xml
-.idea/**/libraries
-
-# Gradle and Maven with auto-import
-# When using Gradle or Maven with auto-import, you should exclude module files,
-# since they will be recreated, and may cause churn.  Uncomment if using
-# auto-import.
-# .idea/artifacts
-# .idea/compiler.xml
-# .idea/jarRepositories.xml
-# .idea/modules.xml
-# .idea/*.iml
-# .idea/modules
-*.iml
-# *.ipr
-
-# CMake
-cmake-build-*/
-
-# Mongo Explorer plugin
-.idea/**/mongoSettings.xml
-
-# File-based project format
-*.iws
-
-# IntelliJ
-out/
-
-# mpeltonen/sbt-idea plugin
-.idea_modules/
-
-# JIRA plugin
-atlassian-ide-plugin.xml
-
-# Cursive Clojure plugin
-.idea/replstate.xml
-
-# SonarLint plugin
-.idea/sonarlint/
-
-# Crashlytics plugin (for Android Studio and IntelliJ)
-com_crashlytics_export_strings.xml
-crashlytics.properties
-crashlytics-build.properties
-fabric.properties
-
-# Editor-based Rest Client
-.idea/httpRequests
-
-# Android studio 3.1+ serialized cache file
-.idea/caches/build_file_checksums.ser
-
-### Intellij+all Patch ###
-# Ignore everything but code style settings and run configurations
-# that are supposed to be shared within teams.
-
-.idea/*
-
-!.idea/codeStyles
-!.idea/runConfigurations
-
-### macOS ###
-# General
-.DS_Store
-.AppleDouble
-.LSOverride
-
-# Icon must end with two \r
-Icon
-
-
-# Thumbnails
-._*
-
-# Files that might appear in the root of a volume
-.DocumentRevisions-V100
-.fseventsd
-.Spotlight-V100
-.TemporaryItems
-.Trashes
-.VolumeIcon.icns
-.com.apple.timemachine.donotpresent
-
-# Directories potentially created on remote AFP share
-.AppleDB
-.AppleDesktop
-Network Trash Folder
-Temporary Items
-.apdisk
-
-### Python ###
-# Byte-compiled / optimized / DLL files
-__pycache__/
-*.py[cod]
-*$py.class
-
-# C extensions
-*.so
-
-# Distribution / packaging
-.Python
-build/
-develop-eggs/
-dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-share/python-wheels/
-*.egg-info/
-.installed.cfg
-*.egg
-MANIFEST
-
-# PyInstaller
-#  Usually these files are written by a python script from a template
-#  before PyInstaller builds the exe, so as to inject date/other infos into it.
-*.manifest
-*.spec
-
-# Installer logs
-pip-log.txt
-pip-delete-this-directory.txt
-
-# Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
-.coverage
-.coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
-.hypothesis/
-.pytest_cache/
-cover/
-
-# Translations
-*.mo
-*.pot
-
-# Django stuff:
-*.log
-local_settings.py
-db.sqlite3
-db.sqlite3-journal
-
-# Flask stuff:
-instance/
-.webassets-cache
-
-# Scrapy stuff:
-.scrapy
-
-# Sphinx documentation
-docs/_build/
-
-# PyBuilder
-.pybuilder/
-target/
-
-# Jupyter Notebook
-.ipynb_checkpoints
-
-# IPython
-profile_default/
-ipython_config.py
-
-# pyenv
-#   For a library or package, you might want to ignore these files since the code is
-#   intended to run in multiple environments; otherwise, check them in:
-# .python-version
-
-# pipenv
-#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
-#   However, in case of collaboration, if having platform-specific dependencies or dependencies
-#   having no cross-platform support, pipenv may install dependencies that don't work, or not
-#   install all needed dependencies.
-#Pipfile.lock
-
-# poetry
-#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
-#   This is especially recommended for binary packages to ensure reproducibility, and is more
-#   commonly ignored for libraries.
-#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
-#poetry.lock
-
-# PEP 582; used by e.g. github.com/David-OConnor/pyflow
-__pypackages__/
-
-# Celery stuff
-celerybeat-schedule
-celerybeat.pid
-
-# SageMath parsed files
-*.sage.py
-
-# Environments
-.env
-.venv
-env/
-venv/
-ENV/
-env.bak/
-venv.bak/
-
-# Spyder project settings
-.spyderproject
-.spyproject
-
-# Rope project settings
-.ropeproject
-
-# mkdocs documentation
-/site
-
-# mypy
-.mypy_cache/
-.dmypy.json
-dmypy.json
-
-# Pyre type checker
-.pyre/
-
-# pytype static type analyzer
-.pytype/
-
-# Cython debug symbols
-cython_debug/
-
-# PyCharm
-#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
-#  be found at /~https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
-#  and can be added to the global gitignore or merged into this file.  For a more nuclear
-#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
-
-# End of https://www.toptal.com/developers/gitignore/api/intellij+all,python,macos

python/README.md

@@ -20,3 +20,78 @@ under the License.
 # PyWayang
 
 Implementation of a python API for Apache Wayang
+
+## Building from source
+To build and install `pywy` as a python library, `build` is needed. It
+can be installed using:
+```shell
+cd ./python
+pip install --upgrade build
+```
+Installing this might require `python3.8-venv` to be installed on the
+system.
+
+After building the python package, execute the following steps to make
+it available for your system:
+```shell
+python3 -m pip install dist/pywy-0.7.1.tar.gz
+```
+
+## Executing python code
+In order to execute python code, the REST API needs to be running.
+Compiling the assembly and executing the Main class for the REST API can
+be done with the following steps:
+
+
+Before compiling your code, make sure the required configuration variables
+are set correctly in `wayang-api/wayang-api-python/src/main/resources/wayang-api-python-defaults.properties`.
+This example `wayang-api-python-defaults.properties` can be used as a
+guideline:
+
+```
+wayang.api.python.worker = /var/www/html/python/src/pywy/execution/worker.py
+wayang.api.python.path = python3
+wayang.api.python.env.path = /usr/local/lib/python3.8/dist-packages
+```
+The first configuration value needs to point to the location of your
+apache wayang repository so that it can find the python worker that
+executes UDFs (usually your current work directory +
+`python/src/pypy/execution/worker.py`)
+The second value is your command used for invoking python scripts.
+Usually either python3 or just python.
+The third value points to the directory in which python libraries are to
+be found (usually where pip installs them).
+
+- Package the project
+```shell
+./mvnw clean package -pl :wayang-assembly -Pdistribution
+```
+
+- Starting the REST API as a background process
+```shell
+cd wayang-assembly/target/
+tar -xvf apache-wayang-assembly-0.7.1-SNAPSHOT-incubating-dist.tar.gz
+cd wayang-0.7.1-SNAPSHOT
+./bin/wayang-submit org.apache.wayang.api.json.Main &
+```
+
+Now, create and execute a python script like this:
+
+```python
+from pywy.dataquanta import WayangContext
+from pywy.platforms.java import JavaPlugin
+from pywy.platforms.spark import SparkPlugin
+
+def word_count():
+    ctx = WayangContext() \
+        .register({JavaPlugin, SparkPlugin}) \
+        .textfile("file:///README.md") \
+        .flatmap(lambda w: w.split()) \
+        .filter(lambda w: w.strip() != "") \
+        .map(lambda w: (w.lower(), 1)) \
+        .reduce_by_key(lambda t: t[0], lambda t1, t2: (t1[0], int(t1[1]) + int(t2[1]))) \
+        .store_textfile("file:///wordcount-out-python.txt")
+
+if __name__ == "__main__":
+    word_count
+```