[MXNET-1224]: improve scala maven jni build. #13493

frankfliu · 2018-11-30T19:44:36Z

Description

Scala jni shared library requires compile flags inherited from make. But there is no way to enforce those flags to be the same as libmxnet.so was built. And current Scala package is built is manually. This is error prone can has caused issues in past.

This fix is trying make scala jni library not depends on any mxnet build flags, and dynamically link libmxnet.so directly. libmxnet-scala.so file will be compiled dynamically link to libmxnet.so, and allows load libmxnet.so from the same directory. Scala code is also updated to extract libmxnet.so file into temp folder and load it temp folder.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

nswamy · 2018-11-30T19:59:04Z

@frankfliu could you please explain what you intend to do.

vandanavk · 2018-11-30T20:55:49Z

@mxnet-label-bot add [Scala, pr-work-in-progress]

lanking520 · 2018-12-03T18:26:21Z

@nswamy this PR is intended to drop the requirement to have USE_XXX flags applied in the scala build process. Through using libmxnet.so and not libmxnet.a, it would help to improve the build process to be simpler.

@frankfliu could you please check if you have the link for ps-lite?

nswamy · 2018-12-04T17:57:57Z

scala-package/init-native/linux-x86_64/pom.xml

-            <compilerEndOption>-I${project.basedir}/../../../include</compilerEndOption>
-            <compilerEndOption>${all_includes}</compilerEndOption>
-            <compilerEndOption>${cflags}</compilerEndOption>
+            <compilerEndOption>-I${MXNET_DIR}/include</compilerEndOption>


Can we use the C_FLAGS and LD_FLAGS defined and passed from the Makefile

andrewfayres

LGTM, I really like these changes!

zachgk · 2018-12-06T00:11:30Z

scala-package/pom.xml

@@ -39,6 +39,8 @@
    <scala.version>2.11.8</scala.version>
    <scala.binary.version>2.11</scala.binary.version>
    <build.platform />
+    <cxx>g++</cxx>
+    <dollor>$</dollor>


Should this be dollar?

True, will fix this in next PR with maven build improvement.

piyushghai

Thanks for your contributions @frankfliu !

lanking520

Hi @frankfliu I am still not confident to merge this PR, could you please provide a short README in the Scala package explains the backend build procedure:

1. the reason you set these dependencies and why we limited to only dlmc, mshadow, dlpack? If I want to build with MKLDNN what will happen?
1. Structure of the result jar file

lanking520 · 2018-12-09T07:06:11Z

scala-package/init-native/osx-x86_64/pom.xml

+            <compilerEndOption>-I${MXNET_DIR}/3rdparty/dlpack/include</compilerEndOption>
+            <compilerEndOption>-I${MXNET_DIR}/3rdparty/tvm/nnvm/include</compilerEndOption>
+            <compilerEndOption>-DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0</compilerEndOption>
+            <compilerEndOption>-g -O0 -fPIC -msse3 -mf16c</compilerEndOption>


Why we are using these build options? What are these for?

lanking520 · 2018-12-09T07:08:11Z

scala-package/init-native/linux-x86_64/pom.xml

+            <compilerEndOption>-I${MXNET_DIR}/3rdparty/mshadow</compilerEndOption>
+            <compilerEndOption>-I${MXNET_DIR}/3rdparty/dlpack/include</compilerEndOption>
+            <compilerEndOption>-I${MXNET_DIR}/3rdparty/tvm/nnvm/include</compilerEndOption>
+            <compilerEndOption>-DMSHADOW_USE_MKL=0 -DMSHADOW_USE_CUDA=0</compilerEndOption>


Why we make them =0, if we build with GPU what will happened?

Added a README.md file to describe compiler and linker flags.

lanking520 · 2018-12-09T07:13:02Z

@marcoabreu looks like the CI passed however not reflect on this PR?

lanking520

Thanks for the readme and that make things clear! LGTM!

piyushghai

The readme is super useful!
Thanks @frankfliu

* upstream/master: (38 commits) Feature/mkldnn static (apache#13628) Fix the bug of BidirectionalCell (apache#13575) Set install path for libmxnet.so dynamic lib on Mac OS (apache#13629) add batch norm test (apache#13625) Scripts for building dependency libraries of MXNet (apache#13282) fix quantize pass error when the quantization supported Op are excluded in the model (apache#13596) Optimize C++ API (apache#13496) Fix warning in waitall doc (apache#13618) [MXNET-1225] Always use config.mk in make install instructions (apache#13364) [MXNET-1224]: improve scala maven jni build and packing. (apache#13493) [MXNET-1155] Add scala packageTest utility (apache#13046) fix the Float not showing correctly problem (apache#13617) apache#13385 [Clojure] - Turn examples into integration tests (apache#13554) Add Intel MKL blas to Jenkins (apache#13607) Revert "[MXNET-1198] MXNet Java API (apache#13162)" Reducing the length of setup tutorial (apache#13306) [MXNET-1182] Predictor example (apache#13237) [MXNET-1187] Added Java SSD Inference Tutorial for website (apache#13201) add defaults and clean up the tests (apache#13295) [MXNET-1181] Added command line alternative to IntelliJ in install instructions (apache#13267) ...

Major JNI feature changes. Please find more info here: https://cwiki.apache.org/confluence/display/MXNET/Scala+maven+build+improvement

frankfliu requested review from nswamy and yzhliu as code owners November 30, 2018 19:44

marcoabreu added pr-work-in-progress PR is still work in progress Scala labels Nov 30, 2018

lanking520 self-requested a review December 3, 2018 18:23

frankfliu force-pushed the jni branch 3 times, most recently from 8b99ff3 to abfd430 Compare December 4, 2018 03:33

frankfliu changed the title ~~[WIP]: improve scala maven jni build and packing.~~ [MXNET-1224]: improve scala maven jni build. Dec 4, 2018

nswamy reviewed Dec 4, 2018

View reviewed changes

andrewfayres approved these changes Dec 5, 2018

View reviewed changes

zachgk approved these changes Dec 6, 2018

View reviewed changes

piyushghai approved these changes Dec 6, 2018

View reviewed changes

lanking520 reviewed Dec 9, 2018

View reviewed changes

[MXNET-1224]: improve scala maven jni build and packing.

021d24e

frankfliu force-pushed the jni branch from abfd430 to 021d24e Compare December 11, 2018 03:43

lanking520 approved these changes Dec 11, 2018

View reviewed changes

andrewfayres approved these changes Dec 11, 2018

View reviewed changes

piyushghai approved these changes Dec 11, 2018

View reviewed changes

lanking520 merged commit b242b0c into apache:master Dec 11, 2018

frankfliu deleted the jni branch January 9, 2019 17:08

zachgk mentioned this pull request Jan 12, 2019

[Scala][macOS] Trying to build from source #12568

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-1224]: improve scala maven jni build. #13493

[MXNET-1224]: improve scala maven jni build. #13493

frankfliu commented Nov 30, 2018 •

edited

Loading

nswamy commented Nov 30, 2018

vandanavk commented Nov 30, 2018

lanking520 commented Dec 3, 2018

nswamy Dec 4, 2018

andrewfayres left a comment

zachgk Dec 6, 2018

frankfliu Dec 7, 2018

piyushghai left a comment

lanking520 left a comment

lanking520 Dec 9, 2018

lanking520 Dec 9, 2018

frankfliu Dec 11, 2018

lanking520 commented Dec 9, 2018

lanking520 left a comment

piyushghai left a comment

[MXNET-1224]: improve scala maven jni build. #13493

[MXNET-1224]: improve scala maven jni build. #13493

Conversation

frankfliu commented Nov 30, 2018 • edited Loading

Description

Checklist

Essentials

nswamy commented Nov 30, 2018

vandanavk commented Nov 30, 2018

lanking520 commented Dec 3, 2018

nswamy Dec 4, 2018

Choose a reason for hiding this comment

andrewfayres left a comment

Choose a reason for hiding this comment

zachgk Dec 6, 2018

Choose a reason for hiding this comment

frankfliu Dec 7, 2018

Choose a reason for hiding this comment

piyushghai left a comment

Choose a reason for hiding this comment

lanking520 left a comment

Choose a reason for hiding this comment

lanking520 Dec 9, 2018

Choose a reason for hiding this comment

lanking520 Dec 9, 2018

Choose a reason for hiding this comment

frankfliu Dec 11, 2018

Choose a reason for hiding this comment

lanking520 commented Dec 9, 2018

lanking520 left a comment

Choose a reason for hiding this comment

piyushghai left a comment

Choose a reason for hiding this comment

frankfliu commented Nov 30, 2018 •

edited

Loading