[MXNET-1188] Maven Publishing Test Environments #13707

zachgk · 2018-12-21T01:54:55Z

Description

This PR is the second part of the maven Jenkins publishing that adds testing of the produced packages on 16.04, 18.04, and centOS. It also creates dedicated publishing dockerfiles (on 14.04) and testing dockerfiles that are built without any dependencies to ensure that the static linking worked properly.

Because the packages built on this do not yet use static linking, it instead tests against the snapshot repository.

Note that this PR depends on (and is rebased on top of) #13450. When reviewing the files, please look at only the changes in d4ac689.

Please review so the pipeline can be put into production:
@nswamy @lanking520 @andrewfayres @piyushghai @marcoabreu @yzhliu

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

zachgk · 2018-12-21T01:55:16Z

@mxnet-label-bot add [CI, Java, Maven, Scala, pr-awaiting-review]

marcoabreu

Approved d4ac689 only (as requested in the description)

sandeep-krishnamurthy · 2018-12-26T20:32:00Z

@frankfliu - You may be interested in this PR

lanking520

Overall looks good, please address the comments

ci/docker/install/centos7_scala.sh

lanking520

Overall looks good. Please also apply changes to the scripts in dev folder which reflect Frank's build changes

ci/docker/Dockerfile.publish.test.ubuntu1404_gpu

lanking520 · 2019-01-09T20:26:35Z

ci/docker/Dockerfile.publish.ubuntu1404_cpu

+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build and run MXNet on Ubuntu 16.04 for CPU


14.04, same as the one below

ci/docker/Dockerfile.publish.ubuntu1404_gpu

lanking520 · 2019-01-09T20:47:14Z

@aaronmarkham I tried a couple of time to restart the tests however it seemed not to be flaky. Could you please take a look in here?

piyushghai · 2019-01-09T20:48:18Z

ci/build.py

@@ -92,22 +92,24 @@ def get_dockerfiles_path():

 def get_platforms(path: str = get_dockerfiles_path()) -> List[str]:
    """Get a list of architectures given our dockerfiles"""
-    dockerfiles = glob.glob(os.path.join(path, "Dockerfile.build.*"))
+    dockerfiles = glob.glob(os.path.join(path, "Dockerfile.*"))


All the files added by you have Dockerfile.publish as the naming convention.
Should we add ".publish" as well to the wildcard file search ? Since for the purpose of this method is to get platforms for publishing artifacts.
WDYT ?

This is part of the main CI code and tries to find all Dockerfiles (for caching purposes). So, it should cover both the .build and .publish dockerfiles.

aaronmarkham · 2019-01-09T22:49:22Z

@aaronmarkham I tried a couple of time to restart the tests however it seemed not to be flaky. Could you please take a look in here?

I've seen this "bad address" error come up a few times recently. Restarting the CI pipeline for the failed test seems to work. I restarted it.

aaronmarkham · 2019-01-09T22:53:55Z

ci/docker/install/ubuntu_base.sh

+set -ex
+apt-get update || true
+apt-get install -y \
+    build-essential \


Why all of this when it happens with ubuntu_core.sh? What's the difference supposed to be?

The difference is the mxnet dependencies (opencv, cuda, blas, etc.). Core should include them because it uses them to build and test mxnet throughout the CI. base should not include them because it is only for testing whether the static linking works in the publishing pipeline and the tests might erroneously pass if the dependencies are installed on the system.

base should not include them because it is only for testing whether the static linking works in the publishing pipeline and the tests might erroneously pass if the dependencies are installed on the system.

@zachgk In that case we better test on a vanilla system? Why install all the development dependencies listed in ubuntu_base.sh if only testing the binaries? Could you help to clarify?

The goal is less to test binaries than to test Scala. So, we still need a few dependencies (jdk, maven, git, wget, etc). It's mostly the dependencies that are statically linked that need to be excluded.

lanking520

LGTM

andrewfayres

LGTM

* upstream/master: (109 commits) Code modification for testcases of various network models in directory example (apache#12498) [CI] Prevent timeouts when rebuilding containers with docker. (apache#13818) fix Makefile for rpkg (apache#13590) change to compile time (apache#13835) Disabled flaky test (apache#13758) Improve license_header tool by only traversing files under revision c… (apache#13803) Removes unneeded nvidia driver ppa installation (apache#13814) Add Local test stage and option to jump directly to menu item from commandline (apache#13809) Remove MXNET_STORAGE_FALLBACK_LOG_VERBOSE from test_autograd.py (apache#13830) Fix scala doc build break for v1.3.1 (apache#13820) [MXNET-1263] Unit Tests for Java Predictor and Object Detector APIs (apache#13794) [MXNET-1260] Float64 DType computation support in Scala/Java (apache#13678) onnx export ops (apache#13821) [MXNET-880] ONNX export: Random uniform, Random normal, MaxRoiPool (apache#13676) fix minor indentation (apache#13827) Fixing a symlink issue with R install (apache#13708) remove useless code (apache#13777) ONNX ops: norm exported and lpnormalization imported (apache#13806) Add new Maven build for Scala package (apache#13819) Dockerfiles for Publish Testing (apache#13707) ...

zachgk requested review from marcoabreu, nswamy, szha and yzhliu as code owners December 21, 2018 01:54

marcoabreu added CI Java Label to identify Java API component Maven pr-awaiting-review PR is waiting for code review Scala labels Dec 21, 2018

marcoabreu approved these changes Dec 23, 2018

View reviewed changes

lanking520 reviewed Dec 29, 2018

View reviewed changes

ci/docker/install/centos7_scala.sh Show resolved Hide resolved

zachgk mentioned this pull request Jan 3, 2019

[MXNET-1191] Maven Jenkins Pipeline with Static Binary and GPU #13767

Merged

5 tasks

zachgk force-pushed the testMaven branch from d4ac689 to 4e42062 Compare January 8, 2019 19:44

lanking520 reviewed Jan 9, 2019

View reviewed changes

piyushghai reviewed Jan 9, 2019

View reviewed changes

ci/docker/Dockerfile.publish.ubuntu1404_gpu Outdated Show resolved Hide resolved

piyushghai reviewed Jan 9, 2019

View reviewed changes

Dockerfiles for Publish Testing

16fcf34

aaronmarkham reviewed Jan 9, 2019

View reviewed changes

zachgk force-pushed the testMaven branch from 4e42062 to 16fcf34 Compare January 9, 2019 23:12

lanking520 approved these changes Jan 10, 2019

View reviewed changes

andrewfayres approved these changes Jan 10, 2019

View reviewed changes

lanking520 merged commit 30b552d into apache:master Jan 10, 2019

zachgk deleted the testMaven branch April 12, 2019 22:25

lanking520 pushed a commit to lanking520/incubator-mxnet that referenced this pull request Apr 26, 2019

Dockerfiles for Publish Testing (apache#13707)

e2a6c17

lanking520 pushed a commit to lanking520/incubator-mxnet that referenced this pull request Apr 30, 2019

Dockerfiles for Publish Testing (apache#13707)

abdd247

zachgk added a commit to zachgk/incubator-mxnet that referenced this pull request May 16, 2019

Dockerfiles for Publish Testing (apache#13707)

fede6d9

haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019

Dockerfiles for Publish Testing (apache#13707)

998048b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MXNET-1188] Maven Publishing Test Environments #13707

[MXNET-1188] Maven Publishing Test Environments #13707

zachgk commented Dec 21, 2018 •

edited

Loading

zachgk commented Dec 21, 2018

marcoabreu left a comment

sandeep-krishnamurthy commented Dec 26, 2018

lanking520 left a comment

lanking520 left a comment

lanking520 Jan 9, 2019

lanking520 commented Jan 9, 2019

piyushghai Jan 9, 2019

zachgk Jan 9, 2019

aaronmarkham commented Jan 9, 2019

aaronmarkham Jan 9, 2019

zachgk Jan 9, 2019

leezu Apr 6, 2020

zachgk Apr 6, 2020

lanking520 left a comment

andrewfayres left a comment

[MXNET-1188] Maven Publishing Test Environments #13707

[MXNET-1188] Maven Publishing Test Environments #13707

Conversation

zachgk commented Dec 21, 2018 • edited Loading

Description

Checklist

Essentials

zachgk commented Dec 21, 2018

marcoabreu left a comment

Choose a reason for hiding this comment

sandeep-krishnamurthy commented Dec 26, 2018

lanking520 left a comment

Choose a reason for hiding this comment

lanking520 left a comment

Choose a reason for hiding this comment

lanking520 Jan 9, 2019

Choose a reason for hiding this comment

lanking520 commented Jan 9, 2019

piyushghai Jan 9, 2019

Choose a reason for hiding this comment

zachgk Jan 9, 2019

Choose a reason for hiding this comment

aaronmarkham commented Jan 9, 2019

aaronmarkham Jan 9, 2019

Choose a reason for hiding this comment

zachgk Jan 9, 2019

Choose a reason for hiding this comment

leezu Apr 6, 2020

Choose a reason for hiding this comment

zachgk Apr 6, 2020

Choose a reason for hiding this comment

lanking520 left a comment

Choose a reason for hiding this comment

andrewfayres left a comment

Choose a reason for hiding this comment

zachgk commented Dec 21, 2018 •

edited

Loading