@mxnet-label-bot add [Build, Doc, Installation]

Also @aaronmarkham

Why are we documenting make instead of CMake? We want to completely switch over to CMake...

@larroy I am documenting make because as far as I know the scala package only works through make. If it works through cmake, is there anything blocking us from entirely switching all make commands to cmake throughout the install instructions?

Also, does cmake use config.mk as well? If it does, I think the same thing applies such that all cmake commands should use config.mk instead of passing arguments directly. I ran into a few cmake commands that did not use config.mk in the build_from_source.md, but I was unsure whether they could be switched or not.

I'm not familiar with the compilation in the Scala package. And no CMake doesn't work with config.mk

Just wanted to make you aware of the CMake efforts.

@mxnet-label-bot add [pr-awaiting-review]

@larroy The scala package compilation process currently requires a number of building and linking flags used to build the backend to be passed to the Java Native Interface as well. So, we currently call make and then make calls maven with those flags. Right now, we are working on improving the scala build so that it doesn't require those flags, but it doesn't seem to be trivial. Therefore, I added redundant flags to all cmake calls for now since it would at least result in the scala build always working.

Could you please raise a discussion on devlist?

I ran through this to test. Everything seemed to work fine except I had to specify USE_CUDA = 0 in the ubuntu_setup instructions.

@zachgk , @lanking520 , do you guys think it makes sense to merge this in?

I would suggest to converge in using CMake, I would propose to use a thin wrapper over cmake so we can use a file to define options in this fashion:

/~https://github.com/apache/incubator-mxnet/blob/master/cmake/cmake_options.yml

This would then work in the same way as config.mk now. But parsed into a commandline as it's done here:

/~https://github.com/apache/incubator-mxnet/blob/master/dev_menu.py#L48

Also, some of the changes proposed here don't work at all, have they been tested locally?

@larroy @aaronmarkham
I discussed the issue offline with lanking520 and andrewfayres. Right now, this PR is just a stopgap until we complete the final solution worked on in https://issues.apache.org/jira/browse/MXNET-1224. After that, we can just migrate all backend builds into cmake and that will be fine from the scala end.

In the meantime, we have a redundancy in our docs between the build_from_source instructions and the os specific files. Our current plan is to take advantage of it by leaving cmake in the build_from_source and make in the os specific. Then, we will tell all scala users that they must use the os specific instructions and not the build from source instructions. If they did build using cmake and the build_from_source instructions, they should "make clean" and remake it with the os specific instructions.

@larroy Can you point me to more info on "dev menu"? I think I understand what you're suggesting, but I don't see anything on the wiki or in the repo, other than the code itself.
Let me try to play back my understanding:

1. users only use cmake (so our instructions are simple and clear)
2. cmake uses the cmake_options.yml file
3. cmake or maybe dev_menu.py generates an artifact (config.mk) that make can use
4. scala's current make requirement is satisfied by this artifact

@aaronmarkham I extended dev_menu.py with a CMake builder helper that uses cmake options from a file, as the pattern of copying config.mk was useful for several people:

You can try what I mean by using ./dev_menu.py and choosing "Local build" check the CMake class inside that file:

/~https://github.com/apache/incubator-mxnet/blob/e0309f453175d066f8a147a3ff5afb01c575327f/dev_menu.py#L54

Or when this PR gets merged, one can do

cp cmake/cmake_options.yaml .
<edit if needed> 
./dev_menu.py build

/~https://github.com/apache/incubator-mxnet/pull/13529/files

Maybe you can give some feedback about this method, to see if you consider it would make sense to use this simplification on the documentation. So far I haven't documented any of this as I'm refining these convenience scripts and are more targeted to development.

I would like to mention that, unfortunately, the current option for algebra packages in CMake is BLAS and not USE_BLAS as in make.

@larroy
If we use the cmake_options approach, we will also need to configure cmake to work for scala similarly to how make works now. However, our plan is that we will actually drop make altogether in the near future and use mvn directly. So, I would rather just warn users for now instead of doing a temporary technological fix. Also, I corrected the cmake usages in the build_from_source file (they were broken in the old docs as well).

If you have no objections, I will also open a PR to move the rest of the install instructions to cmake once scala is ready.

@zachgk shall we leave Scala as it is with Make but move the others to CMake? that would be my preferred option.

I fixed the CPP instructions for now, but all of the other usages of make in the install docs could be reached on the path to installing scala (scala defers the engine build to the os specific pages)