Improve CCache handling #13456

marcoabreu · 2018-11-29T14:38:14Z

This PR improves our ccache access. Particularly, we are now able to able to cache NVCC calls (which are super expensive) and also add caching to the compilation of submodules (especially ps-lites' submodule zeroMQ likes to hardcode its compilers).

Just to give some numbers:

ARMv8 improved from 35s to 14s.
Dependency compilation on cpu and gpu got recuced from 7 minutes to 10 seconds
TODO: Expand

Full NVCC Ccache support for CMake is blocked by #13459

marcoabreu · 2018-11-29T17:14:46Z

CCache support for NVCC in CMake needs an upgrade. Tracked at #13459

vandanavk · 2018-11-29T17:18:51Z

@mxnet-label-bot add [CI, pr-awaiting-review]

marcoabreu · 2018-11-29T18:11:41Z

Ready for review. I will remove the Jenkinsfile changes when I'm done

This reverts commit e8386a7.

larroy · 2018-11-29T19:56:52Z

ci/docker/install/centos7_adduser.sh

+    # Later on, we have to override the links because underlying build systems ignore our compiler settings. Thus,
+    # we have to give the process the proper permission to these files. This is hacky, but unfortunately 
+    # there's no better way to do this without patching all our submodules.
+    chown -R jenkins_slave /usr/local/bin


Not clear, even with comment I don't understand why. Why can't we have some other folder added to the PATH?

We do, that's why I created /tmp/ccache-redirects. The problem is that in some places (especially submodules), that paths have been hardcoded. That's why we have to do both :(

larroy · 2018-11-29T19:57:46Z

ci/docker/runtime_functions.sh

+    ln -s ccache /tmp/ccache-redirects/clang-5.0
+    ln -s ccache /tmp/ccache-redirects/clang++-6.0
+    ln -s ccache /tmp/ccache-redirects/clang-6.0
+    ln -s ccache /usr/local/bin/gcc


Why don't you set some other folder in the path????

I did. But sometimes (in submodules) /usr/local/bin/gcc is referenced directly. In that case, my folder /tmp/ccache-redirects would be ignored.

Which submodules? Shouldn't we better fix them? This seems hacky, even though is great that you got it to work.

Ps-lite and ZeroMQ are the first ones that stuck to my mind. I didn't benchmark every single one.

In total, we spend about 5-8 minutes compiling submodules. After this PR it's 20 seconds.

Have you tried export CXX and CC? I don't see any references to /usr/local/bin/gcc in our source tree.

Could you point out where in the build files is the problem? This still seems super hacky and fragile to me.

Yes we used that method previously but zeromq ignored it for example. Have you searched for references to "GCC"? Zeromq is downloaded dynamically during the build (it's a zip file), so a grep would not find it.

Well that's the recommended way how ccache writes it in their guide. In the end, you can't control all build systems

https://ccache.samba.org/manual/latest.html#_run_modes

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

* Fix broken links * Make it idempotent fixes #13456 fixes #14117 fixes #11516

larroy · 2020-01-16T20:24:23Z

@josephevans please review, you said you got ccache to work with nvcc.

marcoabreu added 2 commits November 29, 2018 15:37

Remove gitignore entries

24d706c

Modify Makefile

0a2e773

marcoabreu requested a review from szha as a code owner November 29, 2018 14:39

marcoabreu and others added 4 commits November 29, 2018 15:40

Modify user permissions

84865ca

Add new ccache wrapper function

7649124

Change PATH rewrite to a different one to resolve CUDA issues

1d77c94

Add ccache to gpu cmake

f6fd7ec

marcoabreu added CI pr-awaiting-review PR is waiting for code review labels Nov 29, 2018

Enable ccache for every build

2a03107

marcoabreu added pr-work-in-progress PR is still work in progress and removed pr-awaiting-review PR is waiting for code review labels Nov 29, 2018

marcoabreu mentioned this pull request Nov 29, 2018

[WIP] Improve Ccache #11520

Closed

5 tasks

marcoabreu added 7 commits November 29, 2018 18:31

Set permissions for arm dockerfiles

0ceba10

Disable ccache for ASAN

ff92c48

Remove g++-8 ccache redirect

329283c

Update Android Dockerfiles for user permissions

9eb9f6f

Fix ASAN compiler typo

4abbaca

Remove sanity for speed

e8386a7

Move build dir creation in android armv8

ed16144

marcoabreu added pr-awaiting-review PR is waiting for code review and removed pr-work-in-progress PR is still work in progress labels Nov 29, 2018

Revert "Remove sanity for speed"

b61c17b

This reverts commit e8386a7.

marcoabreu changed the title ~~[WIP] New ccache~~ Improve CCache handling Nov 29, 2018

Add ccache for NVCC in Makefile

194d17b

larroy reviewed Nov 29, 2018

View reviewed changes

larroy added a commit to larroy/mxnet that referenced this pull request Apr 5, 2019

Fix build_ccache_wrappers:

13708f8

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 6, 2019

Fix build_ccache_wrappers:

6c8dc4e

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 9, 2019

Fix build_ccache_wrappers:

549d1f1

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 10, 2019

Fix build_ccache_wrappers:

a386644

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 10, 2019

Fix build_ccache_wrappers:

01540a2

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 16, 2019

Fix build_ccache_wrappers:

b1fc3ec

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 17, 2019

Fix build_ccache_wrappers:

3d6ccc4

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 19, 2019

Fix build_ccache_wrappers:

381ff13

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 19, 2019

Fix build_ccache_wrappers:

8546c73

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 19, 2019

Fix build_ccache_wrappers:

e577a79

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 22, 2019

Fix build_ccache_wrappers:

adfe27b

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 24, 2019

Fix build_ccache_wrappers:

af1e74e

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 24, 2019

Fix build_ccache_wrappers:

29eff62

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 24, 2019

Fix build_ccache_wrappers:

5e6ff5f

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 25, 2019

Fix build_ccache_wrappers:

f020fef

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Apr 29, 2019

Fix build_ccache_wrappers:

15611c1

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 2, 2019

Fix build_ccache_wrappers:

715c9f6

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 2, 2019

Fix build_ccache_wrappers:

77873b6

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 2, 2019

Fix build_ccache_wrappers:

b1c268b

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 6, 2019

Fix build_ccache_wrappers:

b62ca46

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 14, 2019

Fix build_ccache_wrappers:

dbfa8a9

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 14, 2019

Fix build_ccache_wrappers:

d8b3a8b

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 17, 2019

Fix build_ccache_wrappers:

2e28d46

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 20, 2019

Fix build_ccache_wrappers:

417ceae

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 21, 2019

Fix build_ccache_wrappers:

1c6c605

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request May 21, 2019

Fix build_ccache_wrappers:

6d98808

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Jun 11, 2019

Fix build_ccache_wrappers:

f336121

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

larroy added a commit to larroy/mxnet that referenced this pull request Jun 20, 2019

Fix build_ccache_wrappers:

bba4cb6

* Fix broken links * Make it idempotent fixes apache#13456 fixes apache#14117 fixes apache#11516

marcoabreu pushed a commit that referenced this pull request Jun 24, 2019

Fix build_ccache_wrappers: (#14631)

7fe478a

* Fix broken links * Make it idempotent fixes #13456 fixes #14117 fixes #11516

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve CCache handling #13456

Improve CCache handling #13456

marcoabreu commented Nov 29, 2018 •

edited

Loading

marcoabreu commented Nov 29, 2018

vandanavk commented Nov 29, 2018

marcoabreu commented Nov 29, 2018

larroy Nov 29, 2018

marcoabreu Nov 30, 2018

larroy Nov 29, 2018

marcoabreu Nov 30, 2018

larroy Nov 30, 2018

marcoabreu Nov 30, 2018

larroy Dec 3, 2018 •

edited

Loading

marcoabreu Dec 4, 2018

marcoabreu Dec 4, 2018

larroy commented Jan 16, 2020

Improve CCache handling #13456

Improve CCache handling #13456

Conversation

marcoabreu commented Nov 29, 2018 • edited Loading

marcoabreu commented Nov 29, 2018

vandanavk commented Nov 29, 2018

marcoabreu commented Nov 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larroy Dec 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

larroy commented Jan 16, 2020

marcoabreu commented Nov 29, 2018 •

edited

Loading

larroy Dec 3, 2018 •

edited

Loading