-
Notifications
You must be signed in to change notification settings - Fork 9
GNI provider building it
- make sure you are building with GNU C, i.e. make sure the PrgEnv-gnu module is loaded. You will need this to pick up recent versions of gcc. The GNI provider requires gcc 4.9 or higher.
- you will need newer versions of automake, etc. than are available by default on Cray CLE 5.X (sles11sp3) systems. Cray developers can use the autotools, etc. installed in /cray/css/users/sungeun/local/bin.
- unload the ATP module, it messes with our cray_runall.sh and auxiliary testing scripts.
Note that the GNI provider requires CLE 5.2up04 or later owing to GNI API features introduced in that release. There are workarounds to build the GNI provider on Edison (running CLE 5.2up03 as of this update) which are described below.
Make sure you have the PrgEnv-gnu loaded.
You will need to install your own autotools -the autoconf, automake, and libtool modules at NERSC don't work because the libtool macros needed by aclocal are installed in a different directory than where aclocal expects them. A known combination of tools which works is
- automake 1.14
- autoconf 2.69
- libtool 2.4.
These packages should all be installed into the same location, i.e. you should use the same --prefix=MYINSTALLDIR argument when configuring/installing all three packages.
Once you installed the packages and made sure that the MYINSTALLDIR/bin is at the front of your PATH, you're almost there. From the top level directory of your libfabric git clone do:
./autogen.sh
./configure --prefix=MY_LIBFABRIC_TEST_INSTALL_DIR
make install
As a sanity check, you may wish to configure to build the criterion tests. See Building-and-running-the-unit-tests-(gnitest). Note that installation of criterion itself is non-trivial.
Note on many Cray systems (esp. with CLE 6) there are sufficient ibverbs related components installed to cause the libfabric configure to try to build the verbs provider, but in fact that is not the case. To avoid build problems, it may be necessary to add the
--disable-verbs
option to the configure line.
Another configure option to consider enables kdreg support in the GNI provider. kdreg serves much the same purpose as the ummunotify device driver was to serve, but which was never accepted into mainline Linux. In order to use kdreg, an application has to have a couple of header files from the kdreg devel rpm installed on the system. Cray has only provided this devel rpm to customers as part of CLE 6. To enable kdreg support in the GNI provider, the following configury option should be used
--with-kdreg=/opt/cray/kdreg/default
The GNI provider uses functionality from the uGNI library that is only available in CLE 5.2up04 and later. However, since the only changes were in user space components (the uGNI library and gni_pub.h include files), it is possible build the GNI provider if you can install the cray-libugni cray-libugni-devel and cray-gni-headers RPMs from a newer release in a non-default location on your system. You will need to manually edit the packages respective pkg-config config files to correctly return the alternate location for the install. This later step is needed or the GNI provider configury will not be able to find the correct uGNI header files and libraries.
In addition to the install of these RPMS, you will also need to adjust your ``PKG_CONFIGandLD_LIBRARY_PATH``` environment variables. Assuming the RPMS are installed starting at some path A, the environment variables will need to be updated as follows:
export PKG_CONFIG_PATH=A/ugni/cray-version-number-for-package/lib64/pkgconfig:A/gni-headers/cray-version-number-for-package/lib64/pkgconfig:${PKG_CONFIG_PATH}
export LD_LIBRARY_PATH=A/ugni/cray-version-number-for-packages/lib64:${LD_LIBRARY_PATH}
Users of your libfabric build will also need to pre-pend the path to the libugni.so to their LD_LIBRARY_PATH as well, assuming they are linking dynamically.
A similar approach can be taken for using the kdreg configure option on Cray XC's not running CLE 6.0 or higher.