[Fedora-packaging] Draft for the use of environment-modules

Jussi Lehtola jussilehtola at fedoraproject.org
Fri Jul 24 13:17:52 UTC 2009

Hi all,

I've done some updates on the draft at
according to Milos' suggestions.

I'm not quite sure if MPI software spec files, too, should be 3rd party
compiler compatible as the MPI compiler spec files. 

On Fri, 2009-07-24 at 08:46 -0400, Doug Ledford wrote:
> On Jul 23, 2009, at 5:56 PM, Milos Jakubicek wrote:
> > Well done, I think you could take also some inspiration from hints  
> > made by Doug Ledford in BZ#511099:
> >
> > - what about making the $MPI_HOME, $MPI_BIN and $MPI_LIB variables a  
> > MUST?
> I would strongly support this.

OK, but I really don't see what these are used for. AFAIK they're not
used by the MPI compiler for anything, as everything is done by the

> > - we should make the sample of specfile from the BZ#511099 part of  
> > the draft
> >
> > Some other things that come to my mind:
> >
> > - the libraries must have the suffix OR they could be installed  
> > under %{_libdir}/%{name}/%{version}-<MPI compiler>/lib (i.e. there  
> > should similar two choices as for the binaries)
> I would say the libraries MUST be installed under $MPI_LIB and any  
> resulting binaries MUST be installed under $MPI_BIN.  This makes is so  
> that loading the MPI environment module automatically gets you all the  
> associated binaries and libraries needed.

That's a valid option, but makes updating the MPI compiler impossible
since at least currently with openmpi the dirs are versioned:

setenv                  MPI_BIN         /usr/lib/openmpi/1.3.3-gcc/bin
setenv                  MPI_LIB         /usr/lib/openmpi/1.3.3-gcc/lib
setenv                  MPI_HOME        /usr/lib/openmpi/1.3.3-gcc

Of course, installing in a non-versioned directory is a valid option,
this would just need a couple of definitions more in the MPI module
files. I'd rename the existing MPI_{BIN,LIB,HOME} variables to
MPICORE_{BIN,LIB,HOME} and define the following variables:

MPI_BIN		%{_libdir}/openmpi/gcc/bin
MPI_LIB		%{_libdir}/openmpi/gcc/lib
MPI_MAN		%{_libdir}/openmpi/gcc/man
MPI_INCLUDE	%{_libdir}/openmpi/gcc/include
MPI_HOME	%{_libdir}/openmpi/gcc/

Out of these, include and man are not usually needed.

> > - each MPI build of shared libraries should have a separate -devel  
> > subpackage (having them in a single -devel subpackage would need a  
> > require of all MPI builds for it, which is imho not good)
> This is probably good, and would need to be under $MPI_INCLUDE maybe?   
> It might be worth while to consider simply making it a rule that the  
> following directories always exist:

This is very important and as such is a MUST.

> So, as per our conversation in IRC, here's the directory struction I  
> would recommend.  This is brought out by the fact that the current MPI  
> directory structure I'm using seriously abuses %{_libdir} by putting  
> way too much stuff under there.  We don't really want to use /opt  
> (even though that's really the perfect place for this) because of the  
> whole issue of wanting this to be able to be mounted as a read only  
> network filesystem and putting it in /opt adds a new mount point (and  
> also adds complexity to drive partitioning issues).  So, keeping it  
> under /usr would seem to be a very important goal.  However, we have  
> binaries, man pages, libraries, include files, and now we are talking  
> about other packages tossing their stuff under these directories too.   
> That is a rather excessive abuse of %{_libdir}.  The FHS doesn't  
> really have an option to cover this.  At all.  I don't think we can  
> accomplish what needs done while adhering strictly to the FHS.  So, I  
> think there are two options:

This is an issue I have been having for a long time with some packages
that have general names of binaries.

> 1) Create a suitable MPI_HOME that can hold the entire bunch of stuff  
> from a given MPI package (maybe something like /usr/opt, /usr/local/ 
> mpi, or /usr/mpi) and under that top level directory install the  
> multiple mpi packages and also allow for the installation of mpi  
> linked libraries and binaries.

/usr/mpi sounds like a good idea. But it needs to be multiarch/multilib
compatible, e.g. 32-bit and 64-bit libraries need to be able to coexist.
(Well, we can always use e.g. /usr/mpi/openmpi/lib
and /usr/mpi/openmpi/lib64...)

> 2) Try to split things up into normal FHS locations, but don't use  
> binary name suffixes or library name suffixes in the normal  
> directories, instead use things like /bin/%{name}-%{_arch}%{? 
> opt_cc_suffix}, %{_libdir}/%{name}%{?opt_cc_suffix}, %{_includedir}/% 
> {name}-%{_arch}%{?opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{? 
> opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{?opt_cc_suffix}/man.   
> While the above is somewhat clumsy, and also eliminates the  
> possibility of MPI_HOME being useful as we would now need independent  
> definitions of all the various MPI locations, it does have the benefit  
> of being mostly FHS compliant.

OK, this holds for MPI compilers, and the directories can be reused for
other software, but may be a bit messy..

> Regardless of which way people would prefer to go, I do think we  
> should go ahead and standardize all the config files in %{_sysconfdir}/ 
> %{name}-%{_arch}%{?opt_cc_suffix} so that they won't possibly be on a  
> read only filesystem.

So far I haven't seen any MPI software with a config file in /etc. 

> Thoughts?  If we can get this standard more or less agree on, I'll get  
> the F-11 and devel packages of both lam and openmpi updated the same  
> day.

BTW I think the openmpi spec file does things a bit wrong:

./configure --prefix=%{_libdir}/%{mpidir} --with-libnuma=/usr \
	--with-openib=/usr --enable-mpirun-prefix-by-default \
	--mandir=%{_libdir}/%{mpidir}/man --enable-mpi-threads \
	--with-ft=cr --with-valgrind \
	--with-wrapper-cflags="%{?opt_cflags} %{?modeflag}" \
	--with-wrapper-cxxflags="%{?opt_cxxflags} %{?modeflag}" \
	--with-wrapper-fflags="%{?opt_fflags} %{?modeflag}" \
	--with-wrapper-fcflags="%{?opt_fcflags} %{?modeflag}" \
	CC=%{opt_cc} CXX=%{opt_cxx} \
	LDFLAGS='-Wl,-z,noexecstack' \
	CFLAGS="%{?opt_cflags} $RPM_OPT_FLAGS $XFLAGS" \
	CXXFLAGS="%{?opt_cxxflags} $RPM_OPT_FLAGS $XFLAGS" \
	FC=%{opt_fc} FCFLAGS="%{?opt_fcflags} $RPM_OPT_FLAGS $XFLAGS" \
	F77=%{opt_f77} FFLAGS="%{?opt_fflags} $RPM_OPT_FLAGS $XFLAGS"

IMHO if opt_*flags are defined then $RPM_OPT_FLAGS should not be used,
since the compiler might support the flags in $RPM_OPT_FLAGS. Also, you
should use %{optflags} instead of $RPM_OPT_FLAGS as you're using
%{buildroot} instead of $RPM_BUILD_ROOT.

The wrapper flags shouldn't use opt_*flags since they're not using
%{optflags} either - the wrapper flags end up in the compiler call
within mpi{cc,cxx,f77,f90}...
Jussi Lehtola
Fedora Project Contributor
jussilehtola at fedoraproject.org

More information about the Fedora-packaging mailing list