[Fedora-packaging] Draft for the use of environment-modules

Fri Jul 24 13:17:52 UTC 2009

Hi all,

I've done some updates on the draft at
https://fedoraproject.org/wiki/PackagingDrafts/MPI
according to Milos' suggestions.

I'm not quite sure if MPI software spec files, too, should be 3rd party
compiler compatible as the MPI compiler spec files. 

On Fri, 2009-07-24 at 08:46 -0400, Doug Ledford wrote:
> On Jul 23, 2009, at 5:56 PM, Milos Jakubicek wrote:
> > Well done, I think you could take also some inspiration from hints  
> > made by Doug Ledford in BZ#511099:
> >
> > - what about making the $MPI_HOME, $MPI_BIN and $MPI_LIB variables a  
> > MUST?
> 
> I would strongly support this.

OK, but I really don't see what these are used for. AFAIK they're not
used by the MPI compiler for anything, as everything is done by the
wrappers.

> > - we should make the sample of specfile from the BZ#511099 part of  
> > the draft
> >
> > Some other things that come to my mind:
> >
> > - the libraries must have the suffix OR they could be installed  
> > under %{_libdir}/%{name}/%{version}-<MPI compiler>/lib (i.e. there  
> > should similar two choices as for the binaries)
> 
> I would say the libraries MUST be installed under $MPI_LIB and any  
> resulting binaries MUST be installed under $MPI_BIN.  This makes is so  
> that loading the MPI environment module automatically gets you all the  
> associated binaries and libraries needed.

That's a valid option, but makes updating the MPI compiler impossible
since at least currently with openmpi the dirs are versioned:

setenv                  MPI_BIN         /usr/lib/openmpi/1.3.3-gcc/bin
setenv                  MPI_LIB         /usr/lib/openmpi/1.3.3-gcc/lib
setenv                  MPI_HOME        /usr/lib/openmpi/1.3.3-gcc

Of course, installing in a non-versioned directory is a valid option,
this would just need a couple of definitions more in the MPI module
files. I'd rename the existing MPI_{BIN,LIB,HOME} variables to
MPICORE_{BIN,LIB,HOME} and define the following variables:

MPI_BIN		%{_libdir}/openmpi/gcc/bin
MPI_LIB		%{_libdir}/openmpi/gcc/lib
MPI_MAN		%{_libdir}/openmpi/gcc/man
MPI_INCLUDE	%{_libdir}/openmpi/gcc/include
MPI_HOME	%{_libdir}/openmpi/gcc/

Out of these, include and man are not usually needed.

> > - each MPI build of shared libraries should have a separate -devel  
> > subpackage (having them in a single -devel subpackage would need a  
> > require of all MPI builds for it, which is imho not good)
> 
> This is probably good, and would need to be under $MPI_INCLUDE maybe?   
> It might be worth while to consider simply making it a rule that the  
> following directories always exist:

This is very important and as such is a MUST.

> So, as per our conversation in IRC, here's the directory struction I  
> would recommend.  This is brought out by the fact that the current MPI  
> directory structure I'm using seriously abuses %{_libdir} by putting  
> way too much stuff under there.  We don't really want to use /opt  
> (even though that's really the perfect place for this) because of the  
> whole issue of wanting this to be able to be mounted as a read only  
> network filesystem and putting it in /opt adds a new mount point (and  
> also adds complexity to drive partitioning issues).  So, keeping it  
> under /usr would seem to be a very important goal.  However, we have  
> binaries, man pages, libraries, include files, and now we are talking  
> about other packages tossing their stuff under these directories too.   
> That is a rather excessive abuse of %{_libdir}.  The FHS doesn't  
> really have an option to cover this.  At all.  I don't think we can  
> accomplish what needs done while adhering strictly to the FHS.  So, I  
> think there are two options:

This is an issue I have been having for a long time with some packages
that have general names of binaries.

> 1) Create a suitable MPI_HOME that can hold the entire bunch of stuff  
> from a given MPI package (maybe something like /usr/opt, /usr/local/ 
> mpi, or /usr/mpi) and under that top level directory install the  
> multiple mpi packages and also allow for the installation of mpi  
> linked libraries and binaries.

/usr/mpi sounds like a good idea. But it needs to be multiarch/multilib
compatible, e.g. 32-bit and 64-bit libraries need to be able to coexist.
(Well, we can always use e.g. /usr/mpi/openmpi/lib
and /usr/mpi/openmpi/lib64...)

> 2) Try to split things up into normal FHS locations, but don't use  
> binary name suffixes or library name suffixes in the normal  
> directories, instead use things like /bin/%{name}-%{_arch}%{? 
> opt_cc_suffix}, %{_libdir}/%{name}%{?opt_cc_suffix}, %{_includedir}/% 
> {name}-%{_arch}%{?opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{? 
> opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{?opt_cc_suffix}/man.   
> While the above is somewhat clumsy, and also eliminates the  
> possibility of MPI_HOME being useful as we would now need independent  
> definitions of all the various MPI locations, it does have the benefit  
> of being mostly FHS compliant.

OK, this holds for MPI compilers, and the directories can be reused for
other software, but may be a bit messy..

> Regardless of which way people would prefer to go, I do think we  
> should go ahead and standardize all the config files in %{_sysconfdir}/ 
> %{name}-%{_arch}%{?opt_cc_suffix} so that they won't possibly be on a  
> read only filesystem.

So far I haven't seen any MPI software with a config file in /etc. 

> Thoughts?  If we can get this standard more or less agree on, I'll get  
> the F-11 and devel packages of both lam and openmpi updated the same  
> day.

BTW I think the openmpi spec file does things a bit wrong:

./configure --prefix=%{_libdir}/%{mpidir} --with-libnuma=/usr \
	--with-openib=/usr --enable-mpirun-prefix-by-default \
	--mandir=%{_libdir}/%{mpidir}/man --enable-mpi-threads \
	--with-ft=cr --with-valgrind \
	--with-wrapper-cflags="%{?opt_cflags} %{?modeflag}" \
	--with-wrapper-cxxflags="%{?opt_cxxflags} %{?modeflag}" \
	--with-wrapper-fflags="%{?opt_fflags} %{?modeflag}" \
	--with-wrapper-fcflags="%{?opt_fcflags} %{?modeflag}" \
	CC=%{opt_cc} CXX=%{opt_cxx} \
	LDFLAGS='-Wl,-z,noexecstack' \
	CFLAGS="%{?opt_cflags} $RPM_OPT_FLAGS $XFLAGS" \
	CXXFLAGS="%{?opt_cxxflags} $RPM_OPT_FLAGS $XFLAGS" \
	FC=%{opt_fc} FCFLAGS="%{?opt_fcflags} $RPM_OPT_FLAGS $XFLAGS" \
	F77=%{opt_f77} FFLAGS="%{?opt_fflags} $RPM_OPT_FLAGS $XFLAGS"

IMHO if opt_*flags are defined then $RPM_OPT_FLAGS should not be used,
since the compiler might support the flags in $RPM_OPT_FLAGS. Also, you
should use %{optflags} instead of $RPM_OPT_FLAGS as you're using
%{buildroot} instead of $RPM_BUILD_ROOT.

The wrapper flags shouldn't use opt_*flags since they're not using
%{optflags} either - the wrapper flags end up in the compiler call
within mpi{cc,cxx,f77,f90}...
-- 
Jussi Lehtola
Fedora Project Contributor
jussilehtola at fedoraproject.org