[Fedora-packaging] Draft for the use of environment-modules
Doug Ledford
dledford at redhat.com
Fri Jul 24 14:43:00 UTC 2009
On Jul 24, 2009, at 9:17 AM, Jussi Lehtola wrote:
> Hi all,
>
>
>
> I've done some updates on the draft at
> https://fedoraproject.org/wiki/PackagingDrafts/MPI
> according to Milos' suggestions.
>
> I'm not quite sure if MPI software spec files, too, should be 3rd
> party
> compiler compatible as the MPI compiler spec files.
With properly functioning MPI installations, there are only very minor
differences for 3rd party specs. They could build all mpi versions in
one go from a single src rpm, or they could have very slightly
different src rpms, one for each mpi. If they opt to build all of
them in one go, they could do something like this:
Name: mpipackage
%package openmpi
BuildRequires: openmpi-devel
%package mpich2
BuildRequires: mpich2-devel
%package lam
BuildRequires: lam-devel
%install
. /etc/profile.d/modules.sh
export CC=mpicc
for mpi in openmpi mpich2 lam; do
make clean
module load ${mpi}-%{_arch}
./configure --bindir=$MPI_BIN --libdir=$MPI_LIB --includedir=
$MPI_INCLUDE
--mandir=$MPI_MAN --datadir=$MPI_DATA --sysconfdir=$MPI_ETC
make
make DESTDIR=%{buildroot} install
find %{builroot}$MPI_BIN %{builroot}$MPI_MAN %{buildroot}$MPI_DATA \
%{buildroot}$MPI_ETC %{buildroot}$MPI_INCLUDE ! -type d \
> $mpi.filelist
module unload ${mpi}-%{_arch}
done
%files openmpi -f openmpi.filelist
%files mpich2 -f mpich2.filelist
%files lam -f lam.filelist
Something like this would work for simple packages. It would be up to
the package maintainer to determine if there should be separate base
and -devel versions of each package (some have no need of a -devel
package, they are not libraries). If things needs splitting up, then
modifying the find to create two filelists would work. However, doing
a static filelist would likely be problematic as that would present a
real headache in dealing with things like mpis compiled with a
different compiler.
>
> On Fri, 2009-07-24 at 08:46 -0400, Doug Ledford wrote:
>> On Jul 23, 2009, at 5:56 PM, Milos Jakubicek wrote:
>>> Well done, I think you could take also some inspiration from hints
>>> made by Doug Ledford in BZ#511099:
>>>
>>> - what about making the $MPI_HOME, $MPI_BIN and $MPI_LIB variables a
>>> MUST?
>>
>> I would strongly support this.
>
> OK, but I really don't see what these are used for. AFAIK they're not
> used by the MPI compiler for anything, as everything is done by the
> wrappers.
For installation purposes. It's to ease the burden of packaging up
programs compiled against different MPIs.
>>> - we should make the sample of specfile from the BZ#511099 part of
>>> the draft
>>>
>>> Some other things that come to my mind:
>>>
>>> - the libraries must have the suffix OR they could be installed
>>> under %{_libdir}/%{name}/%{version}-<MPI compiler>/lib (i.e. there
>>> should similar two choices as for the binaries)
>>
>> I would say the libraries MUST be installed under $MPI_LIB and any
>> resulting binaries MUST be installed under $MPI_BIN. This makes is
>> so
>> that loading the MPI environment module automatically gets you all
>> the
>> associated binaries and libraries needed.
>
> That's a valid option, but makes updating the MPI compiler impossible
> since at least currently with openmpi the dirs are versioned:
>
> setenv MPI_BIN /usr/lib/openmpi/1.3.3-gcc/bin
> setenv MPI_LIB /usr/lib/openmpi/1.3.3-gcc/lib
> setenv MPI_HOME /usr/lib/openmpi/1.3.3-gcc
lam is already non-versioned, and as of openmpi-1.3.2, they expect
version upgrades to no longer be binary incompatible, so I intend to
unversion it as well. At a minimum we could always leave the current
version unversioned and just make it so that if someone wants to
install an older version, then they need to put the version back.
> Of course, installing in a non-versioned directory is a valid option,
> this would just need a couple of definitions more in the MPI module
> files. I'd rename the existing MPI_{BIN,LIB,HOME} variables to
> MPICORE_{BIN,LIB,HOME} and define the following variables:
>
> MPI_BIN %{_libdir}/openmpi/gcc/bin
> MPI_LIB %{_libdir}/openmpi/gcc/lib
> MPI_MAN %{_libdir}/openmpi/gcc/man
> MPI_INCLUDE %{_libdir}/openmpi/gcc/include
> MPI_HOME %{_libdir}/openmpi/gcc/
>
> Out of these, include and man are not usually needed.
This is doable, but would mean we also need both the MPICORE_BIN and
MPI_BIN added to the PATH, and MPICORE_LIB and MPI_LIB added to the
LD_LIBRARY_PATH.
>>> - each MPI build of shared libraries should have a separate -devel
>>> subpackage (having them in a single -devel subpackage would need a
>>> require of all MPI builds for it, which is imho not good)
>>
>> This is probably good, and would need to be under $MPI_INCLUDE maybe?
>> It might be worth while to consider simply making it a rule that the
>> following directories always exist:
>
> This is very important and as such is a MUST.
>
>> So, as per our conversation in IRC, here's the directory struction I
>> would recommend. This is brought out by the fact that the current
>> MPI
>> directory structure I'm using seriously abuses %{_libdir} by putting
>> way too much stuff under there. We don't really want to use /opt
>> (even though that's really the perfect place for this) because of the
>> whole issue of wanting this to be able to be mounted as a read only
>> network filesystem and putting it in /opt adds a new mount point (and
>> also adds complexity to drive partitioning issues). So, keeping it
>> under /usr would seem to be a very important goal. However, we have
>> binaries, man pages, libraries, include files, and now we are talking
>> about other packages tossing their stuff under these directories too.
>> That is a rather excessive abuse of %{_libdir}. The FHS doesn't
>> really have an option to cover this. At all. I don't think we can
>> accomplish what needs done while adhering strictly to the FHS. So, I
>> think there are two options:
>
> This is an issue I have been having for a long time with some packages
> that have general names of binaries.
>
>> 1) Create a suitable MPI_HOME that can hold the entire bunch of stuff
>> from a given MPI package (maybe something like /usr/opt, /usr/local/
>> mpi, or /usr/mpi) and under that top level directory install the
>> multiple mpi packages and also allow for the installation of mpi
>> linked libraries and binaries.
>
> /usr/mpi sounds like a good idea. But it needs to be multiarch/
> multilib
> compatible, e.g. 32-bit and 64-bit libraries need to be able to
> coexist.
> (Well, we can always use e.g. /usr/mpi/openmpi/lib
> and /usr/mpi/openmpi/lib64...)
I'm leary of trying to put 64bit and 32bit binaries in the same tree.
I'm not at all certain that a 64bit mpi runtime from all the various
mpis can run a 32bit application. So, given that uncertainty, I would
prefer that the 32bit and 64bit versions be entirely separate,
including separate bindirs and include dirs (in particular, I think
most every single mpi has different include files for 32bit and 64bit
due to fortran module binding differences).
>> 2) Try to split things up into normal FHS locations, but don't use
>> binary name suffixes or library name suffixes in the normal
>> directories, instead use things like /bin/%{name}-%{_arch}%{?
>> opt_cc_suffix}, %{_libdir}/%{name}%{?opt_cc_suffix}, %{_includedir}/%
>> {name}-%{_arch}%{?opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{?
>> opt_cc_suffix}, %{_datadir}/%{name}-%{_arch}%{?opt_cc_suffix}/man.
>> While the above is somewhat clumsy, and also eliminates the
>> possibility of MPI_HOME being useful as we would now need independent
>> definitions of all the various MPI locations, it does have the
>> benefit
>> of being mostly FHS compliant.
>
> OK, this holds for MPI compilers, and the directories can be reused
> for
> other software, but may be a bit messy..
It is messy, I don't argue that one bit ;-) But I suspect we are less
likely to get yelled at for this than, say, creating /usr/mpi (just
think back to how much people complained about /usr/java, and they did
away with /usr/X11 entirely, I just expect they would see /usr/mpi as
a step in the wrong direction).
>> Regardless of which way people would prefer to go, I do think we
>> should go ahead and standardize all the config files in %
>> {_sysconfdir}/
>> %{name}-%{_arch}%{?opt_cc_suffix} so that they won't possibly be on a
>> read only filesystem.
>
> So far I haven't seen any MPI software with a config file in /etc.
By default, openmpi and lam both have some files they would put in /
etc/ if I didn't have their --prefix set to MPI_HOME. And I got
yelled at on f-d for having the etc files in MPI_HOME/etc since
someone might want to make local modifications while MPI_HOME might be
on a read only network fs mount.
>> Thoughts? If we can get this standard more or less agree on, I'll
>> get
>> the F-11 and devel packages of both lam and openmpi updated the same
>> day.
>
> BTW I think the openmpi spec file does things a bit wrong:
>
> ./configure --prefix=%{_libdir}/%{mpidir} --with-libnuma=/usr \
> --with-openib=/usr --enable-mpirun-prefix-by-default \
> --mandir=%{_libdir}/%{mpidir}/man --enable-mpi-threads \
> --with-ft=cr --with-valgrind \
> --with-wrapper-cflags="%{?opt_cflags} %{?modeflag}" \
> --with-wrapper-cxxflags="%{?opt_cxxflags} %{?modeflag}" \
> --with-wrapper-fflags="%{?opt_fflags} %{?modeflag}" \
> --with-wrapper-fcflags="%{?opt_fcflags} %{?modeflag}" \
> CC=%{opt_cc} CXX=%{opt_cxx} \
> LDFLAGS='-Wl,-z,noexecstack' \
> CFLAGS="%{?opt_cflags} $RPM_OPT_FLAGS $XFLAGS" \
> CXXFLAGS="%{?opt_cxxflags} $RPM_OPT_FLAGS $XFLAGS" \
> FC=%{opt_fc} FCFLAGS="%{?opt_fcflags} $RPM_OPT_FLAGS $XFLAGS" \
> F77=%{opt_f77} FFLAGS="%{?opt_fflags} $RPM_OPT_FLAGS $XFLAGS"
>
> IMHO if opt_*flags are defined then $RPM_OPT_FLAGS should not be used,
> since the compiler might support the flags in $RPM_OPT_FLAGS.
There might be some options that aren't supported by other compilers,
but there are a lot of general options in optflags that we would like
to preserve. For alternate compilers, I think it's reasonable to have
people use sed to remove unwanted options.
> Also, you
> should use %{optflags} instead of $RPM_OPT_FLAGS as you're using
> %{buildroot} instead of $RPM_BUILD_ROOT.
But then they can't use sed to filter out the options they don't
want ;-) In order to sed filter the options, they have to be a shell
variable not an rpm macro.
> The wrapper flags shouldn't use opt_*flags since they're not using
> %{optflags} either - the wrapper flags end up in the compiler call
> within mpi{cc,cxx,f77,f90}...
This was intentional. The opt_*flags variables are specifically
intended for compiler specific options. We have all the right options
since we use gcc, but for icc they likely might want different
optimization flags, etc. The opt_*flags are intended to be the place
for that. And since the optional compiler wrappers will be calling
the optional compiler, they should be passed on down to the mpi*
wrappers. While it's a bit of work to get right, the idea was that
people would filter out options from RPM_OPT_FLAGS that flat don't
work with their compiler, they would set only just the necessary
compiler specific options in opt_*flags, they would set the correct
modeflag for their compiler (I kept it separate to make sure it was
always set, using a non-primary arch mpicc fails otherwise), and the
necessary items in opt_*flags and modeflag would go on to the eventual
mpicc invocation, while our build opts in RPM_OPT_FLAGS would only be
used on our build.
--
Doug Ledford <dledford at redhat.com>
GPG KeyID: CFBFF194
http://people.redhat.com/dledford
InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 203 bytes
Desc: This is a digitally signed message part
URL: <http://listman.redhat.com/archives/fedora-packaging/attachments/20090724/6bec1e2d/attachment.sig>
More information about the Fedora-packaging
mailing list