[lvm-devel] lvm and locales memory issue

[Zdenek Kabelac]

On 19.2.2010 17:30, Alasdair G Kergon wrote:
> On Fri, Feb 19, 2010 at 05:11:09PM +0100, Zdenek Kabelac wrote:
>> Jakub suggest solution to use  a tiny small forked() process with very limited
>> funcionality to handle mlockall() task  without any usage of locales.
> 
> Nack.  It's not tiny - it's most of the code - and it's a *state* of usage of
> the code - a way of using the *same* code as we also use without mlockall at
> other times - the code knows to behave slightly differently depending whether
> or not it is in this state.
> 
> Surely most people don't need most of the locales - they should be able to
> choose which ones to cache in the archive file - the ones they regularly use -
> and take the performance hit on the occasions (if any) they want to use the
> non-cached ones.
> 
> I notice the code to build the archive is in a 'fedora' subdir in the spec file
> with hard-coded pathnames.  Is there an 'upstream' approach here or do different
> distributions handle this differently and if so why?
> 
>> Eventually error reports, debugs and other things should be handled in
>> surrounding environment - just like we have discussed already as a one
>> potential solution.
>  
> Opposite way around.  The hack would be to push all the things that attempt to
> access that locale archive file out into a subprocess.  That would first require
> an audit of all the the glibc functions we use to determine which ones can attempt
> to access that file.  Then we'd have to place wrappers around those functions 
> to push them into a subprocess and avoid using them synchronously when in this
> 'mlocked' 

state.
> 

Here is outcome of another chat with Jakub:

It looks like there is no easy way to modify API of glibc anytime soon in the
near future and this change would have to go through Uli first...
(there seems to be things like 'newlocale()' which make things more complicated)

As the workaround for 'memory-limited' environments here is suggested  workadound.

remove cache file:
'rm -f /usr/lib/locale/locale-archive'

and create just locales you need to use:
'localedef -f UTF-8 -i cs_CZ /usr/lib/locale/cs_CZ.utf8'
(In this case small separate files in the directory:
"/usr/lib/locale/cs_CZ.utf8"  are create and opened during application runtime
- this is probably less effiecient then this second way:

or eventually recreate /usr/lib/locale/locale-archive with this command:
'localedef -i cs_CZ -c -f UTF-8 -A /usr/share/locale/locale.alias cs_CZ.UTF-8'
(In this case only .5MB file is generated - for adding another locales - just
another call with localedef is needed - i.e. en_US, de_DE, cs_CZ will lead to
~3MB files at my current installation.

Memory footprint for just cs_CZ test case is - i.e. main() { setlocale();
mloockall{} } is approximately 4MB - compared to 100MB with normal - full
locale-archive.

Zdenek