[Thincrust-devel] issues with zip packaging for appliances

[Perry Myers]

Daniel P. Berrange wrote:
> On Thu, Nov 06, 2008 at 01:08:29AM -0500, Perry Myers wrote:
>> Perry Myers wrote:
>>> I've been playing with the -p zip option to appliance-creator and ran 
>>> into an issue that I'm not sure how we can solve...
>>>
>>> If the output format of a disk image is raw there are two sizes, the 
>>> logical size and the actual size.  So if I create a 20GB disk partition 
>>> in the kickstart, but only use 1.5 GB the file appears to be 20GB but 
>>> actually only takes 1.5GB on disk.
>>>
>>> When appliance-creator attempts to add this disk image to the zip 
>>> archive it fails because the logical size of the file is > 4GB which is 
>>> the limitation of an individual file in a zip archive.  You can pass the 
>>> allowZip64=True parameter to Zip in python to allow the zip64 extensions 
>>> to remove this limitation, but then the zip file can no longer be 
>>> extracted using the standard unzip command line tools (also not sure if 
>>> native Windows zip can handle this)
>>>
>>> It will probably be fairly common that appliance images will have 
>>> logical sizes of > 4GB, so zip files are really only useful for 
>>> packaging appliances if the image is qcow compressed instead of raw.  If 
>>> the images are qcow, then the logical=actual and will generally be < 4GB.
>>>
>>> We can consider using another archiving format like tar, but then we 
>>> can't  extract appliances on Windows with the default tools there.  We 
>>> can force qcow compression for zip packaging but that limits our 
>>> hypervisor usage.
>> Other random interesting datapoints...
>>
>> for the oVirt Appliance, if we have the output format as raw the image 
>> size is 1.5GB since it is sparse
>>
>> with the output format as qcow2 the image only drops to 1.2GB
>>
>> if I take that qcow2 image and put it in a zip file the resulting zip file 
>> is around 400MB
>>
>> if I zip the raw 1.5GB image it's also around 400MB (maybe a little bit 
>> more but not much)  This only worked with the Zip64 extensions turned on 
>> though
>>
>> So for distribution of appliances qcow2 does not really get you any 
>> savings.  You'd still have to use a compressed package format.  We've been 
>> doing that up until now by putting the image into an RPM which compresses 
>> it.
>>
>> So the fundamental question here is... what packaging format can we use 
>> that will have good compression, is natively supported on a wide variety 
>> of platforms (Unix, Windows, Linux) and will hold raw images that are 
>> fairly large.
> 
> Well for Linux world ZIP, tar.gz, tar.bz2 are commonly installed. You
> won't find things like 7-zip in many commercial distros - eg RHEL.
> 
> For Windows world, ZIP is obviously standard natively done by Explorer,
> but WinZIP supports tar.gz too. 
> 
> Not sure what coverage of zip64 is like in Windows ?

Perhaps what we need to do is just use tar.gz file and require that 
Windows users grab 7-zip or WinZIP in order to extract.

Perry