[edk2-devel] Multithreaded compression with LZMA2
Daniel Schaefer
daniel.schaefer at hpe.com
Fri Dec 4 08:19:41 UTC 2020
On 12/3/20 11:57 PM, Bret Barkelew via groups.io wrote:
> Wasn’t there another push (somewhere in the last 8 months, my brain is foggy) to adopt LZMA2? Or was it a different algorithm?
I couldn't find anything: https://edk2.groups.io/g/devel/search?q=LZMA2
>
> - Bret
>
> *From: *Daniel Schaefer via groups.io <mailto:daniel.schaefer=hpe.com at groups.io>
> *Sent: *Thursday, December 3, 2020 4:12 AM
> *To: *Laszlo Ersek <mailto:lersek at redhat.com>; devel at edk2.groups.io <mailto:devel at edk2.groups.io>
> *Cc: *Lin, Derek (HPS SW) <mailto:derek.lin2 at hpe.com>
> *Subject: *[EXTERNAL] Re: [edk2-devel] Multithreaded compression with LZMA2
>
> *From:* Laszlo Ersek <lersek at redhat.com>
>
> *Sent:*Thursday, December 3, 2020 18:24
> *To:* devel at edk2.groups.io <devel at edk2.groups.io>; Schaefer, Daniel <daniel.schaefer at hpe.com>
> *Cc:* Lin, Derek (HPS SW) <derek.lin2 at hpe.com>
> *Subject:* Re: [edk2-devel] Multithreaded compression with LZMA2
>
> On 12/02/20 03:59, Daniel Schaefer wrote:
> > Hi everyone,
> >
> > I'm looking into how to speed up the build process and noticed that our
> > build
> > uses LZMA to encrypt the main firmware volume. Since it's quite big it
> > takes a
> > while but only uses one CPU thread.
> >
> > LZMA2 is a version of LZMA which can be multi-threaded and achieve much
> > faster
> > compression times. I did a quick benchmark using the `xz` command-line
> > tool,
> > which uses a modified version of the LZMA SDK that EDK2 uses. The
> > results are:
> >
> > Uncompressed size: 64M
> >
> > | Algo | Comp Time | Decomp Time | Size | Threads |
> > | ----- | --------- | ----------- | ---- | ------- |
> > | LZMA | 19.67s | 0.9s | 9.1M | 1 |
> > | LZMA2 | 20.11s | 1.2s | 9.2M | 1 |
> > | LZMA2 | 8.31s | 1.0s | 9.4M | 4 |
> >
> > Using those commands:
> >
> > time xz --format=lzma testfile
> > time unlzma testfile.lzma
> >
> > time xz --lzma2 testfile
> > time unxz testfile.xz
> >
> > time xz -T4 --lzma2 testfile
> > time unxz testfile.xz
> >
> > This is quite a significant improvement of build time, while
> > decompression time
> > and size only slightly increase. If that's a concern, then LZMA2 could
> > be used
> > for development only.
> >
> > I haven't investigated the details of how to support this in the code
> > but it
> > appears to be a simple change, since the LZMA SDK that we use already
> > supports
> > LZMA2.
> >
> > What do you think?
>
> "xz -T" works by splitting the input into blocks, and it generates a
> multi-block compressed output.
>
> Yes, that's correct.
>
> > I'm unsure if the current LZMA
> decompressor that runs inside the firmware (= guided section extractor)
> copes with multi-block input.
>
> I think you're right that it doesn't. But we can make the guided section extractor use that same algorithm(LZMA2) and assign it a different GUID, right?
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#68334): https://edk2.groups.io/g/devel/message/68334
Mute This Topic: https://groups.io/mt/78651832/1813853
Group Owner: devel+owner at edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [edk2-devel-archive at redhat.com]
-=-=-=-=-=-=-=-=-=-=-=-
More information about the edk2-devel-archive
mailing list