<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word"> <div class="WordSection1"> Created a PR for this patch. <a href="https://github.com/tianocore/edk2/pull/1648"> https://github.com/tianocore/edk2/pull/1648</a><o:p></o:p> <o:p> </o:p> Thanks,<o:p></o:p> Bob<o:p></o:p> <o:p> </o:p> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Bob Feng Sent: Friday, May 14, 2021 3:58 PM To: Chen, Christine <yuwei.chen@intel.com>; devel@edk2.groups.io Subject: Re: [edk2-devel] [PATCH 1/1] BaseTools/Brotli: Fix compressed data loss issue<o:p></o:p> </div> <o:p> </o:p> Reviewed-by: Bob Feng <<a href="mailto:bob.c.feng@intel.com">bob.c.feng@intel.com</a>> On Fri, May 14, 2021 at 02:04 PM, Yuwei Chen wrote:<o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> REF: <a href="https://bugzilla.tianocore.org/show_bug.cgi?id=2946" target="_blank"> https://bugzilla.tianocore.org/show_bug.cgi?id=2946</a> Currenly, when using the Brotli tool to compress data, the output compressed binary file does not record complete compressed data when size of input file is too large, which makes the data loss and will trigger decompress-check issue. The Brotli document mentioned: The brotli tool use BrotliEncoderCompressStream method to compresses input stream to output stream. Under some circumstances (e.g. lack of output stream capacity) the BrotliEncoderOperation would require several calls to BrotliEncoderCompressStream. The method must be called again until both input stream is depleted and encoder has no more output after the method is called. This patch fixes this issue based on the Brotli document. Cc: Bob Feng <<a href="mailto:bob.c.feng@intel.com">bob.c.feng@intel.com</a>> Cc: Liming Gao <<a href="mailto:gaoliming@byosoft.com.cn">gaoliming@byosoft.com.cn</a>> Signed-off-by: Yuwei Chen <<a href="mailto:yuwei.chen@intel.com">yuwei.chen@intel.com</a>> --- .../Source/C/BrotliCompress/BrotliCompress.c | 61 ++++++++++++------- 1 file changed, 38 insertions(+), 23 deletions(-) diff --git a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c index 5a1400fda310..62a6aed3dbd0 100644 --- a/BaseTools/Source/C/BrotliCompress/BrotliCompress.c +++ b/BaseTools/Source/C/BrotliCompress/BrotliCompress.c @@ -156,6 +156,7 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_ uint8_t *NextOut; uint8_t *Input; uint8_t *Output; + size_t TotalOut; size_t OutSize; uint32_t SizeHint; BROTLI_BOOL IsOk; @@ -214,39 +215,53 @@ int CompressFile(char *InputFile, uint8_t *InputBuffer, char *OutputFile, uint8_ IsEof = !HasMoreInput(InputFileHandle); } - if (!BrotliEncoderCompressStream(EncodeState, - IsEof ? BROTLI_OPERATION_FINISH : BROTLI_OPERATION_PROCESS, - &AvailableIn, &NextIn, &AvailableOut, &NextOut, NULL)) { - printf("Failed to compress data [%s]\n", InputFile); - IsOk = BROTLI_FALSE; - goto Finish; - } - if (AvailableOut == 0) { - OutSize = (size_t)(NextOut - Output); - if (OutSize > 0) { - fwrite(Output, 1, OutSize, OutputFileHandle); - if (ferror(OutputFileHandle)) { - printf("Failed to write output [%s]\n", OutputFile); + if (!IsEof){ + do{ + if (!BrotliEncoderCompressStream(EncodeState, + BROTLI_OPERATION_FLUSH, + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) { + printf("Failed to compress data [%s]\n", InputFile); IsOk = BROTLI_FALSE; goto Finish; } + OutSize = (size_t)(NextOut - Output); + if (OutSize > 0) { + fwrite(Output, 1, OutSize, OutputFileHandle); + if (ferror(OutputFileHandle)) { + printf("Failed to write output [%s]\n", OutputFile); + IsOk = BROTLI_FALSE; + goto Finish; + } + } + NextOut = Output; + AvailableOut = kFileBufferSize; } - AvailableOut = kFileBufferSize; - NextOut = Output; + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState)); } - if (BrotliEncoderIsFinished(EncodeState)) { - OutSize = (size_t)(NextOut - Output); - if (OutSize > 0) { - fwrite(Output, 1, OutSize, OutputFileHandle); - if (ferror(OutputFileHandle)) { - printf("Failed to write output [%s]\n", OutputFile); + else{ + do{ + if (!BrotliEncoderCompressStream(EncodeState, + BROTLI_OPERATION_FINISH, + &AvailableIn, &NextIn, &AvailableOut, &NextOut, &TotalOut)) { + printf("Failed to compress data [%s]\n", InputFile); IsOk = BROTLI_FALSE; goto Finish; } - AvailableOut = 0; + OutSize = (size_t)(NextOut - Output); + if (OutSize > 0) { + fwrite(Output, 1, OutSize, OutputFileHandle); + if (ferror(OutputFileHandle)) { + printf("Failed to write output [%s]\n", OutputFile); + IsOk = BROTLI_FALSE; + goto Finish; + } + } + NextOut = Output; + AvailableOut = kFileBufferSize; } + while (AvailableIn > 0 || BrotliEncoderHasMoreOutput(EncodeState)); } - if (IsEof) { + if (BrotliEncoderIsFinished(EncodeState)){ break; } } -- 2.27.0.windows.1<o:p></o:p> </blockquote> <div> </o:p> </div> </div> </body> </html> <div width="1" style="color:white;clear:both">_._,_._,_</div> <hr> Groups.io Links: You receive all messages sent to this group. <a target="_blank" href="https://edk2.groups.io/g/devel/message/75162">View/Reply Online (#75162)</a> | | <a target="_blank" href="https://groups.io/mt/82817984/1813853">Mute This Topic</a> | <a href="https://edk2.groups.io/g/devel/post">New Topic</a> <a href="https://edk2.groups.io/g/devel/editsub/1813853">Your Subscription</a> | <a href="mailto:devel+owner@edk2.groups.io">Contact Group Owner</a> | <a href="https://edk2.groups.io/g/devel/unsub">Unsubscribe</a> [edk2-devel-archive@redhat.com] <div width="1" style="color:white;clear:both">_._,_._,_</div>