emCompress-Flex User Guide & Reference Manual
LZMA compression system.
Introduction to emCompress-Flex
This section presents an overview of emCompress-Flex, its structure,
and its capabilities.
What is emCompress-Flex?
emCompress-Flex is a compression system that is able to reduce the storage
requirements of data that must be embedded into an application. Typical
uses of emCompress-Flex are:
- Firmware images that must be dynamically expanded on device reprogramming.
- Configuration bitstreams to program FPGA and CPLD devices.
- Permanent files for embedded web server static content.
Of course, emCompress-Flex is not limited to these applications, it
can be used whenever it’s beneficial to reduce the size of stored
content.
Features
emCompress-Flex is written in standard ANSI C and can run on virtually
any CPU. Here’s a list summarizing the main features of emCompress-Flex:
- Clean ISO/ANSI C source code.
- Small decompressor ROM footprint.
- Completely tunable decompressor RAM footprint.
- Easy-to-understand and simple-to-use API.
- Simple configuration.
- Royalty free.
Recommended project structure
We recommend keeping emCompress-Flex separate from your application files.
It is good practice to keep all the program files (including the
header files) together in the COMPRESS subdirectory of your project’s
root directory. This practice has the advantage of being very easy to
update to newer versions of emCompress-Flex by simply replacing the COMPRESS
and SEGGER directories. Your application files can be stored
anywhere.
Note
When updating to a newer emCompress-Flex version: as files may have been added, moved or deleted, the project directories may need to be updated accordingly.
Package content
emCompress-Flex is provided in source code and contains everything needed.
The following table shows the content of the emCompress Package:
| Files | Description |
|---|
| Application | Supporting example application sources. |
| Config | Configuration header files. |
| Doc | emCompress-Flex documentation. |
| COMPRESS | emCompress-Flex decompressor and optional compressor source code. |
| Windows/COMPRESS | Compressor DLL and supporting applications in binary form. |
Compressor include directories
You should make sure that the include path contains the following directories
(the order of inclusion is of no importance):
Note
Always make sure that you have only one version of each file!
It is frequently a major problem when updating to a new version of emCompress-Flex
if you have old files included and therefore mix different versions. If you
keep emCompress-Flex in the directories as suggested (and only in these), this
type of problem cannot occur. When updating to a newer version, you should
be able to keep your configuration files and leave them unchanged. For safety
reasons, we recommend backing up (or at least renaming) the COMPRESS
directories before updating.
Compressor
This section describes the emCompress-Flex LZMA compressor interface.
Distribution types
emCompress-Flex is shipped with the decompressor in souce and a DLL and
corresponding library where the compressor is preconfigured with a fixed in
memory footprint.
DLL and library distribution
Configuration of the compressor with the DLL is not possible: it is
preconfigured to allow compression with all configuration parameters
up to their maximum value, and with dictionary (window) sizes up to
one megabyte. The files CFLEX_ENCODE_Conf.h and
COMPRESS_LZMA_ENCODE_Conf.h reflect the build configuration of the
DLL and must not be modified.
Sample applications
emCompress-Flex ships with a number of sample applications that show how
to integrate shell capability into your application. Each sample application
adds an additional capability to the shell and is a small incremental step
from the previous version.
The sample applications are:
| Application | Description |
|---|
| CFLEX_OneShot.c | Single-shot compression. |
| CFLEX_Compress.c | Stream data into and out of the compressor. |
| CFLEX_Optimize.c | Search for optimal compressor parameters. |
A note on the samples
Each sample that we present in this section is written in a style
that makes it easy to describe and that fits comfortably within the
margins of printed paper. Therefore, it may well be that you would
rewrite the sample to have a slightly different structure that fits
better, but please keep in mind that these examples are written with
clarity as the prime objective, and to that end we sacrifice some
brevity and efficiency.
Where to find the sample code
Target-independent code
All target-independent samples are included in the Application
directory of the emCompress-Flex distribution.
The entry point for all target-dependent code is the function MainTask(),
which is usually run by the host operating system in a task context. To
run this code on a PC or workstation, simply call MainTask() from main():
void MainTask(void);
int main(int argc, char **argv) {
MainTask();
return 0;
}
Host-only code
All applications that require a file system and are intended to run
on a PC or workstation are included in the Windows/COMPRESS
directory of the emCompress-Flex distribution.
Single-shot compression
The first example, CFLEX_OneShot.c, provides a simple
way to compress a single block of data without streaming. This is
the smallest working example but is not not how you would typically
compress data in your application.
For a complete listing of this application, see CFLEX_OneShot.c complete listing.
Include files
The emCompress-FLEX encoding API is exposed by including the file CFLEX_ENCODE.h:
#include "CFLEX.h"
#include "CFLEX_ENCODE.h"
Setting up the compressor
Application entry starts to set up compression parameters.
void main(int argc, char **argv) {
COMPRESS_LZMA_STREAM Stream;
COMPRESS_LZMA_PARAS Paras;
int Status;
//
Paras.LC = 0;
Paras.LP = 0;
Paras.PB = 0;
Paras.WindowSize = 1024;
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
//
CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
Step 1: Configure the LZMA probabilty model
The three parameters LC, LP, and PB configure the LZMA probability
model and how the LZMA encoder performs. For better compression,
these parameters are usually nonzero but as they grow more memory
is needed to hold the encoder state. This example uses the minimum
amount of working store possible by setting these parameters to zero.
Step 2: Configure the window size
The window size is the maximum number of octets that the decoder
will need to maintain as context in order to copy a reference to
previously-decoded content. As the window size increases, the
compressed output usually decreases as there are more opportunities
to find matching substrings within the window.
Step 3: Configure match lengths
The match length range defined by LZMA is between 2 and 273 octets.
However, a minimum match of two octets produces compressed images
that are larger than images compressed with a minimum match of three
bytes, so a good default is a minimum match length of three octets.
Step 4: Configure optimization
The optimization level, between 0 and 9, configures the trade-off
between compression efficiency and encoding time. Lower optimization
levels will compress faster but will result in larger compressed images,
whereas higher optimization will compress slower and will result in
smaller compressed images. A default value of 5 is a good
trade-off.
Step 5: Initialize compressor
Once the compression parameters are set up, the compressor context
is initialized for compression using CFLEX_LZMA_ENCODE_Init.
Running the compressor
Once the compressor has been configured, data can be passed through
it for compression. This example compresses some fixed, static data:
static const U8 _aMessage[] = {
"With increasing complexity of today's devices, "
"customers expect firmware updates over the life "
"of a device. It's important to be able to replace "
"both firmware images and FPGA configuration bitstreams "
"in the field. Typically, firmware upgrades and other "
"static content only grow in size over the lifetime of "
"a device: features are added to software, not taken "
"away, it's just what happens. Using quickly-written "
"ad-hoc utilities to convert content directly to compilable "
"code (possibly with some form of simple compression) is "
"usual, but it's towards the end of a project when you "
"realize that the static content just won't fit into your "
"device with the baggage of new firmware features."
};
And proceed to the code that prepares the compressor to accept it:
//
Stream.pIn = &_aMessage[0];
Stream.AvailIn = sizeof(_aMessage);
Stream.pOut = &_aOutput[0];
Stream.AvailOut = sizeof(_aOutput);
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, 1);
//
if (Status < 0) {
printf("Error encoding data\n");
} else if (Status == 0) {
printf("Did not encode completely, output buffer too small\n");
} else {
printf("%d bytes compressed to %u bytes\n",
sizeof(_aMessage),
sizeof(_aOutput) - Stream.AvailOut);
}
}
Step 1: Set up data to compress
The input stream to the compressor is defined by the pIn and
AvailIn members of the COMPRESS_LZMA_STREAM structure.
The pIn member points to the first octet of data to be
compressed and AvailIn defines the number of octets of input
that are available to the compressor.
Step 2: Set up bitstream output buffer
In a similar manner to the input, the output stream is defined
by the pOut and AvailOut members of the COMPRESS_LZMA_STREAM
structure. The pOut member points to the object that
receives the octets output by the compressor and AvailOut
defines the number of octets that buffer can hold.
Step 3: Run the compressor
Once the input and output are set up, the compression is performed
by calling CFLEX_LZMA_ENCODE_Run. This function read data from
the input stream, compresses it, and writes it to the output stream.
We provide the encoding context, the stream object that specifies
the data areas, and a final non-zero “flush” value that indicates
to the compressor that this is all the input data it must deal with.
Step 4: Decode the processing status
When the compressor has worked its way through the compression
algorithm, it returns a status code indicating its processing
status. There are three different states that the compressor
will return:
- Values less than zero indicate a processing error. The
specific return value indicates the type of error detected.
- A zero status indicates that the compressor is still
in its data compression phase and must be called again to
continue compressing.
- Stats values greater than zero indicates that the compressor
has finished processing and all input data is consumed and
all output data provided.
In this case we expect the compressor to immediately finish as
the output buffer that holds the compressed bitstream is much
larger than the text input to the compressor. Whilst this
will not always be the case, the text provided to the compressor
in this case does indeed have redundancy and so compresses
to a smaller size—running the application reveals how
much.
Step 5: Extract the compressed data
When the compressor returns with a nonnegative return code, the
compressed data can be read from the buffer given to the
compressor. On return, the AvailOut member is decreased
by one and the pIn pointer increased by one for each octet
written to the buffer. The number of available octets, therefore,
is equal to the number of octets remaining in the buffer on return
subtracted from the size of the buffer on entry.
The above application generates the following output:
C:> CFLEX_OneShot.exe
687 bytes compressed to 428 bytes
C:> _
CFLEX_OneShot.c complete listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Compress.c
Purpose : Example emCompress-FLEX one-shot compression.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>
/*********************************************************************
*
* Static const data
*
**********************************************************************
*/
static const U8 _aMessage[] = {
"With increasing complexity of today's devices, "
"customers expect firmware updates over the life "
"of a device. It's important to be able to replace "
"both firmware images and FPGA configuration bitstreams "
"in the field. Typically, firmware upgrades and other "
"static content only grow in size over the lifetime of "
"a device: features are added to software, not taken "
"away, it's just what happens. Using quickly-written "
"ad-hoc utilities to convert content directly to compilable "
"code (possibly with some form of simple compression) is "
"usual, but it's towards the end of a project when you "
"realize that the static content just won't fit into your "
"device with the baggage of new firmware features."
};
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8 _aOutput[512];
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_STREAM Stream;
COMPRESS_LZMA_PARAS Paras;
int Status;
//
Paras.LC = 0;
Paras.LP = 0;
Paras.PB = 0;
Paras.WindowSize = 1024;
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
//
CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
//
Stream.pIn = &_aMessage[0];
Stream.AvailIn = sizeof(_aMessage);
Stream.pOut = &_aOutput[0];
Stream.AvailOut = sizeof(_aOutput);
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, 1);
//
if (Status < 0) {
printf("Error encoding data\n");
} else if (Status == 0) {
printf("Did not encode completely, output buffer too small\n");
} else {
printf("%u bytes compressed to %u bytes\n",
(unsigned)sizeof(_aMessage),
(unsigned)(sizeof(_aOutput) - Stream.AvailOut));
}
}
/*************************** End of file ****************************/
Compressing a file
This example, CFLEX_Compress.c, expands on the previous
example to compress a file, block by block, writing the output to a
compressed file with a correct header.
For a complete listing of this application, see
CFLEX_Compress.c complete listing.
Incremental compression
Rather than describe the boilerplate code of processing a command
line, opening files, and setting up the encoder as previously, we
present the code that incrementally compresses a file:
Status = 0;
while (Status == 0) {
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
//
Flush = Stream.AvailIn != sizeof(_aInBuffer);
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = sizeof(_aOutBuffer);
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
if (Status >= 0) {
fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
}
}
}
Step 1: Set up main compress loop
The compressor maintains the current compression status in the variable
Status; when the status is nonzero, compression has completed either
successfully or with an error.
Step 2: Set up input
There are no surprises here, simply set up the block to be passed to
the compressor by reading the source file.
Step 3: Set up flush flag
The compressor needs to be told that there will be no more input and
finish up compression: that happens when we have not completely filled
the input buffer from the source file.
Step 4: Compress until block consumed
The inner loop will continually call the compressor to consume the
input buffer and deliver data to the output buffer. The while
loop ensures that we call the compressor when we have more data or
we need to flush. When the status indicates that compression is
complete, the loop terminates.
Step 5: Set up output buffer
The output buffer can be any size—larger buffers may lead to
faster compression, but the overhead is not very significant.
Step 6: Compress and write output buffer
This is the same as the previous application, only this time
we write the compressed output to the file, and this concludes
the compression loop.
Encapsulating the bitstream
The compression loop generates a compressed bitstream, but
the header which the decompressor requires is missing. The header
cannot be constructed until the entire bitstream is compressed,
so there additional code is required that deals with this.
The header is 13 bytes in size:
U8 aHeader[13];
Before writing the compressed bitstream in the loop above the
file pointer is advanced to reserve space for a header:
fseek(pOutFile, sizeof(aHeader), SEEK_SET);
Once the compressor has competed compression, the header can
be constructed and written to the start of the file:
CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
fseek(pOutFile, 0, SEEK_SET);
fwrite(aHeader, 1, sizeof(aHeader), pOutFile);
This concludes the example.
CFLEX_Compress.c complete listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Compress.c
Purpose : Example emCompress-FLEX one-shot compression.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
/*********************************************************************
*
* Defines, configurable
*
**********************************************************************
*/
#define BUFFER_IN_SIZE (32*1024)
#define BUFFER_OUT_SIZE (32*1024)
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8 _aInBuffer [BUFFER_IN_SIZE];
static U8 _aOutBuffer[BUFFER_OUT_SIZE];
/*********************************************************************
*
* Static code
*
**********************************************************************
*/
/*********************************************************************
*
* _GetStatusText()
*
* Function description
* Decode emCompress-LZMA status code.
*
* Parameters
* Status - Status code.
*
* Return value
* Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
if (Status >= 0) {
return "OK";
}
switch (Status) {
case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
case CFLEX_STATUS_USAGE_ERROR: return "Usage error";
default: return "Unknown error";
}
}
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_STREAM Stream;
COMPRESS_LZMA_PARAS Paras;
FILE* pInFile;
FILE* pOutFile;
U8 aHeader[13];
int Flush;
int Status;
//
if (argc != 3) {
fprintf(stderr, "Usage: %s <input-file> <output-file>\n", argv[0]);
exit(100);
}
//
pInFile = fopen(argv[1], "rb");
if (pInFile == 0) {
printf("%s: can't open %s for reading\n", argv[0], argv[1]);
exit(100);
}
//
pOutFile = fopen(argv[2], "wb");
if (pOutFile == 0) {
printf("%s: can't open %s for writing\n", argv[0], argv[2]);
fclose(pInFile);
exit(100);
}
//
Paras.LC = 0;
Paras.LP = 0;
Paras.PB = 0;
Paras.WindowSize = 1024;
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
//
CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
//
// Reserve space for an LZMA header in the output file.
//
fseek(pOutFile, sizeof(aHeader), SEEK_SET);
//
Status = 0;
while (Status == 0) {
//
// Read next chunk and set up coder buffers.
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
//
// Flush compressor on end of input file.
//
Flush = Stream.AvailIn != sizeof(_aInBuffer);
//
// Encode input data, writing encoded data to file.
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = sizeof(_aOutBuffer);
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
if (Status >= 0) {
fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
}
}
}
//
// Rewind file and write correct header.
//
CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
fseek(pOutFile, 0, SEEK_SET);
fwrite(aHeader, 1, sizeof(aHeader), pOutFile);
//
fclose(pInFile);
fclose(pOutFile);
//
if (Status < 0) {
printf("FATAL: %s\n", _GetStatusText(Status));
exit(100);
}
}
/*************************** End of file ****************************/
Optimizing compression parameters
If you require the best compression possible, you can compress a file
with varying parameters and search for the best. It’s not possible
to find optimal compression parameters, in general, but it is
possible to change the context parameters and search for those that
provide the best compression.
The optimizer runs over the input file multiple times, varying the
probability model’s parameters:
for (Paras.LC = 0; Paras.LC <= 8; ++Paras.LC) {
for (Paras.LP = 0; Paras.LP <= 4; ++Paras.LP) {
for (Paras.PB = 0; Paras.PB <= 4; ++Paras.PB) {
The compressor is run inside the innermost loop, using the selected
parameters, and the result printed. The output from the application
shows how the search progresses:
C:> CFLEX_Optimize.exe K66_SEGGER_emPower.bin
LC=0, LP=0, PB=0: 150842
LC=0, LP=0, PB=1: 149306
LC=0, LP=1, PB=0: 147939
LC=0, LP=1, PB=1: 146402
LC=1, LP=1, PB=1: 145641
LC=2, LP=1, PB=1: 145139
LC=3, LP=1, PB=1: 144967
C:> _
The application itself is very straightforward and is not explained further.
CFLEX_Optimize.c complete listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Optimize.c
Purpose : Find optimal compression parameters for a file.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*********************************************************************
*
* Defines, configurable
*
**********************************************************************
*/
#define BUFFER_IN_SIZE (32*1024)
#define BUFFER_OUT_SIZE (32*1024)
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8 _aInBuffer [BUFFER_IN_SIZE];
static U8 _aOutBuffer[BUFFER_OUT_SIZE];
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_STREAM Stream;
COMPRESS_LZMA_PARAS Paras;
const char * sInputPathname;
FILE * pInFile;
U64 MinSize;
U64 EncodedSize;
int Flush;
int Status;
int i;
unsigned MaxLP;
unsigned MaxLC;
unsigned MaxPB;
//
Paras.WindowSize = COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX;
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
MaxLC = COMPRESS_LZMA_CONFIG_ENCODE_LC_MAX;
MaxLP = COMPRESS_LZMA_CONFIG_ENCODE_LP_MAX;
MaxPB = COMPRESS_LZMA_CONFIG_ENCODE_PB_MAX;
//
sInputPathname = 0;
//
for (i = 1; i < argc; ++i) {
const char *sArg = argv[i];
if (strncmp(sArg, "-lc=", 4) == 0) {
MaxLC = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLC);
} else if (strncmp(sArg, "-lp=", 4) == 0) {
MaxLP = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLP);
} else if (strncmp(sArg, "-pb=", 4) == 0) {
MaxPB = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxPB);
} else if (strncmp(sArg, "-ws=", 4) == 0) {
Paras.WindowSize = atoi(&sArg[4]);
} else if (strncmp(sArg, "-O", 2) == 0) {
Paras.Optimize = atoi(&sArg[2]);
} else if (strncmp(sArg, "-k", 2) == 0) {
Paras.LC = 3;
Paras.LP = 0;
Paras.PB = 2;
} else if (strncmp(sArg, "-minlen=", 8) == 0) {
Paras.MinLen = atoi(&sArg[8]);
} else if (strncmp(sArg, "-maxlen=", 8) == 0) {
Paras.MaxLen = atoi(&sArg[8]);
} else if (sArg[0] == '-') {
printf("%s: unknown option %s\n", argv[0], sArg);
exit(100);
} else if (sInputPathname == 0) {
sInputPathname = sArg;
} else {
printf("%s: too many filenames\n", argv[0]);
exit(100);
}
}
//
if (sInputPathname == 0) {
printf("%s: require input filename\n", argv[0]);
exit(100);
}
//
pInFile = fopen(sInputPathname, "rb");
if (pInFile == 0) {
printf("%s: can't open %s for reading\n", argv[0], argv[1]);
exit(100);
}
MinSize = ~0ULL;
//
for (Paras.LC = 0; Paras.LC <= MaxLC; ++Paras.LC) {
for (Paras.LP = 0; Paras.LP <= MaxLP; ++Paras.LP) {
for (Paras.PB = 0; Paras.PB <= MaxPB; ++Paras.PB) {
//
Status = CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
if (Status < 0) {
continue;
}
//
fseek(pInFile, 0, SEEK_SET);
//
EncodedSize = 0;
Status = 0;
while (Status == 0) {
//
// Read next chunk and set up coder buffers.
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
//
// Flush compressor on end of input file.
//
Flush = Stream.AvailIn != sizeof(_aInBuffer);
//
// Encode input data.
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = sizeof(_aOutBuffer);
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
EncodedSize += sizeof(_aOutBuffer) - Stream.AvailOut;
}
}
//
printf("\rLC=%d, LP=%d, PB=%d: %llu \r",
Paras.LC, Paras.LP, Paras.PB, EncodedSize);
if (EncodedSize < MinSize) {
MinSize = EncodedSize;
printf("\n");
}
}
}
}
//
printf("\r \r");
}
/*************************** End of file ****************************/
Prefiltering
It is possible to improve the effectiveness of compression when the
compressor knows the structure of the data and can use a domain-specific
filter. In particular, for firmware upgrades, preprocessing
the input to the LZMA compressor to provide improve the probability of
matches detected by the LZSS engine is particularly helpful.
Here is a typical application without prefiltering, searching for optimized
compression parameters:
C:> CFLEX_Optimize.exe K66_SEGGER_emPower.bin -ws=4096
LC=0, LP=0, PB=0: 160402
LC=0, LP=0, PB=1: 158931
LC=0, LP=1, PB=0: 155551
LC=0, LP=1, PB=1: 154080
LC=1, LP=1, PB=1: 152754
LC=2, LP=1, PB=1: 151886
LC=3, LP=1, PB=1: 151309
C:> _
With prefiltering using architecture-specific transformations (in this case
for the Thumb-2 instruction set), we further reduce the compressed size of the
firmware:
C:> CFLEX_Filter.exe K66_SEGGER_emPower.bin -ws=4096
LC=0, LP=0, PB=0: 153128
LC=0, LP=0, PB=1: 152131
LC=0, LP=1, PB=0: 149096
LC=0, LP=1, PB=1: 148099
LC=1, LP=1, PB=1: 147282
LC=2, LP=1, PB=0: 146972
LC=2, LP=1, PB=1: 145975
LC=3, LP=1, PB=1: 145622
C:> _
Decompressor
This section describes the emCompress-Flex decompressor interface.
The emCompress-Flex decompressor is delivered as source code
for integration into your target system.
Streaming decompression
The first and only example, CFLEX_Decompress.c, is
an application that will decompress the output file from the previous
compressor.
For a complete listing of this application, see
CFLEX_Decompress.c complete listing.
Decompressor include files
The emCompress-Flex decoding API is exposed by including the file
CFLEX_DECODE.h:
#include "CFLEX.h"
#include "CFLEX_DECODE.h"
Decompressor loop
The decompressor uses the same stream interface object as the
compressor, COMPRESS_LZMA_STREAM. And the decompressor
“run” code uses an identical set of parameters and return
codes which makes it very simple to use.
In fact, the decompressor loop is identical in form to the compressor
loop:
Status = 0;
while (Status == 0) {
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
EncodedLen += Stream.AvailIn;
//
Flush = Stream.AvailIn != _BlockInSize;
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = sizeof(_aOutBuffer);
//
Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
if (Status >= 0) {
fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
}
}
}
CFLEX_Decompress.c complete listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH & Co. KG *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Decompress.c
Purpose : Example emCompress-FLEX one-shot decompression.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
/*********************************************************************
*
* Defines, configurable
*
**********************************************************************
*/
#define BUFFER_IN_SIZE (32*1024)
#define BUFFER_OUT_SIZE (32*1024)
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static U8 _aInBuffer [BUFFER_IN_SIZE];
static U8 _aOutBuffer[BUFFER_OUT_SIZE];
/*********************************************************************
*
* Static code
*
**********************************************************************
*/
/*********************************************************************
*
* _GetStatusText()
*
* Function description
* Decode emCompress-LZMA status code.
*
* Parameters
* Status - Status code.
*
* Return value
* Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
if (Status >= 0) {
return "OK";
}
switch (Status) {
case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
case CFLEX_STATUS_USAGE_ERROR: return "Usage error";
default: return "Unknown error";
}
}
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_STREAM Stream;
FILE* pInFile;
FILE* pOutFile;
int Flush;
int Status;
//
if (argc != 3) {
fprintf(stderr, "Usage: %s <input-file> <output-file>\n", argv[0]);
exit(100);
}
//
pInFile = fopen(argv[1], "rb");
if (pInFile == 0) {
printf("%s: can't open %s for reading\n", argv[0], argv[1]);
exit(100);
}
//
pOutFile = fopen(argv[2], "wb");
if (pOutFile == 0) {
printf("%s: can't open %s for writing\n", argv[0], argv[2]);
fclose(pInFile);
exit(100);
}
//
CFLEX_LZMA_DECODE_Init(&_Decoder);
//
Status = 0;
while (Status == 0) {
//
// Read next chunk and set up coder buffers.
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
//
// Flush compressor on end of input file.
//
Flush = Stream.AvailIn != sizeof(_aInBuffer);
//
// Encode input data, writing encoded data to file.
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = sizeof(_aOutBuffer);
//
Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
if (Status >= 0) {
fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
}
}
}
//
fclose(pInFile);
fclose(pOutFile);
//
if (Status < 0) {
printf("FATAL: %s\n", _GetStatusText(Status));
exit(100);
}
}
/*************************** End of file ****************************/
Configuring the decompressor
The following definitions must be configured for emCompress-Flex’s decompressor.
The user configuration files for this are CFLEX_DECODE_Conf.h and
COMPRESS_LZMA_DECODE_Conf.h.
The configuration options are described in the following sections.
Decompressor data types
Default
There is no default as the required emCompress-Flex data types are
configured by using a type definition rather than a preprocessor symbol.
Recommended configuration
typedef unsigned long long U64;
typedef unsigned long U32;
typedef unsigned short U16;
typedef unsigned char U8;
typedef unsigned int ULEAST16;
typedef int ILEAST16;
Description
Defines the types that the emCompress-Flex decompressor will in its
data structures when compressing.
The types above are typical for a 32-bit machine, but the decompressor
will work with any type definition for each as long as each type can
store the prescribed number of bits without loss. For instance, it may
well be that the compilation system provides a <stdint.h> header
file, in which case the following is acceptable:
typedef uint64_t U64;
typedef uint32_t U32;
typedef uint16_t U16;
typedef uint8_t U8;
typedef int_least16_t ILEAST16;
typedef uint_least16_t ULEAST16;
Maximum accepted literal context bits
Default
#define COMPRESS_LZMA_CONFIG_DECODE_LC_MAX 8
Description
Configures the maximum number of literal context bits that the decompressor
will support. Valid configuration values for this symbol are 0 through 8.
Smaller values will reduce the memory required for the decoder context but
will also limit the capability of the compressor when encoding a bitstream
acceptable to the decompressor.
It is a (diagnosed) error to attempt decoding a bitstream which is
encoded with an LC value greater than the configured maximum.
Maximum accepted literal position context bits
Default
#define COMPRESS_LZMA_CONFIG_DECODE_LP_MAX 4
Description
Configures the maximum number of literal position context bits that the
decompressor will support. Valid configuration values for this symbol are 0
through 4. Smaller values will reduce the memory required for the decoder
context but will also limit the capability of the compressor when encoding
a bitstream acceptable to the decompressor.
It is a (diagnosed) error to attempt decoding a bitstream which is
encoded with an LP value greater than the configured maximum.
Maximum accepted position bits
Default
#define COMPRESS_LZMA_CONFIG_DECODE_PB_MAX 4
Description
Configures the maximum number of position context bits that the decompressor
will support. Valid configuration values for this symbol are 0 through 4.
Smaller values will reduce the memory required for the decoder context but
will also limit the capability of the compressor when encoding a bitstream
acceptable to the decompressor.
It is a (diagnosed) error to attempt decoding a bitstream which is
encoded with an PB value greater than the configured maximum.
Maximum accepted window size
Default
#define COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX (1024*1024)
Description
Configures the maximum window size that the decompressor will support.
Valid configuration values for this symbol are 1 through 0xFFFFFFFF.
Smaller values will reduce the memory required for the decoder context but
will also limit the capability of the compressor when encoding a bitstream
acceptable to the decompressor.
It is a (diagnosed) error to attempt decoding a bitstream which is
encoded with a window size greater than the configured maximum
decoder window size.
Shipped decompressor configuration
This is the shipped configuration file, CFLEX_DECODE_Conf.h.
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_DECODE_Conf.h
Purpose : Configuration settings for emCompress-FLEX decoder.
*/
#ifndef CFLEX_DECODE_CONF_H
#define CFLEX_DECODE_CONF_H
#ifdef __cplusplus
extern "C" {
#endif
#ifdef __cplusplus
}
#endif
#endif
/*************************** End of file ****************************/
This is the shipped configuration file, COMPRESS_LZMA_DECODE_Conf.h.
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : COMPRESS_LZMA_DECODE_Conf.h
Purpose : Configuration settings for emCompress-FLEX LZMA decoder.
*/
#ifndef COMPRESS_LZMA_DECODE_CONF_H
#define COMPRESS_LZMA_DECODE_CONF_H
#ifdef __cplusplus
extern "C" {
#endif
// Maximum number of literal context (LC) bits the decoder will accept [0, 8].
#define COMPRESS_LZMA_CONFIG_DECODE_LC_MAX 8
// Maximum number of literal position (LP) bits the decoder will accept [0, 4].
#define COMPRESS_LZMA_CONFIG_DECODE_LP_MAX 4
// Maximum number of position bits (PB) bits the decoder will accept [0, 4].
#define COMPRESS_LZMA_CONFIG_DECODE_PB_MAX 4
// Maximum size of LZMA window
#define COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX (1024*1024)
#ifdef __cplusplus
}
#endif
#endif
/*************************** End of file ****************************/
Decompressor memory use
The decompressor has no static data requirement, the only memory required
to decompress a stream is provided to the decompressor by the client
in a COMPRESS_LZMA_DECODE_CONTEXT object.
The decoder context size varies according to selected emCompress-Flex LZMA
decoder configuration and the specific compilation options selected
to compile the decompressor.
Because the target processor’s type sizes and alignments are not known
to the generic compressor, and cannot easily be modeled without significant
probability of error, the sizes presented here are for a 32-bit Cortex-M
processor which should generally translate well to many 32-bit architectures
including 32-bit “x86” targets. For nonstandard architectures, such as
pure DSPs which are typically word-addressed machines, the memory requirement
in words can be derived by configuring the compiler and using the
sizeof operator to query the memory size in units of char rather
than byte units.
The context size of the decoder is modeled using the four parameters
LC, LP, PB, and WindowSize. The application
CFLEX_Sizes.c will estimate the memory required for a
decompressor context.
It must be stressed that this computation is an estimate and the exact
memory requirement can be computed with the sizeof operator applied
to the COMPRESS_LZMA_DECODE_CONTEXT. However, the exact size of
the decoding context is generally not required, only a sense of its
magnitude.
The following table displays the decompressor context size for each
combination of LC. LP, and PB. Each size is measured in kilobytes
with a zero window size, for a typical 32-bit byte-addressed machine.
| LP | PB | LC=0 | LC=1 | LC=2 | LC=3 | LC=4 | LC=5 | LC=6 | LC=7 | LC=8 |
|---|
| 0 | 0 | 3.8 | 5.3 | 8.3 | 14.3 | 26.3 | 50.3 | 98.3 | 194.3 | 386.3 |
| 0 | 1 | 4.0 | 5.5 | 8.5 | 14.5 | 26.5 | 50.5 | 98.5 | 194.5 | 386.5 |
| 0 | 2 | 4.3 | 5.8 | 8.8 | 14.8 | 26.8 | 50.8 | 98.8 | 194.8 | 386.8 |
| 0 | 3 | 4.9 | 6.4 | 9.4 | 15.4 | 27.4 | 51.4 | 99.4 | 195.4 | 387.4 |
| 0 | 4 | 6.2 | 7.7 | 10.7 | 16.7 | 28.7 | 52.7 | 100.7 | 196.7 | 388.7 |
| 1 | 0 | 5.3 | 8.3 | 14.3 | 26.3 | 50.3 | 98.3 | 194.3 | 386.3 | 770.3 |
| 1 | 1 | 5.5 | 8.5 | 14.5 | 26.5 | 50.5 | 98.5 | 194.5 | 386.5 | 770.5 |
| 1 | 2 | 5.8 | 8.8 | 14.8 | 26.8 | 50.8 | 98.8 | 194.8 | 386.8 | 770.8 |
| 1 | 3 | 6.4 | 9.4 | 15.4 | 27.4 | 51.4 | 99.4 | 195.4 | 387.4 | 771.4 |
| 1 | 4 | 7.7 | 10.7 | 16.7 | 28.7 | 52.7 | 100.7 | 196.7 | 388.7 | 772.7 |
| 2 | 0 | 8.3 | 14.3 | 26.3 | 50.3 | 98.3 | 194.3 | 386.3 | 770.3 | 1538.3 |
| 2 | 1 | 8.5 | 14.5 | 26.5 | 50.5 | 98.5 | 194.5 | 386.5 | 770.5 | 1538.5 |
| 2 | 2 | 8.8 | 14.8 | 26.8 | 50.8 | 98.8 | 194.8 | 386.8 | 770.8 | 1538.8 |
| 2 | 3 | 9.4 | 15.4 | 27.4 | 51.4 | 99.4 | 195.4 | 387.4 | 771.4 | 1539.4 |
| 2 | 4 | 10.7 | 16.7 | 28.7 | 52.7 | 100.7 | 196.7 | 388.7 | 772.7 | 1540.7 |
| 3 | 0 | 14.3 | 26.3 | 50.3 | 98.3 | 194.3 | 386.3 | 770.3 | 1538.3 | 3074.3 |
| 3 | 1 | 14.5 | 26.5 | 50.5 | 98.5 | 194.5 | 386.5 | 770.5 | 1538.5 | 3074.5 |
| 3 | 2 | 14.8 | 26.8 | 50.8 | 98.8 | 194.8 | 386.8 | 770.8 | 1538.8 | 3074.8 |
| 3 | 3 | 15.4 | 27.4 | 51.4 | 99.4 | 195.4 | 387.4 | 771.4 | 1539.4 | 3075.4 |
| 3 | 4 | 16.7 | 28.7 | 52.7 | 100.7 | 196.7 | 388.7 | 772.7 | 1540.7 | 3076.7 |
| 4 | 0 | 26.3 | 50.3 | 98.3 | 194.3 | 386.3 | 770.3 | 1538.3 | 3074.3 | 6146.3 |
| 4 | 1 | 26.5 | 50.5 | 98.5 | 194.5 | 386.5 | 770.5 | 1538.5 | 3074.5 | 6146.5 |
| 4 | 2 | 26.8 | 50.8 | 98.8 | 194.8 | 386.8 | 770.8 | 1538.8 | 3074.8 | 6146.8 |
| 4 | 3 | 27.4 | 51.4 | 99.4 | 195.4 | 387.4 | 771.4 | 1539.4 | 3075.4 | 6147.4 |
| 4 | 4 | 28.7 | 52.7 | 100.7 | 196.7 | 388.7 | 772.7 | 1540.7 | 3076.7 | 6148.7 |
Compressor-decompressor verification application
The file CFLEX_Verify will compress and decompress multiple
files using all valid compression configurations for LC, LP, and PB.
The command line options are similar to the sample full-feature application
and allow setting the window sizes and match lengths.
CFLEX_Verify.c complete listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Verify.c
Purpose : Run compress-decompress cycle and validate output.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_ENCODE.h"
#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*********************************************************************
*
* Defines, configurable
*
**********************************************************************
*/
#define BUFFER_SIZE (101*1024*1024) // 101 MB
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static U8 _aFileBuf [BUFFER_SIZE];
static U8 _aEncodedBuf[BUFFER_SIZE];
static U8 _aDecodedBuf[BUFFER_SIZE];
/*********************************************************************
*
* Static code
*
**********************************************************************
*/
/*********************************************************************
*
* _GetStatusText()
*
* Function description
* Decode emCompress-LZMA status code.
*
* Parameters
* Status - Status code.
*
* Return value
* Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
switch (Status) {
case 0: return "Still processing";
case 1: return "Complete";
case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
case CFLEX_STATUS_USAGE_ERROR: return "Usage error";
default: return "Unknown error";
}
}
/*********************************************************************
*
* _Verify()
*
* Function description
* Verify a compress-decompress cycle works.
*
* Parameters
* pParas - Pointer to encoder parameters.
* pInFile - Pointer to input file at position 0.
*
* Return value
* 0 - test passed.
* 1 - test failed.
*/
static int _Verify(const COMPRESS_LZMA_PARAS *pParas, FILE *pInFile) {
COMPRESS_LZMA_STREAM EncoderStream;
COMPRESS_LZMA_STREAM DecoderStream;
int EncoderStatus;
int DecoderStatus;
U64 OriginalSize;
U64 EncodedSize;
U64 DecodedSize;
int Res;
//
Res = 0;
printf("LC=%d, LP=%d, PB=%d: ", pParas->LC, pParas->LP, pParas->PB);
//
EncoderStatus = CFLEX_LZMA_ENCODE_Init(&_Encoder, pParas);
if (EncoderStatus < 0) {
printf("Skipped, encoder will not initialize\n");
return 1;
}
CFLEX_LZMA_DECODE_Init(&_Decoder);
//
EncodedSize = 0;
DecodedSize = 0;
EncoderStatus = 0;
DecoderStatus = 0;
//
EncoderStream.pIn = _aFileBuf;
EncoderStream.AvailIn = fread(_aFileBuf, 1, sizeof(_aFileBuf), pInFile);
EncoderStream.pOut = &_aEncodedBuf[13];
EncoderStream.AvailOut = sizeof(_aEncodedBuf) - 13;
OriginalSize = EncoderStream.AvailIn;
//
EncoderStatus = CFLEX_LZMA_ENCODE_Run(&_Encoder, &EncoderStream, 1);
if (EncoderStatus <= 0) {
printf("Encoder error: %s\n", _GetStatusText(EncoderStatus));
return 1;
}
EncodedSize = sizeof(_aEncodedBuf) - 13 - EncoderStream.AvailOut;
CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &_aEncodedBuf[0]);
//
DecoderStream.pIn = &_aEncodedBuf[0];
DecoderStream.AvailIn = sizeof(_aEncodedBuf) - EncoderStream.AvailOut + 13;
DecoderStream.pOut = &_aDecodedBuf[0];
DecoderStream.AvailOut = sizeof(_aDecodedBuf);
//
DecoderStatus = CFLEX_LZMA_DECODE_Run(&_Decoder, &DecoderStream, 1);
if (DecoderStatus <= 0) {
printf("Decoder error: %s\n", _GetStatusText(DecoderStatus));
Res = 1;
}
DecodedSize = sizeof(_aDecodedBuf) - DecoderStream.AvailOut;
//
if (OriginalSize != DecodedSize) {
printf("Compress-decompress size mismatch\n");
Res = 1;
} else if (memcmp(_aFileBuf, _aDecodedBuf, (unsigned)DecodedSize) != 0) {
printf("Compress-decompress content mismatch\n");
Res = 1;
} else {
printf("OK %5.2fx %lld -> %lld -> %lld\n", (float)DecodedSize / EncodedSize, OriginalSize, EncodedSize, DecodedSize);
}
return Res;
}
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_PARAS Paras;
FILE * pInFile;
const char * sArg;
int i;
unsigned MaxLP;
unsigned MaxLC;
unsigned MaxPB;
int Result;
//
Paras.WindowSize = COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX;
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
MaxLC = COMPRESS_LZMA_CONFIG_ENCODE_LC_MAX;
MaxLP = COMPRESS_LZMA_CONFIG_ENCODE_LP_MAX;
MaxPB = COMPRESS_LZMA_CONFIG_ENCODE_PB_MAX;
//
for (i = 1; i < argc; ++i) {
sArg = argv[i];
if (strncmp(sArg, "-lc=", 4) == 0) {
MaxLC = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLC);
} else if (strncmp(sArg, "-lp=", 4) == 0) {
MaxLP = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLP);
} else if (strncmp(sArg, "-pb=", 4) == 0) {
MaxPB = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxPB);
} else if (strncmp(sArg, "-ws=", 4) == 0) {
Paras.WindowSize = atoi(&sArg[4]);
} else if (strncmp(sArg, "-O", 2) == 0) {
Paras.Optimize = atoi(&sArg[2]);
} else if (strncmp(sArg, "-k", 2) == 0) {
Paras.LC = 3;
Paras.LP = 0;
Paras.PB = 2;
} else if (strncmp(sArg, "-minlen=", 8) == 0) {
Paras.MinLen = atoi(&sArg[8]);
} else if (strncmp(sArg, "-maxlen=", 8) == 0) {
Paras.MaxLen = atoi(&sArg[8]);
} else if (sArg[0] == '-') {
printf("%s: unknown option %s\n", argv[0], sArg);
exit(100);
}
}
//
Result = 0;
for (i = 1; i < argc; ++i) {
sArg = argv[i];
if (sArg[0] != '-') {
pInFile = fopen(sArg, "rb");
if (pInFile == 0) {
printf("%s: can't open %s for reading\n", argv[0], argv[1]);
exit(100);
}
//
printf("\nVerification of %s:\n\n", sArg);
//
for (Paras.LC = 0; Paras.LC <= MaxLC; ++Paras.LC) {
for (Paras.LP = 0; Paras.LP <= MaxLP; ++Paras.LP) {
for (Paras.PB = 0; Paras.PB <= MaxPB; ++Paras.PB) {
fseek(pInFile, 0, SEEK_SET);
Result |= _Verify(&Paras, pInFile);
}
}
}
fclose(pInFile);
}
}
if (Result) {
printf("some tests *** FAILED ***\n");
}
}
/*************************** End of file ****************************/
Sample full-feature application
emCompress-Flex ships with an application, in source form, which compresses
files into LZMA-alone format. These files can be fed into the emCompress-Flex
decompressor, byte for byte, and be decompressed on target hardware.
Command line and options
The emCompress-Flex example compression application accepts the command
line options described in the following sections.
The command line syntax is:
CFLEX_Util [options] inputfile [outputfile]
The output file is optional: if given, the compressed LZMA bitstream is
written to it.
The example application displays some information relating to the compression
process and selected parameters.
Example
C:> CFLEX_Util.exe -O9 -ws=1024 ptt5
(c) 2016-2019 SEGGER Microcontroller GmbH www.segger.com
emCompress-FLEX V3.10 compiled Feb 05 2019 12:09:19
Encoding parameters:
Coding: LC=0, LP=0, PB=0
Window size: 1024 bytes
Match length: 3 to 273 bytes
Match distance: 1 to 1024 bytes
Coder performance:
Max. length: 273 bytes
Max. distance: 750 bytes
Statistics:
Decoded size: 513216 bytes
Encoded size: 49403 bytes (including header)
Ratio: 10.39x
Space savings: 90.38% (of original)
Standardized: 0.77 bits/byte
C:> _
Set literal context bits (-lc)
Syntax
-lc=number
Description
Set the number of literal context bits to use when compressing.
Acceptable values are 0 through 8 inclusive.
The default is zero.
Set literal position bits (-lp)
Syntax
-lp=number
Description
Set the number of literal position bits to use when compressing.
Acceptable values are 0 through 4 inclusive.
The default is zero.
Set literal position bits (-pb)
Syntax
-pb=number
Description
Set the number of position bits to use when compressing.
Acceptable values are 0 through 4 inclusive.
The default is zero.
Set window size (-ws)
-ws=number
Description
Set the window size to when compressing. The window size is
the naming convention used by emCompress and emCompress-LZMA
to describe the maximum match distance and, therefore, the
maximum number of octets that must be stored to satisfy
references made by the decompressor. The LZMA SDK refers to this
as the “dictionary size” and both terms should be considered
equivalent.
Increasing the window size will usually increase compression
ratios and reduce the size of the compressed bitstream at the
expense of requiring extra RAM during decompression.
The default is 131,072.
Set minimum match length (-minlen)
-minlen=number
Description
Set the minimum match length when compressing. Acceptable
values are 2 through 273, with the default being 3. We highly
recommend that this value is left unchanged.
Although the LZMA compression scheme supports match lengths of
two, this usually leads to worse compressor performance in terms
of both speed and compressed bitstream size.
Set maximum match length (-maxlen)
Syntax
-maxlen=number
Description
Set the maximum match length when compressing. Acceptable
values are 2 through 273, with the default being 273. We highly
recommend that this value is left unchanged.
The window size and the maximum match length are not independent.
If the window size is reduced, emCompress-LZMA will also reduce
the maximum match length so that it does not exceed the capacity
of the window to store the matched sequence. The emCompress-LZMA
example application does this outside of the emCompress-LZMA DLL
as the DLL will diagnose incorrect compression parameters with an
error.
Syntax
-bi=number
Description
Set the block input size, the maximum amount of data sent to the
compressor in a single call. Acceptable values are 1 through 65536,
with the default being 65536.
This has no effect on the compressed output and is present to enable
the compressor to be tested, if necessary, with a known input and
known block sizes.
Syntax
-bo=number
Description
Set the block output size, the maximum amount of output space available
to the compressor in a single call. Acceptable values are 1 through 65536,
with the default being 65536.
This has no effect on the compressed output and is present to enable
the compressor to be tested, if necessary, with a known input and
known block sizes.
Set optimization level (-O)
Syntax
-Onumber
Description
Set the optimization level to use when compression. Acceptable
values are 0 (faster, good compression) to 9 (slower, best
compression).
Utility listing
/*********************************************************************
* (c) SEGGER Microcontroller GmbH *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------/*********************************************************************
* (c) SEGGER Microcontroller GmbH & Co. KG *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : CFLEX_Util.c
Purpose : Example emCompress-FLEX application.
*/
/*********************************************************************
*
* #include section
*
**********************************************************************
*/
#include "CFLEX_ENCODE.h"
#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/*********************************************************************
*
* Static data
*
**********************************************************************
*/
static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static unsigned _BlockInSize;
static unsigned _BlockOutSize;
static U8 _aInBuffer [64*1024];
static U8 _aOutBuffer[64*1024];
/*********************************************************************
*
* Static code
*
**********************************************************************
*/
/*********************************************************************
*
* _GetStatusText()
*
* Function description
* Decode emCompress-LZMA status code.
*
* Parameters
* Status - Status code.
*
* Return value
* Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
if (Status >= 0) {
return "OK";
}
switch (Status) {
case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
case CFLEX_STATUS_USAGE_ERROR: return "Usage error";
default: return "Unknown error";
}
}
/*********************************************************************
*
* _Encode()
*
* Function description
* Encode a file.
*
* Parameters
* pInFile - Pointer to file to encode.
* pOutFile - Pointer to file that receives the encoded bitstream.
* pParas - Pointer to LZMA encoding parameters.
* Verbose - Flag indicating verbose output.
*/
static void _Encode(FILE *pInFile, FILE *pOutFile, COMPRESS_LZMA_PARAS *pParas, int Verbose) {
COMPRESS_LZMA_STREAM Stream;
int Status;
int Flush;
unsigned EncodedLen;
unsigned DecodedLen;
U8 aHeader[13];
//
// Initialize encoder.
//
CFLEX_LZMA_ENCODE_Init(&_Encoder, pParas);
//
// Reserve space for an LZMA header in the output file.
//
if (pOutFile) {
fseek(pOutFile, 13, SEEK_SET);
}
//
// Encode file.
//
EncodedLen = 0;
DecodedLen = 0;
Status = 0;
while (Status == 0) {
//
// Read next chunk and set up coder buffers.
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, _BlockInSize, pInFile);
DecodedLen += Stream.AvailIn;
//
// Flush compressor on end of input file.
//
Flush = Stream.AvailIn != _BlockInSize;
//
// Encode input data, writing encoded data to file.
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = _BlockOutSize;
//
Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
//
if (Status >= 0 && pOutFile != NULL) {
fwrite(_aOutBuffer, 1, _BlockOutSize - Stream.AvailOut, pOutFile);
}
EncodedLen += _BlockOutSize - Stream.AvailOut;
}
}
//
// Rewind file and write correct header.
//
if (pOutFile) {
CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
fseek(pOutFile, 0, SEEK_SET);
fwrite(aHeader, 1, sizeof(aHeader), pOutFile);
}
//
if (Status < 0) {
fprintf(stderr, "FATAL: %s\n", _GetStatusText(Status));
} else {
if (Verbose) {
fprintf(stderr, "Encoding parameters:\n");
fprintf(stderr, " Coding: LC=%u, LP=%u, PB=%u\n", _Encoder.LC, _Encoder.LP, _Encoder.PB);
fprintf(stderr, " Window size: %lu bytes\n", _Encoder.WindowSize);
fprintf(stderr, " Match length: %d to %d bytes\n", _Encoder.LZSSCoder.MinLength, _Encoder.LZSSCoder.MaxLength);
fprintf(stderr, " Match distance: 1 to %d bytes\n", _Encoder.LZSSCoder.MaxDistance);
fprintf(stderr, "\n");
fprintf(stderr, "Coder performance:\n");
fprintf(stderr, " Max. length: %d bytes\n", _Encoder.LZSSCoder.MaxUsedLength);
fprintf(stderr, " Max. distance: %d bytes\n", _Encoder.LZSSCoder.MaxUsedDistance);
fprintf(stderr, "\n");
fprintf(stderr, "Statistics:\n");
fprintf(stderr, " Decoded size: %u bytes\n", DecodedLen);
fprintf(stderr, " Encoded size: %u bytes (including header)\n", EncodedLen + 13);
fprintf(stderr, " Ratio: %.2fx\n", (float)DecodedLen / EncodedLen);
fprintf(stderr, " Space savings: %.2f%% (of original)\n", 100.0f - EncodedLen * 100.0f / DecodedLen);
fprintf(stderr, " Standardized: %.2f bits/byte\n", EncodedLen * 8.0f / DecodedLen);
}
}
}
/*********************************************************************
*
* _Decode()
*
* Function description
* Decode a file.
*
* Parameters
* pInFile - Pointer to file containing the encoded bitstream.
* pOutFile - Pointer to file that receives the decoded bitstream.
* Verbose - Flag indicating verbose output.
*/
static void _Decode(FILE *pInFile, FILE *pOutFile, int Verbose) {
COMPRESS_LZMA_STREAM Stream;
int Status;
int Flush;
unsigned EncodedLen;
unsigned DecodedLen;
//
// Initialize decoder.
//
CFLEX_LZMA_DECODE_Init(&_Decoder);
//
DecodedLen = 0;
EncodedLen = 0;
Status = 0;
while (Status == 0) {
//
// Read next chunk and set up coder buffers.
//
Stream.pIn = _aInBuffer;
Stream.AvailIn = fread(_aInBuffer, 1, _BlockInSize, pInFile);
EncodedLen += Stream.AvailIn;
//
// Exit on end of file.
//
Flush = Stream.AvailIn != _BlockInSize;
//
// Decode input data, writing decoded data to file.
//
while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
//
Stream.pOut = &_aOutBuffer[0];
Stream.AvailOut = _BlockOutSize;
//
Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
if (Status >= 0 && pOutFile) {
fwrite(_aOutBuffer, 1, _BlockOutSize - Stream.AvailOut, pOutFile);
}
DecodedLen += _BlockOutSize - Stream.AvailOut;
}
}
//
if (Status < 0) {
fprintf(stderr, "FATAL: %s\n", _GetStatusText(Status));
} else {
if (Verbose) {
fprintf(stderr, "Decoding parameters:\n");
fprintf(stderr, " Coding: LC=%u, LP=%u, PB=%u\n", _Decoder.LC, _Decoder.LP, _Decoder.PB);
fprintf(stderr, " Window size: %lu bytes\n", _Decoder.WindowSize);
fprintf(stderr, "\n");
fprintf(stderr, "Statistics:\n");
fprintf(stderr, " Encoded size: %u bytes\n", EncodedLen);
fprintf(stderr, " Decoded size: %u bytes \n", DecodedLen);
fprintf(stderr, " Ratio: %.2fx\n", (float)DecodedLen / EncodedLen);
fprintf(stderr, " Space savings: %.2f%% (of original)\n", 100.0f - EncodedLen * 100.0f / DecodedLen);
fprintf(stderr, " Standardized: %.2f bits/byte\n", EncodedLen * 8.0f / DecodedLen);
}
}
}
/*********************************************************************
*
* Public code
*
**********************************************************************
*/
/*********************************************************************
*
* main()
*
* Function description
* Application entry point.
*
* Parameters
* argc - Argument count.
* argv - Argument vector.
*/
void main(int argc, char **argv) {
COMPRESS_LZMA_PARAS Paras;
int i;
int Verbose;
int Action;
const char * sInputPathname;
const char * sOutputPathname;
FILE * pInFile;
FILE * pOutFile;
//
fprintf(stderr, "\n");
fprintf(stderr, "(c) 2015, 2016 SEGGER Microcontroller GmbH & Co. KG www.segger.com\n");
fprintf(stderr, "emCompress-LZMA V%d.%02d ", CFLEX_VERSION / 10000, CFLEX_VERSION / 100 % 100);
fprintf(stderr, "compiled " __DATE__ " " __TIME__ "\n");
fprintf(stderr, "\n");
//
Paras.LC = 0;
Paras.LP = 0;
Paras.PB = 0;
Paras.WindowSize = CFLEX_MIN(COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX,
COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX);
Paras.MinLen = 3;
Paras.MaxLen = 273;
Paras.Optimize = 5;
//
sInputPathname = 0;
sOutputPathname = 0;
Action = -1; // Compress
Verbose = 1;
_BlockInSize = sizeof(_aInBuffer);
_BlockOutSize = sizeof(_aOutBuffer);
//
for (i = 1; i < argc; ++i) {
const char *sArg = argv[i];
if (strncmp(sArg, "-lc=", 4) == 0) {
Paras.LC = atoi(&sArg[4]);
} else if (strncmp(sArg, "-lp=", 4) == 0) {
Paras.LP = atoi(&sArg[4]);
} else if (strncmp(sArg, "-pb=", 4) == 0) {
Paras.PB = atoi(&sArg[4]);
} else if (strncmp(sArg, "-ws=", 4) == 0) {
Paras.WindowSize = atoi(&sArg[4]);
} else if (strncmp(sArg, "-O", 2) == 0) {
Paras.Optimize = atoi(&sArg[2]);
} else if (strncmp(sArg, "-k", 2) == 0) {
Paras.LC = 3;
Paras.LP = 0;
Paras.PB = 2;
} else if (strncmp(sArg, "-bi=", 4) == 0) {
_BlockInSize = CFLEX_MAX(1, CFLEX_MIN(atoi(&sArg[4]), sizeof(_aInBuffer)));
} else if (strncmp(sArg, "-bo=", 4) == 0) {
_BlockOutSize = CFLEX_MAX(1, CFLEX_MIN(atoi(&sArg[4]), sizeof(_aOutBuffer)));
} else if (strncmp(sArg, "-minlen=", 8) == 0) {
Paras.MinLen = atoi(&sArg[8]);
} else if (strncmp(sArg, "-maxlen=", 8) == 0) {
Paras.MaxLen = atoi(&sArg[8]);
} else if (strcmp(sArg, "-d") == 0) {
Action = +1;
} else if (strcmp(sArg, "-c") == 0) {
Action = -1;
} else if (strcmp(sArg, "-v") == 0) {
Verbose = 1;
} else if (strcmp(sArg, "-q") == 0) {
Verbose = 0;
} else if (sArg[0] == '-') {
printf("%s: unknown option %s\n", argv[0], sArg);
exit(100);
} else if (sInputPathname == 0) {
sInputPathname = sArg;
} else if (sOutputPathname == 0) {
sOutputPathname = sArg;
} else {
printf("%s: too many filenames\n", argv[0]);
exit(100);
}
}
//
if (sInputPathname == 0) {
printf("%s: require input filename\n", argv[0]);
exit(100);
}
//
pInFile = fopen(sInputPathname, "rb");
if (pInFile == 0) {
printf("%s: can't open %s for reading\n", argv[0], sInputPathname);
exit(100);
}
//
if (Action < 0 && CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras) < 0) {
printf("%s: inconsistent or invalid compression parameters\n", argv[0]);
exit(100);
}
//
pOutFile = 0;
if (sOutputPathname != 0) {
pOutFile = fopen(sOutputPathname, "wb");
if (pOutFile == 0) {
printf("%s: can't open %s for writing\n", argv[0], sInputPathname);
fclose(pInFile);
exit(100);
}
}
//
if (Action < 0) {
_Encode(pInFile, pOutFile, &Paras, Verbose);
} else {
_Decode(pInFile, pOutFile, Verbose);
}
//
if (pInFile) {
fclose(pInFile);
}
if (pOutFile) {
fclose(pOutFile);
}
}
/*************************** End of file ****************************/
API reference
This section describes the public API for emCompress-Flex. Any
functions or data structures that are not described here but are
exposed through inclusion of either CFLEX_ENCODE.h
or CFLEX_DECODE.h must be considered private and
subject to change.
Compressor functions
CFLEX_LZMA_ENCODE_Init()
Description
Initialize LZMA encoder.
Prototype
int CFLEX_LZMA_ENCODE_Init( COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
const COMPRESS_LZMA_PARAS * pParas);
Parameters
| Parameter | Description |
|---|
| pSelf | Pointer to encoder context. |
| pParas | Pointer to encoder parameters. |
Return value
| < 0 | Error in one or more parameters. |
| ≥ 0 | Success. |
CFLEX_LZMA_ENCODE_Run()
Description
Run LZMA encoder.
Prototype
int CFLEX_LZMA_ENCODE_Run(COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
COMPRESS_LZMA_STREAM * pStream,
int Flush);
Parameters
| Parameter | Description |
|---|
| pSelf | Pointer to LZMA encoder context. |
| pStream | Pointer to stream communication object. |
| Flush | End of stream flag. |
Return value
| ≤ 0 | Processing error. |
| = 0 | OK, continue encoding. |
| > 0 | Encoding complete. |
Description
Write LZMA-alone header.
Prototype
void CFLEX_LZMA_ENCODE_WrHeader(COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
U8 * pHeader);
Parameters
| Parameter | Description |
|---|
| pSelf | Pointer to LZMA encoder context. |
| pHeader | Pointer to object the receives the 13-octet LZMA-alone header. |
Decompressor functions
CFLEX_LZMA_DECODE_Init()
Description
Initialize LZMA decoder.
Prototype
void CFLEX_LZMA_DECODE_Init(COMPRESS_LZMA_DECODE_CONTEXT * pSelf);
Parameters
| Parameter | Description |
|---|
| pSelf | Pointer to decoder context. |
CFLEX_LZMA_DECODE_Run()
Description
Run LZMA decoder.
Prototype
ILEAST16 CFLEX_LZMA_DECODE_Run(COMPRESS_LZMA_DECODE_CONTEXT * pSelf,
COMPRESS_LZMA_STREAM * pStream,
ILEAST16 Flush);
Parameters
| Parameter | Description |
|---|
| pSelf | Pointer to decoder context. |
| pStream | Pointer to stream context. |
| Flush | End of input stream flag. |
Return value
| < 0 | Processing error or bitstream error |
| = 0 | Call again to continue processing |
| > 0 | Decompression complete |