📄 emCompress-ToGo User Guide & Reference Manual
📄 emCrypt User Guide & Reference Manual
📄 emDropbox User Guide & Reference Manual
📄 emFile User Guide & Reference Manual
📄 emNet User Guide & Reference Manual
📄 emSecure-ECDSA User Guide & Reference Manual
📄 emSecure-RSA User Guide & Reference Manual
📄 emSSH User Guide & Reference Manual
📄 emSSL User Guide & Reference Manual
📄 emUSB-Device User Guide & Reference Manual
📄 emUSB-Host User Guide & Reference Manual
📄 emWeb User Guide & Reference Manual
📄 emWin User Guide & Reference Manual
📄 IoT Toolkit User Guide & Reference Manual
📄 SEGGER Assembler User Guide & Reference Manual
📄 SEGGER Floating-point Library User Guide & Reference Manual
📄 SEGGER Linker User Guide & Reference Manual
📄 SEGGER Runtime Library User Guide & Reference Manual
📄 AppWizard User Guide & Reference Manual
📄 embOS & embOS-MPU User Guide & Reference Manual
📄 emCompress-Embed User Guide & Reference Manual
📄 emCompress-Flex User Guide & Reference Manual

emCompress-Flex User Guide & Reference Manual

LZMA compression system.

Introduction to emCompress-Flex

This section presents an overview of emCompress-Flex, its structure, and its capabilities.

What is emCompress-Flex?

emCompress-Flex is a compression system that is able to reduce the storage requirements of data that must be embedded into an application. Typical uses of emCompress-Flex are:

Of course, emCompress-Flex is not limited to these applications, it can be used whenever it’s beneficial to reduce the size of stored content.

Features

emCompress-Flex is written in standard ANSI C and can run on virtually any CPU. Here’s a list summarizing the main features of emCompress-Flex:

We recommend keeping emCompress-Flex separate from your application files. It is good practice to keep all the program files (including the header files) together in the COMPRESS subdirectory of your project’s root directory. This practice has the advantage of being very easy to update to newer versions of emCompress-Flex by simply replacing the COMPRESS and SEGGER directories. Your application files can be stored anywhere.

Note

When updating to a newer emCompress-Flex version: as files may have been added, moved or deleted, the project directories may need to be updated accordingly.

Package content

emCompress-Flex is provided in source code and contains everything needed. The following table shows the content of the emCompress Package:

Files Description
Application Supporting example application sources.
Config Configuration header files.
Doc emCompress-Flex documentation.
COMPRESS emCompress-Flex decompressor and optional compressor source code.
Windows/COMPRESS Compressor DLL and supporting applications in binary form.

Compressor include directories

You should make sure that the include path contains the following directories (the order of inclusion is of no importance):

Note

Always make sure that you have only one version of each file!

It is frequently a major problem when updating to a new version of emCompress-Flex if you have old files included and therefore mix different versions. If you keep emCompress-Flex in the directories as suggested (and only in these), this type of problem cannot occur. When updating to a newer version, you should be able to keep your configuration files and leave them unchanged. For safety reasons, we recommend backing up (or at least renaming) the COMPRESS directories before updating.

Compressor

This section describes the emCompress-Flex LZMA compressor interface.

Distribution types

emCompress-Flex is shipped with the decompressor in souce and a DLL and corresponding library where the compressor is preconfigured with a fixed in memory footprint.

DLL and library distribution

Configuration of the compressor with the DLL is not possible: it is preconfigured to allow compression with all configuration parameters up to their maximum value, and with dictionary (window) sizes up to one megabyte. The files CFLEX_ENCODE_Conf.h and COMPRESS_LZMA_ENCODE_Conf.h reflect the build configuration of the DLL and must not be modified.

Sample applications

emCompress-Flex ships with a number of sample applications that show how to integrate shell capability into your application. Each sample application adds an additional capability to the shell and is a small incremental step from the previous version.

The sample applications are:

Application Description
CFLEX_OneShot.c Single-shot compression.
CFLEX_Compress.c Stream data into and out of the compressor.
CFLEX_Optimize.c Search for optimal compressor parameters.

A note on the samples

Each sample that we present in this section is written in a style that makes it easy to describe and that fits comfortably within the margins of printed paper. Therefore, it may well be that you would rewrite the sample to have a slightly different structure that fits better, but please keep in mind that these examples are written with clarity as the prime objective, and to that end we sacrifice some brevity and efficiency.

Where to find the sample code

Target-independent code

All target-independent samples are included in the Application directory of the emCompress-Flex distribution.

The entry point for all target-dependent code is the function MainTask(), which is usually run by the host operating system in a task context. To run this code on a PC or workstation, simply call MainTask() from main():

void MainTask(void);

int main(int argc, char **argv) {
  MainTask();
  return 0;
}

Host-only code

All applications that require a file system and are intended to run on a PC or workstation are included in the Windows/COMPRESS directory of the emCompress-Flex distribution.

Single-shot compression

The first example, CFLEX_OneShot.c, provides a simple way to compress a single block of data without streaming. This is the smallest working example but is not not how you would typically compress data in your application.

For a complete listing of this application, see CFLEX_OneShot.c complete listing.

Include files

The emCompress-FLEX encoding API is exposed by including the file CFLEX_ENCODE.h:

#include "CFLEX.h"
#include "CFLEX_ENCODE.h"

Setting up the compressor

Application entry starts to set up compression parameters.

void main(int argc, char **argv) {
  COMPRESS_LZMA_STREAM Stream;
  COMPRESS_LZMA_PARAS  Paras;
  int                  Status;
  //
  Paras.LC         = 0;   
  Paras.LP         = 0;
  Paras.PB         = 0;
  Paras.WindowSize = 1024;   
  Paras.MinLen     = 3;   
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;   
  //
  CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);   

Step 1: Configure the LZMA probabilty model

The three parameters LC, LP, and PB configure the LZMA probability model and how the LZMA encoder performs. For better compression, these parameters are usually nonzero but as they grow more memory is needed to hold the encoder state. This example uses the minimum amount of working store possible by setting these parameters to zero.

Step 2: Configure the window size

The window size is the maximum number of octets that the decoder will need to maintain as context in order to copy a reference to previously-decoded content. As the window size increases, the compressed output usually decreases as there are more opportunities to find matching substrings within the window.

Step 3: Configure match lengths

The match length range defined by LZMA is between 2 and 273 octets. However, a minimum match of two octets produces compressed images that are larger than images compressed with a minimum match of three bytes, so a good default is a minimum match length of three octets.

Step 4: Configure optimization

The optimization level, between 0 and 9, configures the trade-off between compression efficiency and encoding time. Lower optimization levels will compress faster but will result in larger compressed images, whereas higher optimization will compress slower and will result in smaller compressed images. A default value of 5 is a good trade-off.

Step 5: Initialize compressor

Once the compression parameters are set up, the compressor context is initialized for compression using CFLEX_LZMA_ENCODE_Init.

Running the compressor

Once the compressor has been configured, data can be passed through it for compression. This example compresses some fixed, static data:

static const U8 _aMessage[] = {
  "With increasing complexity of today's devices, "
  "customers expect firmware updates over the life "
  "of a device. It's important to be able to replace "
  "both firmware images and FPGA configuration bitstreams "
  "in the field. Typically, firmware upgrades and other "
  "static content only grow in size over the lifetime of "
  "a device: features are added to software, not taken "
  "away, it's just what happens. Using quickly-written "
  "ad-hoc utilities to convert content directly to compilable "
  "code (possibly with some form of simple compression) is "
  "usual, but it's towards the end of a project when you "
  "realize that the static content just won't fit into your "
  "device with the baggage of new firmware features."
};

And proceed to the code that prepares the compressor to accept it:

  //
  Stream.pIn      = &_aMessage[0];   
  Stream.AvailIn  = sizeof(_aMessage);
  Stream.pOut     = &_aOutput[0];   
  Stream.AvailOut = sizeof(_aOutput);
  //
  Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, 1);   
  //
  if (Status < 0) {  
    printf("Error encoding data\n");
  } else if (Status == 0) {
    printf("Did not encode completely, output buffer too small\n");
  } else {
    printf("%d bytes compressed to %u bytes\n",
           sizeof(_aMessage),
           sizeof(_aOutput) - Stream.AvailOut);   
  }
}

Step 1: Set up data to compress

The input stream to the compressor is defined by the pIn and AvailIn members of the COMPRESS_LZMA_STREAM structure. The pIn member points to the first octet of data to be compressed and AvailIn defines the number of octets of input that are available to the compressor.

Step 2: Set up bitstream output buffer

In a similar manner to the input, the output stream is defined by the pOut and AvailOut members of the COMPRESS_LZMA_STREAM structure. The pOut member points to the object that receives the octets output by the compressor and AvailOut defines the number of octets that buffer can hold.

Step 3: Run the compressor

Once the input and output are set up, the compression is performed by calling CFLEX_LZMA_ENCODE_Run. This function read data from the input stream, compresses it, and writes it to the output stream. We provide the encoding context, the stream object that specifies the data areas, and a final non-zero “flush” value that indicates to the compressor that this is all the input data it must deal with.

Step 4: Decode the processing status

When the compressor has worked its way through the compression algorithm, it returns a status code indicating its processing status. There are three different states that the compressor will return:

In this case we expect the compressor to immediately finish as the output buffer that holds the compressed bitstream is much larger than the text input to the compressor. Whilst this will not always be the case, the text provided to the compressor in this case does indeed have redundancy and so compresses to a smaller size—running the application reveals how much.

Step 5: Extract the compressed data

When the compressor returns with a nonnegative return code, the compressed data can be read from the buffer given to the compressor. On return, the AvailOut member is decreased by one and the pIn pointer increased by one for each octet written to the buffer. The number of available octets, therefore, is equal to the number of octets remaining in the buffer on return subtracted from the size of the buffer on entry.

The above application generates the following output:

C:> CFLEX_OneShot.exe

687 bytes compressed to 428 bytes

C:> _

CFLEX_OneShot.c complete listing

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Compress.c
Purpose     : Example emCompress-FLEX one-shot compression.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>

/*********************************************************************
*
*       Static const data
*
**********************************************************************
*/

static const U8 _aMessage[] = {
  "With increasing complexity of today's devices, "
  "customers expect firmware updates over the life "
  "of a device. It's important to be able to replace "
  "both firmware images and FPGA configuration bitstreams "
  "in the field. Typically, firmware upgrades and other "
  "static content only grow in size over the lifetime of "
  "a device: features are added to software, not taken "
  "away, it's just what happens. Using quickly-written "
  "ad-hoc utilities to convert content directly to compilable "
  "code (possibly with some form of simple compression) is "
  "usual, but it's towards the end of a project when you "
  "realize that the static content just won't fit into your "
  "device with the baggage of new firmware features."
};

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8                           _aOutput[512];

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_STREAM Stream;
  COMPRESS_LZMA_PARAS  Paras;
  int                  Status;
  //
  Paras.LC         = 0;
  Paras.LP         = 0;
  Paras.PB         = 0;
  Paras.WindowSize = 1024;
  Paras.MinLen     = 3;
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;
  //
  CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
  //
  Stream.pIn      = &_aMessage[0];
  Stream.AvailIn  = sizeof(_aMessage);
  Stream.pOut     = &_aOutput[0];
  Stream.AvailOut = sizeof(_aOutput);
  //
  Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, 1);
  //
  if (Status < 0) {
    printf("Error encoding data\n");
  } else if (Status == 0) {
    printf("Did not encode completely, output buffer too small\n");
  } else {
    printf("%u bytes compressed to %u bytes\n",
           (unsigned)sizeof(_aMessage),
           (unsigned)(sizeof(_aOutput) - Stream.AvailOut));
  }
}

/*************************** End of file ****************************/

Compressing a file

This example, CFLEX_Compress.c, expands on the previous example to compress a file, block by block, writing the output to a compressed file with a correct header.

For a complete listing of this application, see CFLEX_Compress.c complete listing.

Incremental compression

Rather than describe the boilerplate code of processing a command line, opening files, and setting up the encoder as previously, we present the code that incrementally compresses a file:

Status = 0;   
while (Status == 0) {
  Stream.pIn     = _aInBuffer;   
  Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
  //
  Flush = Stream.AvailIn != sizeof(_aInBuffer);   
  //
  while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {   
    //
    Stream.pOut     = &_aOutBuffer[0];   
    Stream.AvailOut = sizeof(_aOutBuffer);
    //
    Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);   
    if (Status >= 0) {
      fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
    }
  }
}

Step 1: Set up main compress loop

The compressor maintains the current compression status in the variable Status; when the status is nonzero, compression has completed either successfully or with an error.

Step 2: Set up input

There are no surprises here, simply set up the block to be passed to the compressor by reading the source file.

Step 3: Set up flush flag

The compressor needs to be told that there will be no more input and finish up compression: that happens when we have not completely filled the input buffer from the source file.

Step 4: Compress until block consumed

The inner loop will continually call the compressor to consume the input buffer and deliver data to the output buffer. The while loop ensures that we call the compressor when we have more data or we need to flush. When the status indicates that compression is complete, the loop terminates.

Step 5: Set up output buffer

The output buffer can be any size—larger buffers may lead to faster compression, but the overhead is not very significant.

Step 6: Compress and write output buffer

This is the same as the previous application, only this time we write the compressed output to the file, and this concludes the compression loop.

Encapsulating the bitstream

The compression loop generates a compressed bitstream, but the header which the decompressor requires is missing. The header cannot be constructed until the entire bitstream is compressed, so there additional code is required that deals with this.

The header is 13 bytes in size:

U8 aHeader[13];

Before writing the compressed bitstream in the loop above the file pointer is advanced to reserve space for a header:

fseek(pOutFile, sizeof(aHeader), SEEK_SET);

Once the compressor has competed compression, the header can be constructed and written to the start of the file:

CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
fseek(pOutFile, 0, SEEK_SET);
fwrite(aHeader, 1, sizeof(aHeader), pOutFile);

This concludes the example.

CFLEX_Compress.c complete listing

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Compress.c
Purpose     : Example emCompress-FLEX one-shot compression.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>

/*********************************************************************
*
*       Defines, configurable
*
**********************************************************************
*/

#define BUFFER_IN_SIZE    (32*1024)
#define BUFFER_OUT_SIZE   (32*1024)

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8                           _aInBuffer [BUFFER_IN_SIZE];
static U8                           _aOutBuffer[BUFFER_OUT_SIZE];

/*********************************************************************
*
*       Static code
*
**********************************************************************
*/

/*********************************************************************
*
*       _GetStatusText()
*  
*  Function description
*    Decode emCompress-LZMA status code.
*
*  Parameters
*    Status - Status code.
*
*  Return value
*    Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
  if (Status >= 0) {
    return "OK";
  }
  switch (Status) {
  case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
  case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
  case CFLEX_STATUS_USAGE_ERROR:     return "Usage error";
  default:                           return "Unknown error";
  }
}

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_STREAM   Stream;
  COMPRESS_LZMA_PARAS    Paras;
  FILE*                  pInFile;
  FILE*                  pOutFile;
  U8                     aHeader[13];
  int                    Flush;
  int                    Status;
  //
  if (argc != 3) {
    fprintf(stderr, "Usage: %s <input-file> <output-file>\n", argv[0]);
    exit(100);
  }
  //
  pInFile = fopen(argv[1], "rb");
  if (pInFile == 0) {
    printf("%s: can't open %s for reading\n", argv[0], argv[1]);
    exit(100);
  }
  //
  pOutFile = fopen(argv[2], "wb");
  if (pOutFile == 0) {
    printf("%s: can't open %s for writing\n", argv[0], argv[2]);
    fclose(pInFile);
    exit(100);
  }
  //
  Paras.LC         = 0;
  Paras.LP         = 0;
  Paras.PB         = 0;
  Paras.WindowSize = 1024;
  Paras.MinLen     = 3;
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;
  //
  CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
  //
  // Reserve space for an LZMA header in the output file.
  //
  fseek(pOutFile, sizeof(aHeader), SEEK_SET);
  //
  Status = 0;
  while (Status == 0) {
    //
    // Read next chunk and set up coder buffers.
    //
    Stream.pIn     = _aInBuffer;
    Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
    //
    // Flush compressor on end of input file.
    //
    Flush = Stream.AvailIn != sizeof(_aInBuffer);
    //
    // Encode input data, writing encoded data to file.
    //
    while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
      //
      Stream.pOut     = &_aOutBuffer[0];
      Stream.AvailOut = sizeof(_aOutBuffer);
      //
      Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
      if (Status >= 0) {
        fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
      }
    }
  }
  //
  // Rewind file and write correct header.
  //
  CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
  fseek(pOutFile, 0, SEEK_SET);
  fwrite(aHeader, 1, sizeof(aHeader), pOutFile);
  //
  fclose(pInFile);
  fclose(pOutFile);
  //
  if (Status < 0) {
    printf("FATAL: %s\n", _GetStatusText(Status));
    exit(100);
  }
}

/*************************** End of file ****************************/

Optimizing compression parameters

If you require the best compression possible, you can compress a file with varying parameters and search for the best. It’s not possible to find optimal compression parameters, in general, but it is possible to change the context parameters and search for those that provide the best compression.

The optimizer runs over the input file multiple times, varying the probability model’s parameters:

for (Paras.LC = 0; Paras.LC <= 8; ++Paras.LC) {
  for (Paras.LP = 0; Paras.LP <= 4; ++Paras.LP) {
    for (Paras.PB = 0; Paras.PB <= 4; ++Paras.PB) {

The compressor is run inside the innermost loop, using the selected parameters, and the result printed. The output from the application shows how the search progresses:

C:> CFLEX_Optimize.exe K66_SEGGER_emPower.bin

LC=0, LP=0, PB=0: 150842
LC=0, LP=0, PB=1: 149306
LC=0, LP=1, PB=0: 147939
LC=0, LP=1, PB=1: 146402
LC=1, LP=1, PB=1: 145641
LC=2, LP=1, PB=1: 145139
LC=3, LP=1, PB=1: 144967

C:> _

The application itself is very straightforward and is not explained further.

CFLEX_Optimize.c complete listing

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Optimize.c
Purpose     : Find optimal compression parameters for a file.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_ENCODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*********************************************************************
*
*       Defines, configurable
*
**********************************************************************
*/

#define BUFFER_IN_SIZE    (32*1024)
#define BUFFER_OUT_SIZE   (32*1024)

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static U8                           _aInBuffer [BUFFER_IN_SIZE];
static U8                           _aOutBuffer[BUFFER_OUT_SIZE];

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_STREAM   Stream;
  COMPRESS_LZMA_PARAS    Paras;
  const char           * sInputPathname;
  FILE                 * pInFile;
  U64                    MinSize;
  U64                    EncodedSize;
  int                    Flush;
  int                    Status;
  int                    i;
  unsigned               MaxLP;
  unsigned               MaxLC;
  unsigned               MaxPB;
  //
  Paras.WindowSize = COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX;
  Paras.MinLen     = 3;
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;
  MaxLC            = COMPRESS_LZMA_CONFIG_ENCODE_LC_MAX;
  MaxLP            = COMPRESS_LZMA_CONFIG_ENCODE_LP_MAX;
  MaxPB            = COMPRESS_LZMA_CONFIG_ENCODE_PB_MAX;
  //
  sInputPathname = 0;
  //
  for (i = 1; i < argc; ++i) {
    const char *sArg = argv[i];
    if (strncmp(sArg, "-lc=", 4) == 0) {
      MaxLC = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLC);
    } else if (strncmp(sArg, "-lp=", 4) == 0) {
      MaxLP = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLP);
    } else if (strncmp(sArg, "-pb=", 4) == 0) {
      MaxPB = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxPB);
    } else if (strncmp(sArg, "-ws=", 4) == 0) {
      Paras.WindowSize = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-O", 2) == 0) {
      Paras.Optimize = atoi(&sArg[2]);
    } else if (strncmp(sArg, "-k", 2) == 0) {
      Paras.LC = 3;
      Paras.LP = 0;
      Paras.PB = 2;
    } else if (strncmp(sArg, "-minlen=", 8) == 0) {
      Paras.MinLen = atoi(&sArg[8]);
    } else if (strncmp(sArg, "-maxlen=", 8) == 0) {
      Paras.MaxLen = atoi(&sArg[8]);
    } else if (sArg[0] == '-') {
      printf("%s: unknown option %s\n", argv[0], sArg);
      exit(100);
    } else if (sInputPathname == 0) {
      sInputPathname = sArg;
    } else {
      printf("%s: too many filenames\n", argv[0]);
      exit(100);
    }
  }
  //
  if (sInputPathname == 0) {
    printf("%s: require input filename\n", argv[0]);
    exit(100);
  }
  //
  pInFile = fopen(sInputPathname, "rb");
  if (pInFile == 0) {
    printf("%s: can't open %s for reading\n", argv[0], argv[1]);
    exit(100);
  }
  MinSize          = ~0ULL;
  //
  for (Paras.LC = 0; Paras.LC <= MaxLC; ++Paras.LC) {
    for (Paras.LP = 0; Paras.LP <= MaxLP; ++Paras.LP) {
      for (Paras.PB = 0; Paras.PB <= MaxPB; ++Paras.PB) {
        //
        Status = CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras);
        if (Status < 0) {
          continue;
        }
        //
        fseek(pInFile, 0, SEEK_SET);
        //
        EncodedSize = 0;
        Status = 0;
        while (Status == 0) {
          //
          // Read next chunk and set up coder buffers.
          //
          Stream.pIn     = _aInBuffer;
          Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
          //
          // Flush compressor on end of input file.
          //
          Flush = Stream.AvailIn != sizeof(_aInBuffer);
          //
          // Encode input data.
          //
          while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
            //
            Stream.pOut     = &_aOutBuffer[0];
            Stream.AvailOut = sizeof(_aOutBuffer);
            //
            Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
            EncodedSize += sizeof(_aOutBuffer) - Stream.AvailOut;
          }
        }
        //
        printf("\rLC=%d, LP=%d, PB=%d: %llu            \r",
               Paras.LC, Paras.LP, Paras.PB, EncodedSize);
        if (EncodedSize < MinSize) {
          MinSize = EncodedSize;
          printf("\n");
        }
      }
    }
  }
  //
  printf("\r                            \r");
}

/*************************** End of file ****************************/

Prefiltering

It is possible to improve the effectiveness of compression when the compressor knows the structure of the data and can use a domain-specific filter. In particular, for firmware upgrades, preprocessing the input to the LZMA compressor to provide improve the probability of matches detected by the LZSS engine is particularly helpful.

Here is a typical application without prefiltering, searching for optimized compression parameters:

C:> CFLEX_Optimize.exe K66_SEGGER_emPower.bin -ws=4096

LC=0, LP=0, PB=0: 160402
LC=0, LP=0, PB=1: 158931
LC=0, LP=1, PB=0: 155551
LC=0, LP=1, PB=1: 154080
LC=1, LP=1, PB=1: 152754
LC=2, LP=1, PB=1: 151886
LC=3, LP=1, PB=1: 151309

C:> _

With prefiltering using architecture-specific transformations (in this case for the Thumb-2 instruction set), we further reduce the compressed size of the firmware:

C:> CFLEX_Filter.exe K66_SEGGER_emPower.bin -ws=4096

LC=0, LP=0, PB=0: 153128
LC=0, LP=0, PB=1: 152131
LC=0, LP=1, PB=0: 149096
LC=0, LP=1, PB=1: 148099
LC=1, LP=1, PB=1: 147282
LC=2, LP=1, PB=0: 146972
LC=2, LP=1, PB=1: 145975
LC=3, LP=1, PB=1: 145622

C:> _

Decompressor

This section describes the emCompress-Flex decompressor interface.

Delivery format

The emCompress-Flex decompressor is delivered as source code for integration into your target system.

Streaming decompression

The first and only example, CFLEX_Decompress.c, is an application that will decompress the output file from the previous compressor.

For a complete listing of this application, see CFLEX_Decompress.c complete listing.

Decompressor include files

The emCompress-Flex decoding API is exposed by including the file CFLEX_DECODE.h:

#include "CFLEX.h"
#include "CFLEX_DECODE.h"

Decompressor loop

The decompressor uses the same stream interface object as the compressor, COMPRESS_LZMA_STREAM. And the decompressor “run” code uses an identical set of parameters and return codes which makes it very simple to use.

In fact, the decompressor loop is identical in form to the compressor loop:

Status = 0;
while (Status == 0) {
  //
  Stream.pIn     = _aInBuffer;
  Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
  EncodedLen    += Stream.AvailIn;
  //
  Flush = Stream.AvailIn != _BlockInSize;
  //
  while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
    //
    Stream.pOut     = &_aOutBuffer[0];
    Stream.AvailOut = sizeof(_aOutBuffer);
    //
    Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
    if (Status >= 0) {
      fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
    }
  }
}

CFLEX_Decompress.c complete listing

/*********************************************************************
*               (c) SEGGER Microcontroller GmbH & Co. KG             *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Decompress.c
Purpose     : Example emCompress-FLEX one-shot decompression.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>

/*********************************************************************
*
*       Defines, configurable
*
**********************************************************************
*/

#define BUFFER_IN_SIZE    (32*1024)
#define BUFFER_OUT_SIZE   (32*1024)

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static U8                           _aInBuffer [BUFFER_IN_SIZE];
static U8                           _aOutBuffer[BUFFER_OUT_SIZE];

/*********************************************************************
*
*       Static code
*
**********************************************************************
*/

/*********************************************************************
*
*       _GetStatusText()
*  
*  Function description
*    Decode emCompress-LZMA status code.
*
*  Parameters
*    Status - Status code.
*
*  Return value
*    Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
  if (Status >= 0) {
    return "OK";
  }
  switch (Status) {
  case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
  case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
  case CFLEX_STATUS_USAGE_ERROR:     return "Usage error";
  default:                                   return "Unknown error";
  }
}

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_STREAM  Stream;
  FILE*                 pInFile;
  FILE*                 pOutFile;
  int                   Flush;
  int                   Status;
  //
  if (argc != 3) {
    fprintf(stderr, "Usage: %s <input-file> <output-file>\n", argv[0]);
    exit(100);
  }
  //
  pInFile = fopen(argv[1], "rb");
  if (pInFile == 0) {
    printf("%s: can't open %s for reading\n", argv[0], argv[1]);
    exit(100);
  }
  //
  pOutFile = fopen(argv[2], "wb");
  if (pOutFile == 0) {
    printf("%s: can't open %s for writing\n", argv[0], argv[2]);
    fclose(pInFile);
    exit(100);
  }
  //
  CFLEX_LZMA_DECODE_Init(&_Decoder);
  //
  Status = 0;
  while (Status == 0) {
    //
    // Read next chunk and set up coder buffers.
    //
    Stream.pIn     = _aInBuffer;
    Stream.AvailIn = fread(_aInBuffer, 1, sizeof(_aInBuffer), pInFile);
    //
    // Flush compressor on end of input file.
    //
    Flush = Stream.AvailIn != sizeof(_aInBuffer);
    //
    // Encode input data, writing encoded data to file.
    //
    while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
      //
      Stream.pOut     = &_aOutBuffer[0];
      Stream.AvailOut = sizeof(_aOutBuffer);
      //
      Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
      if (Status >= 0) {
        fwrite(_aOutBuffer, 1, sizeof(_aOutBuffer) - Stream.AvailOut, pOutFile);
      }
    }
  }
  //
  fclose(pInFile);
  fclose(pOutFile);
  //
  if (Status < 0) {
    printf("FATAL: %s\n", _GetStatusText(Status));
    exit(100);
  }
}

/*************************** End of file ****************************/

Configuring the decompressor

The following definitions must be configured for emCompress-Flex’s decompressor. The user configuration files for this are CFLEX_DECODE_Conf.h and COMPRESS_LZMA_DECODE_Conf.h. The configuration options are described in the following sections.

Decompressor data types

Default

There is no default as the required emCompress-Flex data types are configured by using a type definition rather than a preprocessor symbol.

Recommended configuration

typedef unsigned long long U64;
typedef unsigned long      U32;
typedef unsigned short     U16;
typedef unsigned char       U8;
typedef unsigned int  ULEAST16;
typedef int           ILEAST16;

Description

Defines the types that the emCompress-Flex decompressor will in its data structures when compressing.

The types above are typical for a 32-bit machine, but the decompressor will work with any type definition for each as long as each type can store the prescribed number of bits without loss. For instance, it may well be that the compilation system provides a <stdint.h> header file, in which case the following is acceptable:

typedef uint64_t             U64;
typedef uint32_t             U32;
typedef uint16_t             U16;
typedef uint8_t               U8;
typedef int_least16_t   ILEAST16;
typedef uint_least16_t  ULEAST16;

Maximum accepted literal context bits

Default

#define COMPRESS_LZMA_CONFIG_DECODE_LC_MAX     8

Description

Configures the maximum number of literal context bits that the decompressor will support. Valid configuration values for this symbol are 0 through 8. Smaller values will reduce the memory required for the decoder context but will also limit the capability of the compressor when encoding a bitstream acceptable to the decompressor.

It is a (diagnosed) error to attempt decoding a bitstream which is encoded with an LC value greater than the configured maximum.

Maximum accepted literal position context bits

Default

#define COMPRESS_LZMA_CONFIG_DECODE_LP_MAX     4

Description

Configures the maximum number of literal position context bits that the decompressor will support. Valid configuration values for this symbol are 0 through 4. Smaller values will reduce the memory required for the decoder context but will also limit the capability of the compressor when encoding a bitstream acceptable to the decompressor.

It is a (diagnosed) error to attempt decoding a bitstream which is encoded with an LP value greater than the configured maximum.

Maximum accepted position bits

Default

#define COMPRESS_LZMA_CONFIG_DECODE_PB_MAX     4

Description

Configures the maximum number of position context bits that the decompressor will support. Valid configuration values for this symbol are 0 through 4. Smaller values will reduce the memory required for the decoder context but will also limit the capability of the compressor when encoding a bitstream acceptable to the decompressor.

It is a (diagnosed) error to attempt decoding a bitstream which is encoded with an PB value greater than the configured maximum.

Maximum accepted window size

Default

#define COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX  (1024*1024)

Description

Configures the maximum window size that the decompressor will support. Valid configuration values for this symbol are 1 through 0xFFFFFFFF. Smaller values will reduce the memory required for the decoder context but will also limit the capability of the compressor when encoding a bitstream acceptable to the decompressor.

It is a (diagnosed) error to attempt decoding a bitstream which is encoded with a window size greater than the configured maximum decoder window size.

Shipped decompressor configuration

This is the shipped configuration file, CFLEX_DECODE_Conf.h.

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_DECODE_Conf.h
Purpose     : Configuration settings for emCompress-FLEX decoder.

*/

#ifndef CFLEX_DECODE_CONF_H
#define CFLEX_DECODE_CONF_H

#ifdef __cplusplus
extern "C" {
#endif

#ifdef __cplusplus
}
#endif

#endif

/*************************** End of file ****************************/

This is the shipped configuration file, COMPRESS_LZMA_DECODE_Conf.h.

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : COMPRESS_LZMA_DECODE_Conf.h
Purpose     : Configuration settings for emCompress-FLEX LZMA decoder.

*/

#ifndef COMPRESS_LZMA_DECODE_CONF_H
#define COMPRESS_LZMA_DECODE_CONF_H

#ifdef __cplusplus
extern "C" {
#endif

// Maximum number of literal context (LC) bits the decoder will accept  [0, 8].
#define COMPRESS_LZMA_CONFIG_DECODE_LC_MAX     8

// Maximum number of literal position (LP) bits the decoder will accept [0, 4].
#define COMPRESS_LZMA_CONFIG_DECODE_LP_MAX     4

// Maximum number of position bits (PB) bits the decoder will accept    [0, 4].
#define COMPRESS_LZMA_CONFIG_DECODE_PB_MAX     4

// Maximum size of LZMA window
#define COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX  (1024*1024)

#ifdef __cplusplus
}
#endif

#endif

/*************************** End of file ****************************/

Decompressor memory use

The decompressor has no static data requirement, the only memory required to decompress a stream is provided to the decompressor by the client in a COMPRESS_LZMA_DECODE_CONTEXT object.

The decoder context size varies according to selected emCompress-Flex LZMA decoder configuration and the specific compilation options selected to compile the decompressor.

Because the target processor’s type sizes and alignments are not known to the generic compressor, and cannot easily be modeled without significant probability of error, the sizes presented here are for a 32-bit Cortex-M processor which should generally translate well to many 32-bit architectures including 32-bit “x86” targets. For nonstandard architectures, such as pure DSPs which are typically word-addressed machines, the memory requirement in words can be derived by configuring the compiler and using the sizeof operator to query the memory size in units of char rather than byte units.

The context size of the decoder is modeled using the four parameters LC, LP, PB, and WindowSize. The application CFLEX_Sizes.c will estimate the memory required for a decompressor context.

It must be stressed that this computation is an estimate and the exact memory requirement can be computed with the sizeof operator applied to the COMPRESS_LZMA_DECODE_CONTEXT. However, the exact size of the decoding context is generally not required, only a sense of its magnitude.

The following table displays the decompressor context size for each combination of LC. LP, and PB. Each size is measured in kilobytes with a zero window size, for a typical 32-bit byte-addressed machine.

LP PB LC=0 LC=1 LC=2 LC=3 LC=4 LC=5 LC=6 LC=7 LC=8
0 0 3.8 5.3 8.3 14.3 26.3 50.3 98.3 194.3 386.3
0 1 4.0 5.5 8.5 14.5 26.5 50.5 98.5 194.5 386.5
0 2 4.3 5.8 8.8 14.8 26.8 50.8 98.8 194.8 386.8
0 3 4.9 6.4 9.4 15.4 27.4 51.4 99.4 195.4 387.4
0 4 6.2 7.7 10.7 16.7 28.7 52.7 100.7 196.7 388.7
1 0 5.3 8.3 14.3 26.3 50.3 98.3 194.3 386.3 770.3
1 1 5.5 8.5 14.5 26.5 50.5 98.5 194.5 386.5 770.5
1 2 5.8 8.8 14.8 26.8 50.8 98.8 194.8 386.8 770.8
1 3 6.4 9.4 15.4 27.4 51.4 99.4 195.4 387.4 771.4
1 4 7.7 10.7 16.7 28.7 52.7 100.7 196.7 388.7 772.7
2 0 8.3 14.3 26.3 50.3 98.3 194.3 386.3 770.3 1538.3
2 1 8.5 14.5 26.5 50.5 98.5 194.5 386.5 770.5 1538.5
2 2 8.8 14.8 26.8 50.8 98.8 194.8 386.8 770.8 1538.8
2 3 9.4 15.4 27.4 51.4 99.4 195.4 387.4 771.4 1539.4
2 4 10.7 16.7 28.7 52.7 100.7 196.7 388.7 772.7 1540.7
3 0 14.3 26.3 50.3 98.3 194.3 386.3 770.3 1538.3 3074.3
3 1 14.5 26.5 50.5 98.5 194.5 386.5 770.5 1538.5 3074.5
3 2 14.8 26.8 50.8 98.8 194.8 386.8 770.8 1538.8 3074.8
3 3 15.4 27.4 51.4 99.4 195.4 387.4 771.4 1539.4 3075.4
3 4 16.7 28.7 52.7 100.7 196.7 388.7 772.7 1540.7 3076.7
4 0 26.3 50.3 98.3 194.3 386.3 770.3 1538.3 3074.3 6146.3
4 1 26.5 50.5 98.5 194.5 386.5 770.5 1538.5 3074.5 6146.5
4 2 26.8 50.8 98.8 194.8 386.8 770.8 1538.8 3074.8 6146.8
4 3 27.4 51.4 99.4 195.4 387.4 771.4 1539.4 3075.4 6147.4
4 4 28.7 52.7 100.7 196.7 388.7 772.7 1540.7 3076.7 6148.7

Compressor-decompressor verification application

The file CFLEX_Verify will compress and decompress multiple files using all valid compression configurations for LC, LP, and PB. The command line options are similar to the sample full-feature application and allow setting the window sizes and match lengths.

CFLEX_Verify.c complete listing

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Verify.c
Purpose     : Run compress-decompress cycle and validate output.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_ENCODE.h"
#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*********************************************************************
*
*       Defines, configurable
*
**********************************************************************
*/

#define BUFFER_SIZE         (101*1024*1024)  // 101 MB

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static U8                           _aFileBuf   [BUFFER_SIZE];
static U8                           _aEncodedBuf[BUFFER_SIZE];
static U8                           _aDecodedBuf[BUFFER_SIZE];

/*********************************************************************
*
*       Static code
*
**********************************************************************
*/

/*********************************************************************
*
*       _GetStatusText()
*  
*  Function description
*    Decode emCompress-LZMA status code.
*
*  Parameters
*    Status - Status code.
*
*  Return value
*    Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
  switch (Status) {
  case 0:                                    return "Still processing";
  case 1:                                    return "Complete";
  case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
  case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
  case CFLEX_STATUS_USAGE_ERROR:     return "Usage error";
  default:                                   return "Unknown error";
  }
}

/*********************************************************************
*
*       _Verify()
*  
*  Function description
*    Verify a compress-decompress cycle works.
*
*  Parameters
*    pParas  - Pointer to encoder parameters.
*    pInFile - Pointer to input file at position 0.
*
*  Return value
*    0 - test passed.
*    1 - test failed.
*/
static int _Verify(const COMPRESS_LZMA_PARAS *pParas, FILE *pInFile) {
  COMPRESS_LZMA_STREAM EncoderStream;
  COMPRESS_LZMA_STREAM DecoderStream;
  int                  EncoderStatus;
  int                  DecoderStatus;
  U64                  OriginalSize;
  U64                  EncodedSize;
  U64                  DecodedSize;
  int                  Res;
  //
  Res = 0;
  printf("LC=%d, LP=%d, PB=%d: ", pParas->LC, pParas->LP, pParas->PB);
  //
  EncoderStatus = CFLEX_LZMA_ENCODE_Init(&_Encoder, pParas);
  if (EncoderStatus < 0) {
    printf("Skipped, encoder will not initialize\n");
    return 1;
  }
  CFLEX_LZMA_DECODE_Init(&_Decoder);
  //
  EncodedSize   = 0;
  DecodedSize   = 0;
  EncoderStatus = 0;
  DecoderStatus = 0;
  //
  EncoderStream.pIn      = _aFileBuf;
  EncoderStream.AvailIn  = fread(_aFileBuf, 1, sizeof(_aFileBuf), pInFile);
  EncoderStream.pOut     = &_aEncodedBuf[13];
  EncoderStream.AvailOut = sizeof(_aEncodedBuf) - 13;
  OriginalSize           = EncoderStream.AvailIn;
  //
  EncoderStatus = CFLEX_LZMA_ENCODE_Run(&_Encoder, &EncoderStream, 1);
  if (EncoderStatus <= 0) {
    printf("Encoder error: %s\n", _GetStatusText(EncoderStatus));
    return 1;
  }
  EncodedSize = sizeof(_aEncodedBuf) - 13 - EncoderStream.AvailOut;
  CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &_aEncodedBuf[0]);
  //
  DecoderStream.pIn      = &_aEncodedBuf[0];
  DecoderStream.AvailIn  = sizeof(_aEncodedBuf) - EncoderStream.AvailOut + 13;
  DecoderStream.pOut     = &_aDecodedBuf[0];
  DecoderStream.AvailOut = sizeof(_aDecodedBuf);
  //
  DecoderStatus = CFLEX_LZMA_DECODE_Run(&_Decoder, &DecoderStream, 1);
  if (DecoderStatus <= 0) {
    printf("Decoder error: %s\n", _GetStatusText(DecoderStatus));
    Res = 1;
  }
  DecodedSize = sizeof(_aDecodedBuf) - DecoderStream.AvailOut;
  //
  if (OriginalSize != DecodedSize) {
    printf("Compress-decompress size mismatch\n");
    Res = 1;
  } else if (memcmp(_aFileBuf, _aDecodedBuf, (unsigned)DecodedSize) != 0) {
    printf("Compress-decompress content mismatch\n");
    Res = 1;
  } else {
    printf("OK   %5.2fx   %lld -> %lld -> %lld\n", (float)DecodedSize / EncodedSize, OriginalSize, EncodedSize, DecodedSize);
  }
  return Res;
}

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_PARAS    Paras;
  FILE                 * pInFile;
  const char           * sArg;
  int                    i;
  unsigned               MaxLP;
  unsigned               MaxLC;
  unsigned               MaxPB;
  int                    Result;
  //
  Paras.WindowSize = COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX;
  Paras.MinLen     = 3;
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;
  MaxLC            = COMPRESS_LZMA_CONFIG_ENCODE_LC_MAX;
  MaxLP            = COMPRESS_LZMA_CONFIG_ENCODE_LP_MAX;
  MaxPB            = COMPRESS_LZMA_CONFIG_ENCODE_PB_MAX;
  //
  for (i = 1; i < argc; ++i) {
    sArg = argv[i];
    if (strncmp(sArg, "-lc=", 4) == 0) {
      MaxLC = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLC);
    } else if (strncmp(sArg, "-lp=", 4) == 0) {
      MaxLP = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxLP);
    } else if (strncmp(sArg, "-pb=", 4) == 0) {
      MaxPB = CFLEX_MIN((unsigned)atoi(&sArg[4]), MaxPB);
    } else if (strncmp(sArg, "-ws=", 4) == 0) {
      Paras.WindowSize = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-O", 2) == 0) {
      Paras.Optimize = atoi(&sArg[2]);
    } else if (strncmp(sArg, "-k", 2) == 0) {
      Paras.LC = 3;
      Paras.LP = 0;
      Paras.PB = 2;
    } else if (strncmp(sArg, "-minlen=", 8) == 0) {
      Paras.MinLen = atoi(&sArg[8]);
    } else if (strncmp(sArg, "-maxlen=", 8) == 0) {
      Paras.MaxLen = atoi(&sArg[8]);
    } else if (sArg[0] == '-') {
      printf("%s: unknown option %s\n", argv[0], sArg);
      exit(100);
    }
  }
  //
  Result = 0;
  for (i = 1; i < argc; ++i) {
    sArg = argv[i];
    if (sArg[0] != '-') {
      pInFile = fopen(sArg, "rb");
      if (pInFile == 0) {
        printf("%s: can't open %s for reading\n", argv[0], argv[1]);
        exit(100);
      }
      //
      printf("\nVerification of %s:\n\n", sArg);
      //
      for (Paras.LC = 0; Paras.LC <= MaxLC; ++Paras.LC) {
        for (Paras.LP = 0; Paras.LP <= MaxLP; ++Paras.LP) {
          for (Paras.PB = 0; Paras.PB <= MaxPB; ++Paras.PB) {
            fseek(pInFile, 0, SEEK_SET);
            Result |= _Verify(&Paras, pInFile);
          }
        }
      }
      fclose(pInFile);
    }
  }
  if (Result) {
    printf("some tests *** FAILED ***\n");
  }
}

/*************************** End of file ****************************/

Sample full-feature application

emCompress-Flex ships with an application, in source form, which compresses files into LZMA-alone format. These files can be fed into the emCompress-Flex decompressor, byte for byte, and be decompressed on target hardware.

Command line and options

The emCompress-Flex example compression application accepts the command line options described in the following sections.

The command line syntax is:

CFLEX_Util [options] inputfile [outputfile]

The output file is optional: if given, the compressed LZMA bitstream is written to it.

The example application displays some information relating to the compression process and selected parameters.

Example

C:> CFLEX_Util.exe -O9 -ws=1024 ptt5

(c) 2016-2019 SEGGER Microcontroller GmbH    www.segger.com
emCompress-FLEX V3.10 compiled Feb 05 2019 12:09:19

Encoding parameters:
  Coding:         LC=0, LP=0, PB=0
  Window size:    1024 bytes
  Match length:   3 to 273 bytes
  Match distance: 1 to 1024 bytes

Coder performance:
  Max. length:    273 bytes
  Max. distance:  750 bytes

Statistics:
  Decoded size:   513216 bytes
  Encoded size:   49403 bytes (including header)
  Ratio:          10.39x
  Space savings:  90.38% (of original)
  Standardized:   0.77 bits/byte

C:> _

Set literal context bits (-lc)

Syntax

-lc=number

Description

Set the number of literal context bits to use when compressing. Acceptable values are 0 through 8 inclusive.

The default is zero.

Set literal position bits (-lp)

Syntax

-lp=number

Description

Set the number of literal position bits to use when compressing. Acceptable values are 0 through 4 inclusive.

The default is zero.

Set literal position bits (-pb)

Syntax

-pb=number

Description

Set the number of position bits to use when compressing. Acceptable values are 0 through 4 inclusive.

The default is zero.

Set window size (-ws)

-ws=number

Description

Set the window size to when compressing. The window size is the naming convention used by emCompress and emCompress-LZMA to describe the maximum match distance and, therefore, the maximum number of octets that must be stored to satisfy references made by the decompressor. The LZMA SDK refers to this as the “dictionary size” and both terms should be considered equivalent.

Increasing the window size will usually increase compression ratios and reduce the size of the compressed bitstream at the expense of requiring extra RAM during decompression.

The default is 131,072.

Set minimum match length (-minlen)

-minlen=number

Description

Set the minimum match length when compressing. Acceptable values are 2 through 273, with the default being 3. We highly recommend that this value is left unchanged.

Although the LZMA compression scheme supports match lengths of two, this usually leads to worse compressor performance in terms of both speed and compressed bitstream size.

Set maximum match length (-maxlen)

Syntax

-maxlen=number

Description

Set the maximum match length when compressing. Acceptable values are 2 through 273, with the default being 273. We highly recommend that this value is left unchanged.

The window size and the maximum match length are not independent. If the window size is reduced, emCompress-LZMA will also reduce the maximum match length so that it does not exceed the capacity of the window to store the matched sequence. The emCompress-LZMA example application does this outside of the emCompress-LZMA DLL as the DLL will diagnose incorrect compression parameters with an error.

Set block input size (-bi)

Syntax

-bi=number

Description

Set the block input size, the maximum amount of data sent to the compressor in a single call. Acceptable values are 1 through 65536, with the default being 65536.

This has no effect on the compressed output and is present to enable the compressor to be tested, if necessary, with a known input and known block sizes.

Set block input size (-bo)

Syntax

-bo=number

Description

Set the block output size, the maximum amount of output space available to the compressor in a single call. Acceptable values are 1 through 65536, with the default being 65536.

This has no effect on the compressed output and is present to enable the compressor to be tested, if necessary, with a known input and known block sizes.

Set optimization level (-O)

Syntax

-Onumber

Description

Set the optimization level to use when compression. Acceptable values are 0 (faster, good compression) to 9 (slower, best compression).

Utility listing

/*********************************************************************
*                   (c) SEGGER Microcontroller GmbH                  *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------/*********************************************************************
*               (c) SEGGER Microcontroller GmbH & Co. KG             *
*                        The Embedded Experts                        *
*                           www.segger.com                           *
**********************************************************************

-------------------------- END-OF-HEADER -----------------------------

File        : CFLEX_Util.c
Purpose     : Example emCompress-FLEX application.

*/

/*********************************************************************
*
*       #include section
*
**********************************************************************
*/

#include "CFLEX_ENCODE.h"
#include "CFLEX_DECODE.h"
#include "CFLEX.h"
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*********************************************************************
*
*       Static data
*
**********************************************************************
*/

static COMPRESS_LZMA_ENCODE_CONTEXT _Encoder;
static COMPRESS_LZMA_DECODE_CONTEXT _Decoder;
static unsigned                     _BlockInSize;
static unsigned                     _BlockOutSize;
static U8                           _aInBuffer [64*1024];
static U8                           _aOutBuffer[64*1024];

/*********************************************************************
*
*       Static code
*
**********************************************************************
*/

/*********************************************************************
*
*       _GetStatusText()
*  
*  Function description
*    Decode emCompress-LZMA status code.
*
*  Parameters
*    Status - Status code.
*
*  Return value
*    Non-zero pointer to status description.
*/
static const char * _GetStatusText(int Status) {
  if (Status >= 0) {
    return "OK";
  }
  switch (Status) {
  case CFLEX_STATUS_PARAMETER_ERROR: return "Parameter error";
  case CFLEX_STATUS_BITSTREAM_ERROR: return "Bitstream error";
  case CFLEX_STATUS_USAGE_ERROR:     return "Usage error";
  default:                                   return "Unknown error";
  }
}

/*********************************************************************
*
*       _Encode()
*  
*  Function description
*    Encode a file.
*
*  Parameters
*    pInFile  - Pointer to file to encode.
*    pOutFile - Pointer to file that recieves the encoded bitstream.
*    pParas   - Pointer to LZMA encoding parameters.
*    Verbose  - Flag indicating verbose output.
*/
static void _Encode(FILE *pInFile, FILE *pOutFile, COMPRESS_LZMA_PARAS *pParas, int Verbose) {
  COMPRESS_LZMA_STREAM                Stream;
  int                                 Status;
  int                                 Flush;
  unsigned                            EncodedLen;
  unsigned                            DecodedLen;
  U8                                  aHeader[13];
  //
  // Initialize encoder.
  //
  CFLEX_LZMA_ENCODE_Init(&_Encoder, pParas);
  //
  // Reserve space for an LZMA header in the output file.
  //
  if (pOutFile) {
    fseek(pOutFile, 13, SEEK_SET);
  }
  //
  // Encode file.
  //
  EncodedLen = 0;
  DecodedLen = 0;
  Status     = 0;
  while (Status == 0) {
    //
    // Read next chunk and set up coder buffers.
    //
    Stream.pIn     = _aInBuffer;
    Stream.AvailIn = fread(_aInBuffer, 1, _BlockInSize, pInFile);
    DecodedLen    += Stream.AvailIn;
    //
    // Flush compressor on end of input file.
    //
    Flush = Stream.AvailIn != _BlockInSize;
    //
    // Encode input data, writing encoded data to file.
    //
    while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
      //
      Stream.pOut     = &_aOutBuffer[0];
      Stream.AvailOut = _BlockOutSize;
      //
      Status = CFLEX_LZMA_ENCODE_Run(&_Encoder, &Stream, Flush);
      //
      if (Status >= 0 && pOutFile != NULL) {
        fwrite(_aOutBuffer, 1, _BlockOutSize - Stream.AvailOut, pOutFile);
      }
      EncodedLen += _BlockOutSize - Stream.AvailOut;
    }
  }
  //
  // Rewind file and write correct header.
  //
  if (pOutFile) {
    CFLEX_LZMA_ENCODE_WrHeader(&_Encoder, &aHeader[0]);
    fseek(pOutFile, 0, SEEK_SET);
    fwrite(aHeader, 1, sizeof(aHeader), pOutFile);
  }
  //
  if (Status < 0) {
    fprintf(stderr, "FATAL: %s\n", _GetStatusText(Status));
  } else {
    if (Verbose) {
      fprintf(stderr, "Encoding parameters:\n");
      fprintf(stderr, "  Coding:         LC=%u, LP=%u, PB=%u\n",  _Encoder.LC, _Encoder.LP, _Encoder.PB);
      fprintf(stderr, "  Window size:    %lu bytes\n",            _Encoder.WindowSize);
      fprintf(stderr, "  Match length:   %d to %d bytes\n",       _Encoder.LZSSCoder.MinLength, _Encoder.LZSSCoder.MaxLength);
      fprintf(stderr, "  Match distance: 1 to %d bytes\n",        _Encoder.LZSSCoder.MaxDistance);
      fprintf(stderr, "\n");
      fprintf(stderr, "Coder performance:\n");
      fprintf(stderr, "  Max. length:    %d bytes\n",             _Encoder.LZSSCoder.MaxUsedLength);
      fprintf(stderr, "  Max. distance:  %d bytes\n",             _Encoder.LZSSCoder.MaxUsedDistance);
      fprintf(stderr, "\n");
      fprintf(stderr, "Statistics:\n");
      fprintf(stderr, "  Decoded size:   %u bytes\n",             DecodedLen);
      fprintf(stderr, "  Encoded size:   %u bytes (including header)\n", EncodedLen + 13);
      fprintf(stderr, "  Ratio:          %.2fx\n",                (float)DecodedLen / EncodedLen);
      fprintf(stderr, "  Space savings:  %.2f%% (of original)\n", 100.0f - EncodedLen * 100.0f / DecodedLen);
      fprintf(stderr, "  Standardized:   %.2f bits/byte\n",       EncodedLen * 8.0f / DecodedLen);
    }
  }
}

/*********************************************************************
*
*       _Decode()
*  
*  Function description
*    Decode a file.
*
*  Parameters
*    pInFile  - Pointer to file containing the encoded bitstream.
*    pOutFile - Pointer to file that recieves the decoded bitstream.
*    Verbose  - Flag indicating verbose output.
*/
static void _Decode(FILE *pInFile, FILE *pOutFile, int Verbose) {
  COMPRESS_LZMA_STREAM                Stream;
  int                                 Status;
  int                                 Flush;
  unsigned                            EncodedLen;
  unsigned                            DecodedLen;
  //
  // Initialize decoder.
  //
  CFLEX_LZMA_DECODE_Init(&_Decoder);
  //
  DecodedLen = 0;
  EncodedLen = 0;
  Status     = 0;
  while (Status == 0) {
    //
    // Read next chunk and set up coder buffers.
    //
    Stream.pIn     = _aInBuffer;
    Stream.AvailIn = fread(_aInBuffer, 1, _BlockInSize, pInFile);
    EncodedLen    += Stream.AvailIn;
    //
    // Exit on end of file.
    //
    Flush = Stream.AvailIn != _BlockInSize;
    //
    // Decode input data, writing decoded data to file.
    //
    while (Status == 0 && (Stream.AvailIn > 0 || Flush)) {
      //
      Stream.pOut     = &_aOutBuffer[0];
      Stream.AvailOut = _BlockOutSize;
      //
      Status = CFLEX_LZMA_DECODE_Run(&_Decoder, &Stream, Flush);
      if (Status >= 0 && pOutFile) {
        fwrite(_aOutBuffer, 1, _BlockOutSize - Stream.AvailOut, pOutFile);
      }
      DecodedLen += _BlockOutSize - Stream.AvailOut;
    }
  }
  //
  if (Status < 0) {
    fprintf(stderr, "FATAL: %s\n", _GetStatusText(Status));
  } else {
    if (Verbose) {
      fprintf(stderr, "Decoding parameters:\n");
      fprintf(stderr, "  Coding:         LC=%u, LP=%u, PB=%u\n",  _Decoder.LC, _Decoder.LP, _Decoder.PB);
      fprintf(stderr, "  Window size:    %lu bytes\n",            _Decoder.WindowSize);
      fprintf(stderr, "\n");
      fprintf(stderr, "Statistics:\n");
      fprintf(stderr, "  Encoded size:   %u bytes\n",             EncodedLen);
      fprintf(stderr, "  Decoded size:   %u bytes \n",            DecodedLen);
      fprintf(stderr, "  Ratio:          %.2fx\n",                (float)DecodedLen / EncodedLen);
      fprintf(stderr, "  Space savings:  %.2f%% (of original)\n", 100.0f - EncodedLen * 100.0f / DecodedLen);
      fprintf(stderr, "  Standardized:   %.2f bits/byte\n",       EncodedLen * 8.0f / DecodedLen);
    }
  }
}

/*********************************************************************
*
*       Public code
*
**********************************************************************
*/

/*********************************************************************
*
*       main()
*  
*  Function description
*    Application entry point.
*
*  Parameters
*    argc - Argument count.
*    argv - Argument vector.
*/
void main(int argc, char **argv) {
  COMPRESS_LZMA_PARAS Paras;
  int                 i;
  int                 Verbose;
  int                 Action;
  const char        * sInputPathname;
  const char        * sOutputPathname;
  FILE              * pInFile;
  FILE              * pOutFile;
  //
  fprintf(stderr, "\n");
  fprintf(stderr, "(c) 2015, 2016 SEGGER Microcontroller GmbH & Co. KG    www.segger.com\n");
  fprintf(stderr, "emCompress-LZMA V%d.%02d ", CFLEX_VERSION / 10000, CFLEX_VERSION / 100 % 100);
  fprintf(stderr, "compiled " __DATE__ " " __TIME__ "\n");
  fprintf(stderr, "\n");
  //
  Paras.LC         = 0;
  Paras.LP         = 0;
  Paras.PB         = 0;
  Paras.WindowSize = CFLEX_MIN(COMPRESS_LZMA_CONFIG_ENCODE_WINDOW_SIZE_MAX,
                                       COMPRESS_LZMA_CONFIG_DECODE_WINDOW_SIZE_MAX);
  Paras.MinLen     = 3;
  Paras.MaxLen     = 273;
  Paras.Optimize   = 5;
  //
  sInputPathname   = 0;
  sOutputPathname  = 0;
  Action           = -1;  // Compress
  Verbose          = 1;
  _BlockInSize     = sizeof(_aInBuffer);
  _BlockOutSize    = sizeof(_aOutBuffer);
  //
  for (i = 1; i < argc; ++i) {
    const char *sArg = argv[i];
    if (strncmp(sArg, "-lc=", 4) == 0) {
      Paras.LC = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-lp=", 4) == 0) {
      Paras.LP = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-pb=", 4) == 0) {
      Paras.PB = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-ws=", 4) == 0) {
      Paras.WindowSize = atoi(&sArg[4]);
    } else if (strncmp(sArg, "-O", 2) == 0) {
      Paras.Optimize = atoi(&sArg[2]);
    } else if (strncmp(sArg, "-k", 2) == 0) {
      Paras.LC = 3;
      Paras.LP = 0;
      Paras.PB = 2;
    } else if (strncmp(sArg, "-bi=", 4) == 0) {
      _BlockInSize = CFLEX_MAX(1, CFLEX_MIN(atoi(&sArg[4]), sizeof(_aInBuffer)));
    } else if (strncmp(sArg, "-bo=", 4) == 0) {
      _BlockOutSize = CFLEX_MAX(1, CFLEX_MIN(atoi(&sArg[4]), sizeof(_aOutBuffer)));
    } else if (strncmp(sArg, "-minlen=", 8) == 0) {
      Paras.MinLen = atoi(&sArg[8]);
    } else if (strncmp(sArg, "-maxlen=", 8) == 0) {
      Paras.MaxLen = atoi(&sArg[8]);
    } else if (strcmp(sArg, "-d") == 0) {
      Action = +1;
    } else if (strcmp(sArg, "-c") == 0) {
      Action = -1;
    } else if (strcmp(sArg, "-v") == 0) {
      Verbose = 1;
    } else if (strcmp(sArg, "-q") == 0) {
      Verbose = 0;
    } else if (sArg[0] == '-') {
      printf("%s: unknown option %s\n", argv[0], sArg);
      exit(100);
    } else if (sInputPathname == 0) {
      sInputPathname = sArg;
    } else if (sOutputPathname == 0) {
      sOutputPathname = sArg;
    } else {
      printf("%s: too many filenames\n", argv[0]);
      exit(100);
    }
  }
  //
  if (sInputPathname == 0) {
    printf("%s: require input filename\n", argv[0]);
    exit(100);
  }
  //
  pInFile = fopen(sInputPathname, "rb");
  if (pInFile == 0) {
    printf("%s: can't open %s for reading\n", argv[0], sInputPathname);
    exit(100);
  }
  //
  if (Action < 0 && CFLEX_LZMA_ENCODE_Init(&_Encoder, &Paras) < 0) {
    printf("%s: inconsistent or invalid compression parameters\n", argv[0]);
    exit(100);
  }
  //
  pOutFile = 0;
  if (sOutputPathname != 0) {
    pOutFile = fopen(sOutputPathname, "wb");
    if (pOutFile == 0) {
      printf("%s: can't open %s for writing\n", argv[0], sInputPathname);
      fclose(pInFile);
      exit(100);
    }
  }
  //
  if (Action < 0) {
    _Encode(pInFile, pOutFile, &Paras, Verbose);
  } else {
    _Decode(pInFile, pOutFile, Verbose);
  }
  //
  if (pInFile) {
    fclose(pInFile);
  }
  if (pOutFile) {
    fclose(pOutFile);
  }
}

/*************************** End of file ****************************/

API reference

This section describes the public API for emCompress-Flex. Any functions or data structures that are not described here but are exposed through inclusion of either CFLEX_ENCODE.h or CFLEX_DECODE.h must be considered private and subject to change.

Compressor functions

Function Description
CFLEX_LZMA_ENCODE_Init() Initialize LZMA encoder.
CFLEX_LZMA_ENCODE_Run() Run LZMA encoder.

CFLEX_LZMA_ENCODE_Init()

Description

Initialize LZMA encoder.

Prototype

int CFLEX_LZMA_ENCODE_Init(      COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
                           const COMPRESS_LZMA_PARAS          * pParas);

Parameters

Parameter Description
pSelf Pointer to encoder context.
pParas Pointer to encoder parameters.

Return value

< 0 Error in one or more parameters.
≥ 0 Success.

CFLEX_LZMA_ENCODE_Run()

Description

Run LZMA encoder.

Prototype

int CFLEX_LZMA_ENCODE_Run(COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
                          COMPRESS_LZMA_STREAM         * pStream,
                          int                            Flush);

Parameters

Parameter Description
pSelf Pointer to LZMA encoder context.
pStream Pointer to stream communication object.
Flush End of stream flag.

Return value

≤ 0 Processing error.
= 0 OK, continue encoding.
> 0 Encoding complete.

CFLEX_LZMA_ENCODE_WrHeader()

Description

Write LZMA-alone header.

Prototype

void CFLEX_LZMA_ENCODE_WrHeader(COMPRESS_LZMA_ENCODE_CONTEXT * pSelf,
                                U8                           * pHeader);

Parameters

Parameter Description
pSelf Pointer to LZMA encoder context.
pHeader Pointer to object the receives the 13-octet LZMA-alone header.

Decompressor functions

Function Description
CFLEX_LZMA_DECODE_Init() Initialize LZMA decoder.
CFLEX_LZMA_DECODE_Run() Run LZMA decoder.

CFLEX_LZMA_DECODE_Init()

Description

Initialize LZMA decoder.

Prototype

void CFLEX_LZMA_DECODE_Init(COMPRESS_LZMA_DECODE_CONTEXT * pSelf);

Parameters

Parameter Description
pSelf Pointer to decoder context.

CFLEX_LZMA_DECODE_Run()

Description

Run LZMA decoder.

Prototype

ILEAST16 CFLEX_LZMA_DECODE_Run(COMPRESS_LZMA_DECODE_CONTEXT * pSelf,
                               COMPRESS_LZMA_STREAM         * pStream,
                               ILEAST16                       Flush);

Parameters

Parameter Description
pSelf Pointer to decoder context.
pStream Pointer to stream context.
Flush End of input stream flag.

Return value

< 0 Processing error or bitstream error
= 0 Call again to continue processing
> 0 Decompression complete