SREP is a useful command line utility that supports dictionaries larger than RAM available and helps you to process files that are 20x larger than the RAM size.
Default settings (-l512) allows to process files that are 20x larger than RAM size. Memory requirements are proportional to 1/L, so by increasing -l value it's possible to process even larger files.
When compressing, memory usage for hash: filesize/L*20 + filesize/L*4 + roundup(filesize/L*4 * 5/4) + roundup(filesize/(L*8) * 4) where roundup() rounds up to the next power of 2 and first memory block (i.e. *20) used only in -m1 mode. So, overall, hash includes 3 or 4 chunks. Besides hash, compression uses 3*8mb buffers.
Decompression uses only 2*8mb buffers and no hash. Repeated data copied directly from output file, though, so you need to have enough RAM used for disk cache in order to make decompression fast.
What's New in This Release:
· -m2 -lN now is the same as -m3 -lN -cN: compression ratio is average between -m1 and -m3, while speed is the same as in old versions
· -a0: the same compresssion ratio as -a1, memory usage is smaller by 5-10%, but 1.5-2x slower
· -a32/-a64: sometimes faster than -a16 (only with large pages), but needs even more memory
· -slp[+/-/]: force/disable/try(default) large pages support
Minor changes:
· -v[0..2]: verbosity level
· -pcMAX_OFFSET: print performance counters for matches closer than MAX_OFFSE
· -l64k/-c1mb syntax support (k/m/kb/mb suffixes for kilobytes/megabytes)
· Both 32-bit and 64-bit default executables are compiled with GCC 4.7
· 32/64-bit dynamic/static linux builds