Amazing results with compression!

Development
1

So I actually did some tests today comparing Unishox2 and SMAZ. Unishox2 clearly wins for the Meshtastic use case. I am seeing at least 20% to even 45% size reduction. We definitely need to implement this in our system. Less size = more successful transmission.

I am posting some sample tests below, if anyone wants the Python code, I can post that as well. I am going to open a Github Feature Request as well.

GPS Co-ordinates:
Input String: 26.2754584, -80.1055376
Original Size: 23
SMAZ Size: 30
Unishox2 Size: 15
SMAZ Compression Ratio: -0.30434782608695654
Unishox2 Compression Ratio: 0.34782608695652173

34% reduction!

Input String: Requesting emergency backup. Please send supplies quickly.
Original Size: 58
SMAZ Size: 43
Unishox2 Size: 40
SMAZ Compression Ratio: 0.2586206896551724
Unishox2 Compression Ratio: 0.31034482758620685

Input String: hello
Original Size: 5
SMAZ Size: 3
Unishox2 Size: 4
SMAZ Compression Ratio: 0.4
Unishox2 Compression Ratio: 0.19999999999999996

Input String: The quick brown fox jumps over the lazy dog
Original Size: 43
SMAZ Size: 30
Unishox2 Size: 30
SMAZ Compression Ratio: 0.3023255813953488
Unishox2 Compression Ratio: 0.3023255813953488

Should we only compress the message payload, or the entire packet? I think beginning with compressing only the text will be easier. Or perhaps we can compress the payload message with Unishox and the entire packet with gzip?

@geeksville @mc-hamster

2

Python Code:

import unishox2
import smaz

# the string we want to compress
original_data = "The quick brown fox jumps over the lazy dog"


compressed_data, original_size = unishox2.compress(original_data)
compressed = smaz.compress(original_data)
compressed_size = len(compressed)
compression_ratio = 1 - compressed_size / original_size
unishox_compression_ratio = 1 - len(compressed_data) / original_size
print("Input String: ", original_data)
print("Original Size: ", original_size)
print("SMAZ Size: ", compressed_size)
print("Unishox2 Size: ", len(compressed_data))
print("SMAZ Compression Ratio: ", compression_ratio)
print("Unishox2 Compression Ratio: ", unishox_compression_ratio)
3

Let’s compress the entire packet, excluding the header.

There’d be less overhead in doing that since we won’t have to modify anything in the payload, just indicate if it’s compressed in the header and only when it’s compressed.

We should also just pick one compressor. We’re now at >93% used space for firmware on the esp32. Still lots of space on the NRF, but we need to be mindful of what libraries we bring in.

2 Likes
4

wow! that’s really amazing!

5

Unishox2 guarantees compression, so we should always use it.
I wonder what the compiled size of Unishox2 is. I don’t have a C compiler on my laptop and I don’t know if there is a different compiler for the ESP32.

6

Yes, lol. A whole one letter but something is better than nothing.

7

It’s just GCC under the hood.

Follow these instructions to get you setup with a dev environment . It won’t take more than a few minutes:

8

. deleted, because there are already a lot of good ideas regarding compressing.
I’m reading up a lot of older threads right now

10

I tested my theory today. I got a 1.7 KM range through an extremely crowded capital city. At the farthest point, I sent 3 character messages, they went through fine. When I tried to send 30 character messages, only about 50% went through.

When I sent 235 character long messages (which is the limit of the message length), failure was 100%.

Reducing the message size definitely increases the transmission success rate. Implementing compression will 100% improve the range results of all users. We must implement it all costs.

Also, the biggest benefit would be lower channel utilization, lower airtime, lower battery consumption.

@mc-hamster @geeksville @garth

1 Like