Time-Series Compression Algorithms Compared

Gorilla, Delta-of-Delta, and custom algorithms for IoT telemetry data.

The Storage Challenge

IoT devices generate enormous volumes of time-series data. A single sensor sampling at 1Hz produces 31.5 million data points per year. At 8 bytes per reading, that's 252MB per sensor before any metadata.

Efficient compression is essential for:

• Storage costs: Reduce database footprint by 90%+
• Network bandwidth: Transmit more data over constrained links
• Query performance: Smaller data = faster scans

Algorithm Deep Dive

1. Gorilla Compression (Facebook)

Gorilla uses XOR-based compression for timestamps and values, exploiting the fact that consecutive readings are often similar.

1class GorillaEncoder:
2    def __init__(self):
3        self.prev_timestamp = 0
4        self.prev_delta = 0
5        self.prev_value = 0
6        self.prev_xor = 0
7        
8    def encode_timestamp(self, timestamp):
9        delta = timestamp - self.prev_timestamp
10        delta_of_delta = delta - self.prev_delta
11        
12        if delta_of_delta == 0:
13            self.write_bits(0b0, 1)  # Single bit
14        elif -63 <= delta_of_delta <= 64:
15            self.write_bits(0b10, 2)
16            self.write_bits(delta_of_delta + 64, 7)
17        elif -255 <= delta_of_delta <= 256:
18            self.write_bits(0b110, 3)
19            self.write_bits(delta_of_delta + 256, 9)
20        # ... additional ranges
21        
22        self.prev_delta = delta
23        self.prev_timestamp = timestamp
24    
25    def encode_value(self, value):
26        xor = self.prev_value ^ value
27        if xor == 0:
28            self.write_bits(0b0, 1)
29        else:
30            leading = count_leading_zeros(xor)
31            trailing = count_trailing_zeros(xor)
32            # Encode meaningful bits only
33            self.write_meaningful_bits(xor, leading, trailing)
34        self.prev_value = value

Compression Ratio: 1.37 bytes/point (vs 16 bytes uncompressed)

2. Delta-of-Delta Encoding

Simpler approach that works well for monotonic or slowly-changing values:

1def delta_of_delta_encode(values):
2    result = [values[0]]  # First value uncompressed
3    prev_delta = 0
4    
5    for i in range(1, len(values)):
6        delta = values[i] - values[i-1]
7        dod = delta - prev_delta
8        result.append(zigzag_encode(dod))
9        prev_delta = delta
10    
11    return varint_encode(result)
12
13def zigzag_encode(n):
14    return (n << 1) ^ (n >> 63)  # Map negatives to positives

Compression Ratio: 2.1 bytes/point

3. Dictionary + Run-Length Encoding

For categorical or low-cardinality data:

1class DictionaryEncoder:
2    def __init__(self):
3        self.dictionary = {}
4        self.next_code = 0
5    
6    def encode(self, values):
7        result = []
8        current_code = None
9        run_length = 0
10        
11        for value in values:
12            if value not in self.dictionary:
13                self.dictionary[value] = self.next_code
14                self.next_code += 1
15            
16            code = self.dictionary[value]
17            if code == current_code:
18                run_length += 1
19            else:
20                if current_code is not None:
21                    result.append((current_code, run_length))
22                current_code = code
23                run_length = 1
24        
25        result.append((current_code, run_length))
26        return result

Compression Ratio: 0.3-0.8 bytes/point (for categorical data)

Benchmark Results

Testing with real industrial telemetry (1M data points each):

Choosing the Right Algorithm

┌─────────────────────────────────────────────────────────┐
│                    Decision Tree                        │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Data Type?                                             │
│  ├── Floating Point → Gorilla                           │
│  ├── Integer (monotonic) → Delta-of-Delta               │
│  ├── Categorical → Dictionary + RLE                     │
│  └── Mixed → Hybrid with column detection               │
│                                                         │
│  Query Pattern?                                         │
│  ├── Range scans → Prioritize decode speed              │
│  ├── Point lookups → Index + light compression          │
│  └── Aggregations → Pre-aggregate + compress            │
│                                                         │
└─────────────────────────────────────────────────────────┘

Implementation Recommendations

1. Chunk your data: 64KB-1MB blocks balance compression ratio and random access
2. Compress columns separately: Different algorithms for different data types
3. Keep metadata uncompressed: Enable fast filtering without decompression
4. Use SIMD instructions: Modern CPUs can process 4-8 values simultaneously

The right compression strategy can reduce storage costs by 90% while maintaining query performance.