How to correctly implement HDLC frame fragmentation, CRC-16 FCS validation, and AES-GCM deciphering pipeline in DLMS Green Book?

david_miller

[!WARNING]
Technical Redaction & Safe Harbor Notice:
All communication captures, system titles, and cryptographic parameters in this thread have been sanitized and neutralized. Real utility meter identifiers, production encryption keys (GUEK/GAK/MK), SIM IMSIs, and proprietary network envelopes have been substituted with standard IEC 62056 synthetic test vectors. Do not post unredacted production captures or live cryptographic keys.

We are developing a high-performance optical handheld unit (IEC 62056-21 Mode E, 9600 baud) communicating over IEC 62056-46 HDLC with smart meters running Security Suite 1 (General-Glo-Ciphering 0xDB with AES-GCM-128). When reading a large multi-tariff historical billing record (OBIS 1.0.98.1.0.255, Class 7), the meter responds with a 320-byte encrypted APDU split across 3 HDLC segmented I-Frames.

Our embedded parser continuously throws deciphering-failed: invalid-tag or corrupt ASN.1 structure on the continuation segments.

Here is the sanitized 3-segment HDLC capture from the optical probe:

Segment 1 (S=1, Frame Length 132B):

7E A8 84 03 21 10 94 E4 E6 E6 00 DB 08 53 48 41 00 00 00 00 01 30 00 00 00 0A 4F 12 A9 88 BC 11 02 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD 1E 2B 7E

Segment 2 (S=1, Frame Length 128B):

7E A8 80 03 21 30 B2 14 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD 4F 9A 7E

Segment 3 (S=0, Last Segment, Frame Length 66B):

7E A0 42 03 21 50 D8 70 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF 00 11 22 33 44 55 66 77 9B E1 4C 8D 23 AA 12 90 77 1B C0 44 C2 8F 7E

We initially attempted to feed each segment directly into the AES-GCM stream decryptor as it arrived over UART to save RAM. Why is GMAC verification failing completely?

elena_rostova

David, your parser is failing because you are attempting to decrypt across a protocol abstraction layer that cannot be streamed incrementally.

Per DLMS Green Book 10th Ed Section 8.2.2 (HDLC Frame Segmentation) and IEC 62056-46 Section 6.4.4.4.3, HDLC segmentation operates strictly in the Data Link Layer (MAC Sublayer), completely independent of the Application Layer (xDLMS) ciphering.

Look at the HDLC Frame Format field (Bytes 2-3) of your frames:

Segment 1: A8 84 => 1010 1000 1000 0100b. Bit 11 is the Segmentation bit (S-bit). Here S = 1 (segmented frame), length = 132 bytes.
Segment 2: A8 80 => 1010 1000 1000 0000b. S = 1 (continuation segment), length = 128 bytes.
Segment 3: A0 42 => 1010 0000 0100 0010b. S = 0 (final segment), length = 66 bytes.

Notice the LLC sublayer header (E6 E6 00
In HDLC Format Type 3, LLC (0xE6E600) is transmitted ONLY ONCE at the beginning of the information field of the first segment (Segment 1). In continuation segments (Segments 2 and 3), the information field contains pure raw continuation payload bytes directly following the 1-byte Control field and 2-byte HCS!

If your parser expects E6 E6 00 or Application Layer headers on Segment 2, it misaligns the ciphertext stream by 3 bytes, corrupting the AES-GCM input block alignment!

alex_turner

Elena is spot on regarding the LLC framing, but there is an even more fundamental cryptographic reason why on-the-fly streaming decryption is mathematically impossible here:

Per DLMS Green Book 10th Ed Section 9.2.5.3 (Security Suite 1 Cryptographic Processes) and NIST SP 800-38D (GCM Specification), General-Glo-Ciphering (0xDB) applies AES-GCM authenticated encryption to the entire monolithic APDU:

$$\text{Ciphertext Stream} = \text{AES-CTR}{K}(\text{IV}, \text{Plaintext APDU})$$
$$\text{GMAC Tag (12 Bytes)} = \text{GHASH}{H}(\text{AAD} \parallel \text{Ciphertext} \parallel \text{Lengths})$$

Look at Segment 3: the 12-byte GMAC authentication tag (9B E1 4C 8D 23 AA 12 90 77 1B C0 44) is located at the very end of the final segment.

In Galois/Counter Mode (GCM), authentication is not an incremental hash like HMAC; the GHASH polynomial multiplication over $GF(2^{128})$ requires the complete concatenated ciphertext before tag verification can pass.

Golden Rule of DLMS Decryption Order:

Link Layer: Verify each HDLC segment's CRC-16 FCS independently;
Reassembly: Concatenate the payload bytes from Segment 1, 2, and 3 into a single contiguous buffer;
Crypto Layer: Feed the complete 320-byte payload into AES-GCM-128 for GMAC tag verification and deciphering.

sarah_jenkins

To build on Alex's explanation, we must clearly distinguish between two different fragmentation paradigms in DLMS/COSEM:

Per DLMS Green Book 10th Ed Section 9.2.5.4 (Ciphering with DataBlock-G) and DLMS Blue Book Ed. 17 Section 4.3.7:

Paradigm	Layer	Protocol Mechanism	Ciphering Unit
A. HDLC Segmentation	Data Link Layer (MAC)	HDLC S-Bit (`0xA8..` / `0xA0..`), CRC-16 HCS/FCS	1 Single Monolithic Ciphered APDU (`0xDB`) with 1 Invocation Counter and 1 GMAC Tag
B. DataBlock-G Transfer	Application Layer (xDLMS)	`Get-Response-With-Datablock` (`0xC4 0x02`), Block numbers	Multiple Independent APDUs, each carrying its own Invocation Counter and separate GMAC Tag

In David's scenario, the meter is using Paradigm A (Link-Layer HDLC Segmentation) over the optical probe. Therefore, the application layer is completely unaware of the 3 HDLC chunks; it expects a single 0xDB APDU with 1 Invocation Counter (0x0000000A) and 1 GMAC tag at the end.

marcus_vance

Let's formalize the exact multi-stage Link-Layer parsing and CRC-16 state machine.

Per IEC 62056-46 Section 6.4.3 (Frame Check Sequence - FCS) and Section 6.4.4 (Frame Format Type 3), every HDLC frame undergoes a two-tier CRC-16 check using the CCITT polynomial $x^{16} + x^{12} + x^5 + 1$:

Step 1: HCS (Header Check Sequence) Validation

The 2-byte HCS validates the HDLC header (Frame format, Destination address, Source address, Control byte).
$$\text{CRC-16 Calculation: } \text{bytes}[1 \dots 6] \implies \text{Expected: } \text{HCS bytes}$$
If HCS fails, drop the frame immediately without sending link-level acknowledgment.

Step 2: FCS (Frame Check Sequence) Validation

The 2-byte FCS validates the entire frame from Byte 1 up to the last byte before the closing flag 0x7E.
$$\text{FCS Residual Constant: } \text{CRC-16}(\text{bytes}[1 \dots N-1]) == \text{0xF0B8}$$

Step 3: Link Assembly State Machine

Segment 0 (First Segment, S = 1):
- Verify HCS & FCS.
- Verify LLC sublayer == 0xE6 0xE6 0x00.
- Slice payload starting from index +3 (after LLC) to $N-2$ (before FCS).
- Store in reassembly buffer.
Continuation Segment (S = 1):
- Verify HCS & FCS.
- Do NOT look for LLC. Slice payload immediately from index +1 (after HCS) to $N-2$.
- Append to reassembly buffer.
Final Segment (S = 0):
- Verify HCS & FCS.
- Append payload up to $N-2$.
- Pass entire reassembly buffer to the Security Decryption Layer!

david_miller

Marcus and Elena, that explains everything! In my initial optical parser, my state machine was executing:

// Flawed parser logic in continuation frames:
if (hdlc_frame->is_segmented) {
    uint8_t *payload = skip_llc_header(hdlc_frame->info); // Incorrectly assumed LLC on every frame!
}

Because Segment 2 did not contain E6 E6 00, my parser was skipping the first 3 bytes of raw ciphertext (11 22 33), corrupting the block sequence for AES-GCM!

Once I removed the LLC skipping on continuation frames (S = 1 and S = 0), the concatenated 3-segment reassembly buffer became exactly 194 bytes of ciphertext payload.

Let's trace the exact concatenated APDU buffer:

DB 08 53 48 41 00 00 00 00 01 30 00 00 00 0A 4F 12 A9 88 BC 11 02 33 44 55 66 77 88 99 AA BB CC DD EE FF ... 9B E1 4C 8D 23 AA 12 90 77 1B C0 44

alex_turner

Now that we have the complete, pristine reassembled APDU, here is the exact cryptographic deciphering pipeline:

Per DLMS Green Book 10th Ed Section 9.2.5.3 (Security Suite 1):

1. Extract Cryptographic Components:

APDU Tag: 0xDB (General-Glo-Ciphering)
System Title Length: 0x08
System Title: 53 48 41 00 00 00 00 01 (SHA00000001)
Security Control (SC): 0x30 (Global Unicast Key, Authenticated + Encrypted)
Invocation Counter (IC): 0x0000000A (10)
Ciphertext: Bytes 16 to $N-12$
GMAC Authentication Tag: Last 12 bytes (9B E1 4C 8D 23 AA 12 90 77 1B C0 44)

2. Formulate AEAD Parameters:

$$\text{IV}{12\text{ bytes}} = \text{SystemTitle} \parallel \text{InvocationCounter} = \text{53484100000000010000000A}$$
$$\text{AAD}{1\text{ byte}} = \text{SecurityControl} = \text{0x30}$$
$$\text{Key} = \text{Global Encryption Key (GUEK: 16 bytes)}$$

3. Decrypt & Verify Tag:

Execute AES-128-GCM decipher on Ciphertext with IV, AAD, and Tag. If GMAC verification succeeds, it returns the decrypted plain COSEM APDU.

elena_rostova

For resource-constrained embedded systems (e.g. Cortex-M0/M4 meters with limited 16 KB SRAM), allocating large 4 KB dynamic reassembly buffers via malloc causes memory fragmentation.

In firmware architecture, the recommended pattern is a Static Segment Descriptor Ring:

typedef struct {
    uint8_t buffer[1024];  // Static max PDU reassembly arena
    uint16_t current_len;
    uint8_t expected_frame_seq;
    bool in_progress;
} hdlc_reassembly_ctx_t;

int hdlc_process_segment(hdlc_reassembly_ctx_t *ctx, const uint8_t *frame, uint16_t frame_len) {
    if (!verify_hdlc_hcs(frame) || !verify_hdlc_fcs(frame, frame_len)) {
        return -1; // CRC failure, discard
    }
    bool is_segmented = (frame[1] & 0x08) != 0;
    uint16_t header_len = get_hdlc_header_len(frame);
    
    // First segment contains LLC (3 bytes)
    if (!ctx->in_progress) {
        if (frame[header_len] != 0xE6 || frame[header_len+1] != 0xE6) return -2;
        header_len += 3; // Strip LLC
        ctx->current_len = 0;
        ctx->in_progress = true;
    }
    
    uint16_t payload_len = frame_len - header_len - 2; // Subtract FCS
    memcpy(&ctx->buffer[ctx->current_len], &frame[header_len], payload_len);
    ctx->current_len += payload_len;
    
    if (!is_segmented) {
        ctx->in_progress = false;
        return ctx->current_len; // Reassembly complete, return total APDU length
    }
    return 0; // Waiting for more segments
}

This zero-allocation approach eliminates heap leaks while maintaining strict compliance with IEC 62056-46.

sarah_jenkins

Once the plain APDU is decrypted from the 194-byte reassembled buffer, here is the resulting plain COSEM response:

C4 01 C1 00 02 03 09 06 01 00 01 08 00 FF 02 02 0F 00 16 1E 09 0C 07 E8 08 17 06 00 00 00 00 80 00 00

Dissecting the plain A-XDR AST:

C4 01: Get-Response-Normal
C1: Invoke-ID 1 (Priority Normal)
00: data-access-result: success
02 03: Structure with 3 elements (Extended Register Attribute 2, 3, 5):
- 09 06 01 00 01 08 00 FF: Logical Name (1.0.1.8.0.255 Active Energy Total)
- 02 02 0F 00 16 1E: Scaler-Unit (scaler = 0, unit = 30 [Wh])
- 09 0C 07 E8 ...: Capture Time timestamp

Everything decodes cleanly according to DLMS Blue Book Ed. 17 Section 4.3.3 (Extended Register).

david_miller

To synthesize this for the community, here is the complete end-to-end Python pipeline demonstrating:

HDLC CRC-16 FCS checking and segment de-encapsulation;
Contiguous APDU reassembly;
AES-GCM-128 GMAC verification and decryption;
Plain COSEM APDU parsing.

import crcmod
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

# 1. Initialize CCITT CRC-16 (IEC 62056-46 polynomial: x^16 + x^12 + x^5 + 1)
crc16 = crcmod.mkCrcFun(0x11021, initCrc=0xFFFF, rev=True, xorOut=0xFFFF)

def parse_and_decrypt_hdlc_stream(segments, guek_key):
    reassembly_buffer = bytearray()
    
    for idx, frame in enumerate(segments):
        # Step A: Validate FCS
        assert crc16(frame[1:-3]) == int.from_bytes(frame[-3:-1], 'little'), "FCS Mismatch!"
        
        frame_format = int.from_bytes(frame[1:3], 'big')
        is_segmented = bool(frame_format & 0x0800)
        
        # Step B: Extract payload bytes (strip Header and FCS)
        header_len = 7 # 1B flag + 2B format + 2B dest + 1B src + 1B ctrl + 2B HCS
        if idx == 0:
            assert frame[header_len:header_len+3] == b'\xe6\xe6\x00', "Missing LLC"
            header_len += 3 # Strip LLC only on first frame
            
        payload_chunk = frame[header_len:-3]
        reassembly_buffer.extend(payload_chunk)
    
    # Step C: Decrypt Reassembled General-Glo-Ciphering APDU
    apdu = bytes(reassembly_buffer)
    assert apdu[0] == 0xDB, "Not General-Glo-Ciphering"
    
    sys_title = apdu[2:10] # 8 bytes
    sc = apdu[10:11]       # 1 byte (0x30)
    ic = apdu[11:15]       # 4 bytes
    ciphertext_and_tag = apdu[15:] # Ciphertext + 12B GMAC tag
    
    iv = sys_title + ic # 12 bytes IV
    aad = sc            # 1 byte AAD
    
    aesgcm = AESGCM(guek_key)
    decrypted_apdu = aesgcm.decrypt(iv, ciphertext_and_tag, aad)
    
    print(f"[+] Decryption successful! Decrypted Plain APDU: {decrypted_apdu.hex().upper()}")
    return decrypted_apdu

Summary Golden Rules:

Never skip LLC on continuation frames: Only Segment 1 contains 0xE6E600.
Never decrypt HDLC segments individually: Reassemble all fragments into a single contiguous buffer before feeding into AES-GCM.
Verify CRC-16 FCS per frame: Drop corrupted frames at the link layer before touching crypto engines.
Enforce IV Construction: $\text{IV} = \text{SystemTitle} \parallel \text{InvocationCounter}$.