Why do meters reject fresh DLMS connections after TCP keep-alive disconnects with frame-counter-out-of-sequence?

marcus_vance

[!WARNING]
Technical Redaction & Safe Harbor Notice:
All communication captures, system titles, and cryptographic parameters in this thread have been sanitized and neutralized. Real utility meter identifiers, production encryption keys (GUEK/GAK/MK), SIM IMSIs, and proprietary network envelopes have been substituted with standard IEC 62056 synthetic test vectors. Do not post unredacted production captures or live cryptographic keys.

We are managing a fleet of 50,000 smart meters communicating over cellular NB-IoT to a centralized Head-End System (HES). When an active TCP connection drops due to cellular network tower handovers or keep-alive timeouts, the meter reconnects to HES and establishes a fresh TCP socket on port 4059.

Upon reconnecting, HES sends an initial AARQ followed by an encrypted Get-Request using Security Suite 1 (0xDB General-Glo-Ciphering).
However, the meter immediately drops the frame or returns action-response-with-service-error: frame-counter-out-of-sequence!

Our HES server maintains a stateful database, but whenever a worker thread restarts, it re-initializes the Invocation Counter ($IC$) to 1. Why does the meter permanently reject connections after socket restarts?

alex_turner

Marcus, this is the core security invariant of DLMS Security Suite 1: The Invocation Counter ($IC$) is persistent across connection lifetimes.

Per DLMS Green Book 10th Ed Section 9.2.5.3 (Invocation Counter Management) and IEC 62056-53 Section 8.4.2:
The 4-byte Invocation Counter ($IC$) is a strictly monotonic anti-replay sequence number.

The Meter's Non-Volatile Persistence:

In the meter's security engine (Class 64 Security Setup), the meter stores the last verified Invocation Counter from Client SAP 0x0001 in non-volatile memory (EEPROM / Flash):
$$\text{Condition: } IC_{\text{rx}} > IC_{\text{persisted}}$$

If your HES worker thread restarts and resets $IC$ to 1 (or any value $\le IC_{\text{persisted}}$), the meter's AES-GCM decryption engine flags an active replay attack and drops the frame by design!

david_miller

How should HES synchronize its Invocation Counter when state is lost?

Per DLMS Blue Book Ed. 17 Section 4.6.2 (Security Setup Class 64):
Class 64 defines Attribute 4: global_invocation_counter (Double-Long-Unsigned).

The Standard Synchronization Handshake:

Connect via Public Client (Client SAP 0x0010):
Public Client requires no encryption and allows reading Public objects;
Read Current Meter IC:
Issue Get-Request-Normal for OBIS 0.0.43.1.0.255 (or associated Security Setup object) Attribute 4;
Synchronize HES Counter:
$$\text{HES } IC = \text{Meter } IC + 100$$ (advance by a safety margin);
Connect via Management Client (Client SAP 0x0001):
Transmit AARQ and ciphered APDUs using the newly synchronized $IC$.

elena_rostova

From the firmware implementation side, we write the latest $IC$ to Flash every $N$ transactions (and cache in battery-backed RAM) with an anti-wear leveling algorithm.

When the meter powers on or re-establishes an association, it loads $IC_{\text{persisted}}$. If a client attempts a brute-force connection with lower $IC$, the meter will lock out the association after 3 failed attempts per Class 65 (Single Port Protection).

sarah_jenkins

Let's look at the OBIS code for the Invocation Counter across different Security Setup instances:

Management Security Setup (Client SAP 1): 0.0.43.1.0.255
Firmware Upgrade Security Setup (Client SAP 2): 0.0.43.1.1.255
Customer Gateway Security Setup (Client SAP 3): 0.0.43.1.2.255

Each logical client has an independent, non-overlapping Invocation Counter tracking sequence!

carlos_mendez

We tested the Public Client pre-read in the field across 200 failing meters:
In every single case, reading Attribute 4 of 0.0.43.1.0.255 showed the meter was at $IC = 14{,}280$, while our restarted HES was trying to send $IC = 1$.
Once we synced HES to $14{,}300$, all 200 meters reconnected instantly without a single drop!

marcus_vance

Here is the verified Public Client read request for the Invocation Counter:

00 01 00 10 00 01 00 0D C0 01 C1 00 40 00 00 2B 01 00 FF 04 00

Client SAP: 0x0010 (Public Client 16)
Class ID: 0x0040 (Class 64 Security Setup)
OBIS: 0.0.43.1.0.255
Attribute: 4 (global_invocation_counter)

And the meter's response returning current $IC = 14{,}280$ (0x000037C8

00 01 00 10 00 01 00 09 C4 01 C1 00 06 00 00 37 C8

alex_turner

Notice how the Public Client read solves the cold-start problem cleanly without exposing any secret keys!

elena_rostova

Always ensure that your HES advances the counter by a small margin (+10 to +100) before transmitting the first ciphered APDU to absorb any unacknowledged frames in transit.

marcus_vance

To synthesize for the community, here is the complete Python Gurux script demonstrating automated $IC$ synchronization:

from gurux_dlms.objects import GXDLMSSecuritySetup
from gurux_dlms.enums import Security, Authentication

def synchronize_invocation_counter(reader, client, security_obis="0.0.43.1.0.255"):
    # 1. Connect temporarily as Public Client (SAP 16)
    client.clientAddress = 16
    client.authentication = Authentication.NONE
    client.ciphering.security = Security.NONE
    
    # 2. Read Global Invocation Counter (Class 64, Attribute 4)
    ss = GXDLMSSecuritySetup(security_obis)
    current_ic = reader.read(ss, 4)
    print(f"[+] Current Meter Persisted Invocation Counter: {current_ic}")
    
    # 3. Synchronize Management Client with Safety Advance (+50)
    client.clientAddress = 1 # Management Client SAP 1
    client.authentication = Authentication.HIGH_GMAC
    client.ciphering.security = Security.AUTHENTICATION_ENCRYPTION
    client.ciphering.invocation_counter = current_ic + 50
    
    print(f"[+] Management Client IC synchronized to {client.ciphering.invocation_counter}. Ready for AARQ.")

Summary Checklist:

$IC$ Persistence: Invocation Counters survive reboots and socket disconnects (Green Book Section 9.2.5.3).
Cold-Start Sync: Read Attribute 4 of Class 64 via Public Client (SAP 16).
Safety Advance: Set $IC = IC_{\text{meter}} + 50$ to avoid race conditions.