EZSP stability regression on 1.28.0

[Xe138]

What happened?

I am using a Sonoff ZBBridge (EZSP) running Tasmota: https://sonoff.tech/product/smart-home-security/zbbridge/

This device has been stable for about 18 months. On 1.28.0 I lose connectivity to all of my routers (Ikea Tradfri routers) after about 2 hours of operation. Restarting Z2M typically restores connectivity, but it is invariably lost again after some time. Restoring to 1.27.2 fixes the issue.

The following error appears in the log once connectivity is lost:

Error: ZdoRequest error at Driver.zdoRequest (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/driver.ts:551:19) at request (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:270:32) at Object.func (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:295:28) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

What did you expect to happen?

Stable Zigbee network connectivity.

How to reproduce it (minimal and precise)

Install 1.28.0 on a network using a Sonoff ZBBridge as a coordinator.

Zigbee2MQTT version

1.28.0

Adapter firmware version

Tasmota 12.1.1

Adapter

Sonoff ZBBridge

Debug log

Error: ZdoRequest error at Driver.zdoRequest (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/driver.ts:551:19) at request (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:270:32) at Object.func (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:295:28) at Queue.executeNext (/app/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

[kirovilya]

Please record debug herdsman log https://www.zigbee2mqtt.io/guide/usage/debug.html#zigbee-herdsman-debug-logging
I need to see in which parameters this error occurs.

[MattWestb]

@kirovilya I think with the update using hashed TC-Link key is making problems with old saved TC-Link keys that is "real random link keys" in the chip key storage.

I think it needed dropping the network and forming one new network for getting it working OK with all devices (need repairing all devices).

Have you trying if its working updating one network with old key policy with some (not need mans only 3-4 ) routers and see if its working starting using hashed link key without problems ?

ZHA / bellows its possible deleting all keys in the storage with EZSP native commands but the device need getting the new hashed link key for working OK in the network.

[kirovilya]

@MattWestb @Xe138 using a hashed TC-Link key only affects the newly created network (when changing network params). I don't change keys. if the network already exists, then nothing changes - it should work as before.
so I really want to see the start herdsman log

[kirovilya]

@Xe138 try the current dev/edge version of z2m

[Xe138]

@Xe138 try the current dev/edge version of z2m

I updated to 1.28.2-dev commit: 499b993. So far no issue. I'll let this run for about 24hrs and report back.

[Xe138]

@Xe138 try the current dev/edge version of z2m

I updated to 1.28.2-dev commit: 499b993. So far no issue. I'll let this run for about 24hrs and report back.

Still working fine. Issue seems to be resolved on this build. Thank you!

[Xe138]

So I don't know if this is related to the same issue, but Z2M has crashed twice now after a couple days running each time. Here's the error in the logs:

Zigbee2MQTT:info  2022-11-18 13:50:26: MQTT publish: topic 'zigbee2mqtt/Entry Closet Door', payload '{"battery":100,"contact":true,"device_temperature":35,"last_seen":"2022-11-18T18:50:26.260Z","linkquality":180,"power_outage_count":62,"voltage":3005}'
11/18/2022 1:50:34 PM
Zigbee2MQTT:info  2022-11-18 13:50:34: MQTT publish: topic 'zigbee2mqtt/Bedroom Router', payload '{"last_seen":"2022-11-18T18:50:34.975Z","linkquality":144,"update":{"installed_version":"2.3.086","state":"idle"},"update_available":false}'
11/18/2022 1:50:52 PM
Zigbee2MQTT:info  2022-11-18 13:50:52: MQTT publish: topic 'zigbee2mqtt/Entry Door', payload '{"battery":100,"contact":false,"device_temperature":30,"last_seen":"2022-11-18T18:50:52.256Z","linkquality":200,"power_outage_count":1922,"voltage":3045}'
11/18/2022 1:50:56 PM
Zigbee2MQTT:info  2022-11-18 13:50:56: MQTT publish: topic 'zigbee2mqtt/Entry Door', payload '{"battery":100,"contact":true,"device_temperature":30,"last_seen":"2022-11-18T18:50:56.583Z","linkquality":200,"power_outage_count":1922,"voltage":3045}'
11/18/2022 1:50:59 PM
Zigbee2MQTT:info  2022-11-18 13:50:59: MQTT publish: topic 'zigbee2mqtt/Nursery Router', payload '{"last_seen":"2022-11-18T18:50:59.410Z","linkquality":136,"update":{"installed_version":"2.3.086","state":"idle"},"update_available":false}'
11/18/2022 1:51:27 PM
TypeError: reset can only be called on a new connection
11/18/2022 1:51:27 PM
    at SerialDriver.reset (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/uart.ts:294:19)
11/18/2022 1:51:27 PM
    at SerialDriver.onParsed (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/uart.ts:199:22)
11/18/2022 1:51:27 PM
    at Parser.emit (node:events:513:28)
11/18/2022 1:51:27 PM
    at Parser.parseNext (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/parser.ts:37:26)
11/18/2022 1:51:27 PM
    at Parser._transform (/app/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/parser.ts:27:14)
11/18/2022 1:51:27 PM
    at Parser.Transform._write (node:internal/streams/transform:175:8)
11/18/2022 1:51:27 PM
    at writeOrBuffer (node:internal/streams/writable:392:12)
11/18/2022 1:51:27 PM
    at _write (node:internal/streams/writable:333:10)
11/18/2022 1:51:27 PM
    at Parser.Writable.write (node:internal/streams/writable:337:10)
11/18/2022 1:51:27 PM
    at Socket.ondata (node:internal/streams/readable:766:22)

[Hedda]

I am using a Sonoff ZBBridge (EZSP) running Tasmota: https://sonoff.tech/product/smart-home-security/zbbridge/

FYI, should maybe for reference by the way also note the general warning and common recommendation about not using WiFi-based remote Zigbee Coordinator, like Sonoff ZBBridge, as they are by now infamously known for not having a stable connection over Wi-Fi, especially EZSP based Zigbee Coordinator which by design depend on a stable serial connection for communication.

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_sonoff_zbbridge.html

https://www.zigbee2mqtt.io/advanced/remote-adapter/connect_to_a_remote_adapter.html

Home Assistant's ZHA integration has the same warning against using WiFi connected network-attached Zigbee Coordinator:

https://www.home-assistant.io/integrations/zha#warning-about-wi-fi-based-zigbee-to-serial-bridgesgateways

This is not only but mostly due to Silicon Labs EZSP (EmberZNet Serial Protocol) as a serial protocol has not been made to be robust over an unstable connection and instead assumes that got a stable connection over a local serial, and as the EZSP serial protocol is written and maintained as closed source binaries by Silicon Labs there is probably not much that the zigbee-herdsman and Zigbee2MQTT developers can do to prevent lost connections the on their side, so maybe all they can do is make zigbee-herdsman and Zigbee2MQTT better at recovering from lost connections once the connection is already lost?

See the main development discussion about EZSP stability in zigbee-herdsman for how to help -> Koenkk/zigbee-herdsman#319

[dupondje]

Occured here also:

Dec 21 11:56:56 server npm[3446]: Error: ZdoRequest error
Dec 21 11:56:56 server npm[3446]:     at Driver.zdoRequest (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/src/adapter/ezsp/driver/driver.ts:571:19)
Dec 21 11:56:56 server npm[3446]:     at EZSPAdapter.nodeDescriptorInternal (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:373:28)
Dec 21 11:56:56 server npm[3446]:     at Object.func (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/src/adapter/ezsp/adapter/ezspAdapter.ts:363:32)
Dec 21 11:56:56 server npm[3446]:     at Queue.executeNext (/opt/zigbee2mqtt/node_modules/zigbee-herdsman/src/utils/queue.ts:32:32)

This happend on the same moment that my Aqara Temp Sensor disconnected (and had to replace its battery), so might be related?

[Hedda]

Highly recommend migrating to either a Zigbee Coordinator USB adapter or a wired Ethernet-based adapter (such as for example those from TubeZB or ZigStar) and then repurposing the Sonoff ZBBridge as a Zigbee Router device by reflashing the firmware:

#11461

https://community.home-assistant.io/t/inofficial-zigbee-router-firmware-released-for-itead-sonoff-zbbridge-and-zb-gw03-zigbee-gateways-bridges-as-alternative-when-used-with-tasmota-and-or-esphome/393733

[dupondje]

Highly recommend migrating to either a Zigbee Coordinator USB adapter or a wired Ethernet-based adapter (such as for example those from TubeZB or ZigStar) and then repurposing the Sonoff ZBBridge as a Zigbee Router device by reflashing the firmware:

#11461

https://community.home-assistant.io/t/inofficial-zigbee-router-firmware-released-for-itead-sonoff-zbbridge-and-zb-gw03-zigbee-gateways-bridges-as-alternative-when-used-with-tasmota-and-or-esphome/393733

This is a Zigbee USB Adapter :)
SONOFF Zigbee 3.0 USB Dongle Plus V2

[Hedda]

@dupondje Ah ok, (the original report from Xe138 was with a hacked Sonoff ZBBridgebridge over Wi-Fi as Zigbee Coordinator).

Note if now using the latest Zigbee2MQTT dev branch sometimes also referred to as "Zigbee2MQTT Edge" in add-on for HA,
(I do however believe that the latest Zigbee2MQTT master/stable has caught up to recent ezsp adapter additions as of currently).

https://www.zigbee2mqtt.io/advanced/more/switch-to-dev-branch.html

Also note down which exact firmware are using on your SONOFF Zigbee 3.0 USB Dongle Plus V2 (model ZBDongle-E) from ITead.

Tip to follow development discussion about EZSP stability in zigbee-herdsman for reporting -> Koenkk/zigbee-herdsman#319

PS: While the official 6.10.3_V1.0.1 it ships with is compatible you might want to ask if worth flashing to later unofficial firmware.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days