Admin channel timeouts and unresponsiveness

The new admin channel is very useful, particularly for changing settings on inaccessible or remote devices, but using LongSlow, I am more likely to get a “Timed out waiting for node config” than getting a success. In fact, for me it’s odds of around 1 success out of 50 tries. Is there a way to get this more reliable?

Also, I’ve noticed that there seems to be a pattern in behaviour where if I send a command to a –dest, using the admin channel, but forget to include a –reboot command, the device seems to go offline from the mesh and I can no longer command it to reboot (or do anything else), or I just have no luck after at least one success.

For example:
meshtastic --dest !050c1918 ----set range_test_plugin_enabled 1
if it eventually manages to succeed, the device vanishes and can no longer be reached. If I remember, that should have been
meshtastic --dest !050c1918 ----set range_test_plugin_enabled 1 --reboot
to take effect successfully.

Feature request: Would it be possible to include the ability to send –dest commands to ^all? :eyes:

3 Likes

Hmm. To help figure this out: what is the approximate topology of your mesh? Also could you post what your settings are? (Ie output of meshtastic --info)

We have 8 nodes, but most testing takes place within 4 nodes all within 150m of each other. We use LongSlow as our aim is testing in very hilly, forest terrain.

The admin capability fails consistently despite the actual range, including units within immediate proximity of each other.

The typical response 90-95% of attempts timeout, usually at the first “requesting preferences from remote node”, and sometimes after the second or third.

meshtastic --dest !050c1918 --info --reboot
Connected to radio
INFO:root:Requesting preferences from remote node (this could take a while)
INFO:root:Requesting channel 0 info from remote node (this could take a while)
INFO:root:Requesting channel 1 info from remote node (this could take a while)
Aborting due to: Timed out waiting for node config

My info: { “myNodeNum”: 2988735988, “hasGps”: true, “numBands”: 20, “firmwareVersion”: “1.2.23”, “rebootCount”: 26, “messageTimeoutMsec”: 300000, “minAppVersion”: 20200, “maxChannels”: 8 }

Nodes in mesh:
{‘num’: 2988735988, ‘user’: {‘id’: ‘!b2247df4’, ‘longName’: ‘Avron W 7df4’, ‘shortName’: ‘AW7’, ‘macaddr’: ‘rGeyJH30’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337131701, ‘longitudeI’: 1510676772, ‘altitude’: 155, ‘batteryLevel’: 100, ‘time’: 1618443181, ‘latitude’: -33.7131701, ‘longitude’: 151.0676772}, ‘lastHeard’: 1618443181}
{‘num’: 84685580, ‘user’: {‘id’: ‘!050c330c’, ‘longName’: ‘Node Red 330c’, ‘shortName’: ‘NR3’, ‘macaddr’: ‘PGEFDDMM’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337131020, ‘longitudeI’: 1510676716, ‘altitude’: 178, ‘batteryLevel’: 100, ‘time’: 1618443024, ‘latitude’: -33.713102, ‘longitude’: 151.06767159999998}, ‘lastHeard’: 1618443030, ‘snr’: 10.0}
{‘num’: 2988737884, ‘user’: {‘id’: ‘!b224855c’, ‘longName’: ‘Router 855c’, ‘shortName’: ‘R8’, ‘macaddr’: ‘rGeyJIVc’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337132866, ‘longitudeI’: 1510680400, ‘altitude’: 184, ‘batteryLevel’: 100, ‘time’: 1618442145, ‘latitude’: -33.713286599999996, ‘longitude’: 151.06804}, ‘lastHeard’: 1618443088, ‘snr’: 8.75}
{‘num’: 84678936, ‘user’: {‘id’: ‘!050c1918’, ‘longName’: ‘Ricardo’, ‘shortName’: ‘Rcr’, ‘macaddr’: ‘PGEFDBkY’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337131857, ‘longitudeI’: 1510680761, ‘altitude’: 168, ‘batteryLevel’: 8, ‘time’: 1618265967, ‘latitude’: -33.7131857, ‘longitude’: 151.06807609999998}, ‘lastHeard’: 1618266603, ‘snr’: -14.25}
{‘num’: 2930161508, ‘user’: {‘id’: ‘!aea6b764’, ‘longName’: ‘Robin’, ‘shortName’: ‘Rbn’, ‘macaddr’: ‘TBGuprdk’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337072348, ‘longitudeI’: 1510694786, ‘altitude’: 209, ‘batteryLevel’: 94, ‘time’: 1618268115, ‘latitude’: -33.7072348, ‘longitude’: 151.0694786}, ‘lastHeard’: 1618277615, ‘snr’: -9.5}
{‘num’: 84679136, ‘user’: {‘id’: ‘!050c19e0’, ‘longName’: ‘Avron W 19e0’, ‘shortName’: ‘AW1’, ‘macaddr’: ‘PGEFDBng’, ‘hwModel’: ‘TBEAM’}, ‘position’: {‘latitudeI’: -337150445, ‘longitudeI’: 1510678571, ‘altitude’: -147, ‘batteryLevel’: 95, ‘time’: 1618277605, ‘latitude’: -33.7150445, ‘longitude’: 151.0678571}, ‘lastHeard’: 1618277621, ‘snr’: 9.25}

Preferences: { “waitBluetoothSecs”: 20, “lsSecs”: 300, “region”: “ANZ”, “gpsAttemptTime”: 600 }

Channels:
PRIMARY psk=default { “modemConfig”: “Bw125Cr48Sf4096”, “psk”: “AQ==” }
SECONDARY psk=secret { “psk”: “TjMMMmF5Q1eNdxs23Xe5G4rKZayMQSJxFzwGVsgM08E=”, “name”: “admin” }

Primary channel URL: https://www.meshtastic.org/d/#xxx
Complete URL (includes all channels): https://www.meshtastic.org/d/#xxx

2 Likes

I’ve been seeing this since v1.2.17

I thought I opened a Github issue but I can’t seem to find it now.

1 Like

Just updated to 1.2.28 and it still does it.

Device 1.2.28
Python 1.2.30

After a lot more testing, for me it really seems to be dependent on the distance away from the node.

If I have a node within immediate proximity, I have a much higher chance of success, although still a higher chance it will timeout partway through.

If the node is further away than say 150m, I have almost no chance of getting it to work.

This is testing with 1.2.23 and 1.2.25

I have them on the same desk and haven’t had any luck getting it to work yet.

Without knowing the mechanics of what is going on, I’m guessing that the interaction is a multi-step one, which is why we seldom get through the full exchange.

Despite what mqtt might say about my nodes gps location (when they respect my static settings), they are both sitting within a foot of each other. Thats when I tested the --dest feature so I cant comment on range despite what my nodes look like on a map. I havent ventured out in the wild with these yet because I dont trust them and my cases are junk. Not knocking the project at all, I could invest in some nice cases and load the stable firmware.

Looks like some of you are using ----option ? Whats up with that? Ive only used --info and --reboot and always after --port because meshtastic on the mac isnt smart enough to detect the port.

Note that on my mac I have to escape the !

e@m firmware-1.2.28 % meshtastic --dest !050c217c --port /dev/cu.SLAB_USBtoUART --info
Connected to radio

INFO:root:Requesting preferences from remote node (this could take a while)
INFO:root:Requesting channel 0 info from remote node (this could take a while)
INFO:root:Requesting channel 1 info from remote node (this could take a while)
INFO:root:Requesting channel 2 info from remote node (this could take a while)
Preferences: { “lsSecs”: 300, “region”: “US”, “fixedPosition”: true }

Channels:
PRIMARY psk=default { “modemConfig”: “Bw125Cr48Sf4096”, “psk”: “AQ==” }
SECONDARY psk=secret { “psk”: “obfuscated”, “name”: “admin” }

Primary channel URL: https://www.meshtastic.org/d/#CgUYAyIBAQ
Complete URL (includes all channels): https://www.meshtastic.org/d/#obfuscated

e@m firmware-1.2.28 %

Hmm - that’s puzzling. I’ll try to reproduce here (I haven’t seen this problem, though I don’t doubt it occurred). Your settings look fine. Also, I assume your python client is fairly current?

Yes, running python client 1.2.30.

Another thing that I’ve now encountered that I’ve now been able to reproduce (but it seems to occur randomly), is sending a remote node a command to do something, including the –reboot command. If it works, I get the “node will reboot in 10 seconds” message, but the node vanishes and never re-joins the network. I can’t easily ascertain what state these devices are in as they are remote and kinda inaccessible.

I have seen this same behaviour at times where I have forgotten to include the –reboot command in the same call and had the device vanish, unable to receive the subsequent –reboot command.

For example: I sent a remote node the command meshtastic --set is_low_power true, got confirmation that settings had been saved, but the device vanished and could never receive my reboot request.

Same issue here. Macbook connected via USB to tbeam. Another tbeam as dedicated node.

meshtastic --dest !c4f7e0e0 --port /dev/cu.SLAB_USBtoUART --set is_router 1
Connected to radio
INFO:root:Requesting preferences from remote node (this could take a while)
INFO:root:Requesting channel 0 info from remote node (this could take a while)
INFO:root:Requesting channel 1 info from remote node (this could take a while)
INFO:root:Requesting channel 2 info from remote node (this could take a while)
Set is_router to 1
Writing modified preferences to device

a moment later

meshtastic --dest !c4f7e0e0 --port /dev/cu.SLAB_USBtoUART --set is_router 0

Connected to radio
INFO:root:Requesting preferences from remote node (this could take a while)
INFO:root:Requesting channel 0 info from remote node (this could take a while)
Aborting due to: Timed out waiting for node config

and few seconds later

meshtastic --dest !c4f7e0e0 --port /dev/cu.SLAB_USBtoUART --set is_router 1
Connected to radio
INFO:root:Requesting preferences from remote node (this could take a while)
Aborting due to: Timed out waiting for node config

firmware-1.2.30.80e4bc6
meshtastic --version 1.2.33

I’ve found that I have to send all commands in a single admin call to make it work, because if the first call succeeds, the subsequent ones will all fail. So, I send all the commands, including the –reboot command.

Having said that, the chance of getting the admin channel command to succeed is still closer to zero for me.

I’m also having issues with using the admin channel. I get a python error… Any idea what this is about? Meshtastic CLI works for everything else except remote admin commands using the ‘–dest’ command.

CLI output…
"
C:\Users\Jonathan\AppData\Local\Programs\Python\Python310\Scripts>meshtastic --dest !f71e3658 --reboot
Connected to radio
Requesting preferences from remote node.
Be sure:

  1. There is a SECONDARY channel named ‘admin’.
  2. The ‘–seturl’ was used to configure.
  3. All devices have the same modem config. (i.e., ‘–ch-longfast’)
  4. All devices have been rebooted after all of the above. (optional, but recommended)
    Note: This could take a while (it requests remote channel configs, then writes config)
    Requesting channel 0 info from remote node (this could take a while)
    Requesting channel 1 info from remote node (this could take a while)
    ERROR file:stream_interface.py __reader line:171 Error while handling message from radio ‘admin’
    Traceback (most recent call last):
    File “C:\Users\Jonathan\AppData\Local\Programs\Python\Python310\lib\site-packages\meshtastic\stream_interface.py”, line 169, in __reader
    self._handleFromRadio(self._rxBuf[HEADER_LEN:])
    File “C:\Users\Jonathan\AppData\Local\Programs\Python\Python310\lib\site-packages\meshtastic\mesh_interface.py”, line 535, in _handleFromRadio
    self._handlePacketFromRadio(fromRadio.packet)
    File “C:\Users\Jonathan\AppData\Local\Programs\Python\Python310\lib\site-packages\meshtastic\mesh_interface.py”, line 685, in _handlePacketFromRadio
    handler.callback(asDict)
    File “C:\Users\Jonathan\AppData\Local\Programs\Python\Python310\lib\site-packages\meshtastic\node.py”, line 322, in onResponseRequestChannel
    c = p[“decoded”][“admin”][“raw”].get_channel_response
    KeyError: ‘admin’
    Error: Timed out waiting for node config
    "

here is my --info settings
Owner: A.radio (A.r)
My info: { “myNodeNum”: 621168760, “hasGps”: true, “numBands”: 13, “firmwareVersion”: “1.2.59.d81c1c0”, “rebootCount”: 11, “bitrate”: 17.08847, “messageTimeoutMsec”: 300000, “minAppVersion”: 20200, “maxChannels”: 8, “hasWifi”: true, “airUtilTx”: 6.652139 }

Preferences: { “positionBroadcastSecs”: 900, “phoneTimeoutSecs”: 900, “lsSecs”: 300, “region”: “US”, “debugLogEnabled”: true, “positionFlags”: 288 }

Channels:
PRIMARY psk=secret { “modemConfig”: “Bw125Cr48Sf4096”, “psk”: “=", “name”: “JDS1”, “id”: 1234 }
SECONDARY psk=secret { “psk”: "
=”, “name”: “admin” }

Just wanted to bump this topic. Curious if anyone has tips for making remote admin work better. I tried to config the admin channel to use a different modem config of med/fast instead of my primary channel which uses long/slow, but I think that just caused more problems. Anyway, it seems one of my nodes is completely unable to take remote commands and it is just a few meters away on the roof of my house.

Hello,
just tested with configuration Meduim/Fast. Admin channel doesn’t actually work.
–info incomplete
–reboot ok
–export-config not
I gave up testing more commands.

You can’t send the whole config on the admin channel, it is too big. It is being broken out cleaned up in 1.3. Your reboot message is an admin message so the admin channel is working.

I have yet to get this to work at all, I can’t even get the reboot command to work. It just times out every time. I followed the instructions creating a secondary channel called ‘admin’ and then copied all the other radios the complete url. Any idea when this will be fully functional? It’s gonna be tough to put a node in a remote location without this feature.

Are you using the default long slow setting? September is probably about the earliest I would expect this feature to arrive in 1.3.

Yes all the radios I have have been programmed with the complete URL. I’m wondering if I should just factory reset each one and start over?