IP over meshtastic kinda implemented

[CycloMies]

Unreliable UDP connections/packets resembles wireless LoRa packets.

Feature Request: Supporting mesh packets on UDP over IP would allow Meshtastic nodes to create a mesh network over IP, when connected to the same subnet, in the very same way they create it with LoRa.

Furthermore, this would allow testing of the mesh protocol, for developing a network interface agnostic protocol, without LoRa connections, or without any type of other IP network infrastructure (as the Meshtastic nodes would manage the mesh on UDP).

This could also be a easy way to create mesh connections between repeaters over WLAN/LAN.

This feature might be achievable with moderate effort, as the mesh protocol is already designed by @geeksville , and the WLAN-client feature has already been designed by @mc-hamster (as I recall).

MQTT is on the roadmap, as @mc-hamster noted, but does rely on reliable connections, such as TCP over IP, and other external devices, such as MQTT servers.

This feature has been shortly discussed here: iOS Experiments

[geeksville]

Good point. I’ve been reluctant to add UDP support but the cases you list are compelling. Also for aviation applications (or any application where at least one node has great line of sight) a very fast spreadfactor should work fine.

So meshtastic could provide a nice long range zeroconf IP link in that case.

I think it would not be too hard to have the meshtastic python app implement a user space TAP/TUN network interface. So that a zeroconf IP link for any node on the mesh would appear on a “mesh0” interface. I’ll do the preliminary investigation/doc reading this week.

[geeksville]

keeping notes here:

Use the TAP/TUN driver to somehow provide IP connectivity over meshtastic

[CycloMies]

I did read your notes. Interesting research done!

How about “simplifying” things (the most simple things tend to be the most complex ones), at least for the beginning?

Packets on UDP could be carried within the UDP payload.
Therefore, the mesh protocol would handle all the basic things on both LoRa and UDP the same way. However, UDP links should be assigned shorter tx delays.

The IP link can listen for packets, and broadcast packets, on UDP on a fixed IP and port, regardless of the assigned client IP. All mesh packets would be parsed from the UDP payloads. This should mitigate the process, as changing IP addresses, according to the node number, wouldn’t be necessary.

Creating the UDP link between two nodes:

1. Add listening and broadcast features for UDP over IP on the node trough WLAN client (this is mostly done?)
2. Connect nodes to the same net (UDP pass through / no NAT)
3. Add listening for UDP packets on 223.6.5.43:6543:udp
4. Add broadcast mesh packets as UDP payload on 223.6.5.43:6543:udp
5. Send mesh packets trough both LoRa and UDP (the whole ordinary mesh packet within UDP payload), if no known route exists. Send regarding routing table, on LoRa or UDP link, if a valid route exists
6. Listen and receive mesh packets also from UDP payloads on 223.6.5.43:6543:udp
7. Parse the UDP payload for designated node addresses
8. Record node routes from both LoRa and UDP
9. Receive the message if intended to us, change hop counter & retransmit if not (do not retransmit from UDP to UDP)
10. Send ACK if required

UDP transport would be transparent for the Python user, as the nodes mesh protocol would handle the logical routing between LoRa and UDP as necessary.

This approach does not allow direct UDP connections between python host and the node, but a direct UDP mesh connection between two nodes.

What do you think of this approach?

[geeksville]

I don’t have a keyboard until Monday so I can’t write much now. But I was surprised at how easy the TUN api was, so I think just implementing TUN will be easier than what you describe (and it will work for any IP protocol/port number).

[geeksville]

This is now implemented and approximately works. Tested with ICMP pings and UDP echos (I’ve even done SSH over the meshtastic mesh).

Probably not ready for others yet because this testing turned out to be a great stresstest and I found some cases where we are dropping packets. The drops were hidden by the ack/retry mechanism we use most of the time for mesh packets. For IP I don’t use that feature (because IP handles its own retransmissions if needed). So I’ll be debugging and fixing that root cause sometime in the next few weeks.

But if anyone is super brave, this code is in the latest python tool: (It works on linux, might work on OS x, I doubt it works on windows)

meshtastic --tunnel 
Connected to radio
INFO:root:Starting IP to mesh tunnel (you must be root for this *pre-alpha* feature to work).  Mesh members:
INFO:root:Node !2462abf84098 has IP address 10.115.64.152
INFO:root:Node !2462abdddf38 has IP address 10.115.223.56
INFO:root:Node !fd1004ec5d56 has IP address 10.115.93.86
INFO:root:Node !246f28979058 has IP address 10.115.144.88

Use the TUN driver to somehow provide IP connectivity over meshtastic

[geeksville]

btw - if using the default ‘very long range but very slow’ channel settings, this really conveys how slow LORA is and not a good fit for many IP applications

Meshtastic-python$ ping -i 30 -W 30 10.115.64.152
PING 10.115.64.152 (10.115.64.152) 56(84) bytes of data.
64 bytes from 10.115.64.152: icmp_seq=1 ttl=64 time=17785 ms
64 bytes from 10.115.64.152: icmp_seq=2 ttl=64 time=17099 ms
64 bytes from 10.115.64.152: icmp_seq=3 ttl=64 time=18498 ms
64 bytes from 10.115.64.152: icmp_seq=4 ttl=64 time=17495 ms
64 bytes from 10.115.64.152: icmp_seq=5 ttl=64 time=17306 ms

i.e. 17 seconds round trip time for ping

[mc-hamster]

Outstanding! I’ll put this on my todo list to try out very soon.

[geeksville]

btw: we currently send the entire IP packet for every packet. An easy optimization would be to strip out much of the 20 byte header (because it is mostly redundant with our mesh info), which would probably cut ping times by 30%.

[Noki]

@geeksville Correct me if I’m wrong here: The original feature request was clearly to implement using UDP connections to extend the mesh. But as far as I understand it you now implemented IP connection on top of the mesh.

I want to point out that in Europe (and probably other countries as well) the 863 to 870 MHz band has limitations with either a transmission duty cycle of 0.1%, 1% or 10% (link 1 2) depending on the exact band. Limiting the duty cicle is important to stay within the legal regulations.

Does Meshtastic obey those duty cycles?

I think using a IP via LoRa bridge is not really an option for the European version.

[geeksville]

Good point! OMG you are right. When I read @CycloMies original request my brain just immediately convolved it with (almost the opposite) common request of “I’d like to be able to pump IP over these radios.” Which I had never been excited about (and still not) because the data rate is so low.

I think for cyclomies’ true request the best solution will be the (eventual) mqtt gateway and (optional) helper web service.

wrt the 1% duty cycle limit in EU: The original “1.0 meshtastic” of a “radio for hikers that occasionally sends small position messages and a few text messages” seemed fine in my rough spreadsheet. But for all this post 1.0 stuff - I have no idea and you are right to be concerned. There are no limits in the code that stop API clients from talking too much. Perhaps we could solve that eventually with using the existing ack/nak mechanism to return a fault code of “rate limited, packet dropped” to the API consumer. Alas it will be a while (after June 2021?) before I have time to focus on that (though I’d eagerly help anyone interested who wants to add this).

[mc-hamster]

It wouldn’t be much effort to perform air time duty cycle calculations. I can do this.

Once we have actual air time used, a quota or throttle system can be built as a next step.

Calculate air time duty cycles

[Noki]

I think both, MQTT and UDP have their up and downsides, but in terms of building a mesh network UDP would be the winner and focus for me. As CycloMies pointed out UDP connections could form an actual mesh, while the MQTT solution would form more of a star topology which is more likely to fail and requires additional hard and software. The simplicity of UDP would allow to connect multiple radios (e.g. with directional antennas in an isolated location) via WLAN by putting one in access point mode and the others in client mode. This would not require additional hardware running a MQTT server and could help to bridge longer distances and build the global mesh as well. It would also be a good and fast approach to get the community more active. Node operators would need to get into contact with other node operators to form the mesh.

Personally I think having a global mesh with multiple public and private channels as early as possible would really help to get more people involved. Beeing able to chat with people all over the world right after setting up a node would make it way more interesting to contribute. Without like-minded people to talk to on a global mesh the experience when first setting up a node is just not very satisfying and there is no point of leaving a stationary node running if your’re the only user. I would really go for pitching the story “Become part of a global mesh network”.

[geeksville]

Those are really good points.

[CycloMies]

IP and SSH over LoRa - incredible job @geeksville! This feature might be more valuable, in the future, than we might think today…

Thank you @Noki for describing the idea behind meshing over UDP.

UDP could be understood, in this case, similarly to wireless connections: direct connections to all within the reach (within the same LAN, WLAN, VPN…). Therefore, nodes could handle both LoRa and UDP interfaces similar, by building a mesh net from nodes they “hear”/reach to, and by maintaining a list of all nodes (both LoRa and UDP neighbors). Mesh packets could be sent the same way on both LoRa and UDP, as the interfaces are responsible for the lowest levels of networking.

UDP is handy, as you can send packets always to a predefined IP and port, regardless of the IP addresses interfaces are currently using. To do UDP meshing, the IP and port to listen to on UDP, for incoming mesh packets, has to be predefined.

The nicest features of UDP meshing would be the a) speed, and b) the ability to seamlessly link the mesh on both LoRa and IP nets. Think about a Meshtastic repeater on a hill, with a 4G/5G WLAN modem nearby with VPN client capability. You could then connect the node to the modem with WLAN, and the modem to the rest of your VPN network. With this setup, users could send ordinary mesh packets from LoRa trough VPN to anywhere on the mesh (limited by the hop count). This configuration would help the maintenance of repeater nodes, as one could do it trough the VPN with help of predefined mesh commands.

Any comments are always appreciated!

PS. @mc-hamster I’m also a big fan of a automatic air time calculation and restricting system. The app could print a value to inform users about the bandwidth left usable on the mesh (answering to the questions: “When will my packet be sent? and How much air time will my message consume of the allowed air time?”)

[mc-hamster]

I think I’m done with tracking air time usage for transmitted packets, it’s just untested right now. Because of the nature of this, actually testing will take a while. It keeps the history of up to 48 hours – each hour is reportable for historical analysis and graphing. I think it’d look cool to have a line graph of channel utilization and @geeksville suggested historical data may be used to make mesh routing smarter.

The framework is built to support reporting on the airtime of the current device, channel utilization of all devices on the mesh network (as defined by the channel configuration) and all lora devices on the current frequency, including those that are not part of our network. Those last two are just not connected to any hooks. It’s for future use.

I also don’t know how exactly to use this information but I’ll trigger and expose the report on the web interface for now (@crossan007 - if this interests you, i’ll put it in a json response, otherwise I’ll just dump it into a text file).

[CycloMies]

Moreover, UDP meshing would let users easily link nodes on different MHz to the same mesh, by connecting both nodes to the same WLAN. UDP meshing would handle the rest of the routing.

Discussion about linking nodes: Cross Band Relay or Bridge of a channel

[Aivaras-s]

while testing mesh-tunnel functionality on TTGO T-Beam V1.1 433MHz i noticed that sending side will crash after a while. error also was present in 1.1.31.
reproduce:
1.reset device to defaults and then meshtastic --setch-shortfast and start mesh-tunnel on both sides
2.ifconfig mesh0 mtu 200 both sides too
3.start listening on recive side: nc -l -p 608 -v|pv|dd of=/dev/null
4.start send from sender: dd if=/dev/urandom |pv|nc revicer_ip_address 608

after ~225KiB i get

Trigger powerFSM 9
Trigger powerFSM 3
Trigger powerFSM 9
assertion “isReceiving” failed: file “src/mesh/RadioLibInterface.cpp”, line 212, function: void RadioLibInterface::handleReceiveInterrupt()
abort() was called at PC 0x4010cec7 on core 1

ELF file SHA256: 0000000000000000

Backtrace: 0x4008ff5f:0x3ffd1b00 0x400902e5:0x3ffd1b20 0x4010cec7:0x3ffd1b40 0x400dcf33:0x3ffd1b70 0x400dcf5f:0x3ffd1b90 0x400dcfe0:0x3ffd1bc0 0x401d6e83:0x3ffd1be0 0x400d4e4a:0x3ffd1c00 0x400ecbf5:0x3ffd1c20 0x400d9cdf:0x3ffd1c50 0x400fbcf1:0x3ffd1c70

Rebooting…
???00:12:42 Emitting reboot packet for serial shell
00:12:42 booted, wake cause 0 (boot count 1), reset_reason=reset

[geeksville]

Thanks I’ll look into this soon!