Report on Meshoween 2021 network test

Note - this report is written from a Southern California perspective. Depending on your location, as they say YMMV!

During the test the node count on the linked networks locally rose to around 1,090.  N2MH's N2MH-Hub saw 1428, most likely a record for an AREDN network.

Observations:

As we hoped, network storms never appeared from our West Coast point of view, and none have been reported to date.  Network traffic was not substantially higher than normal. Using KN6PLV's Mesh Monitor, messages per second were averaging about 600 before the test.  During the test it hovered closer to 700. Most of the increase in traffic was due to OLSR routing broadcasts, which each node has to handle.

As we expected, older devices struggled. Many wound up with a load significantly greater than 1 (which is a full load for a single-CPU device like these), indicating their CPUs were struggling to keep up with their pending processes.  Ssh'ing into a few of them and running top showed that with node counts around 1,000, loads of around 4-5 were seen.  

Things we didn't expect:

While the slower nodes struggled with displaying UI pages, they were usually still able to pass traffic.  But occasionally that older hardware bogged down so much that it couldn't.  Perhaps some effort should be made to prioritize traffic handling over the UI display.  There were a few reports of Nanostations that stopped responding completely.  

At 900 nodes and greater the Mesh Status page takes a long time to display.  Delays of up to 25 seconds were reported.  Occasionally it would fail the first time but would load on second try.  During the test Tim KN6PLV took a look at the UI code and has some thoughts on how to improve the display speed. We'll see where that goes.

Hardware:

Both the older Nanostations (XM hardware) and the newer version (XW) hardware both seemed to struggle equally.  So I took a deeper look at the Nanostations.  It turns out that while the newer XW hardware has twice the RAM as the older XM stuff, they both use the same 400 MHz CPU.  So you won't see any performance improvement by upgrading Nanostations.

But the Mikrotik equivalents, the SXTsq-2nD, and the SXTsq-5nD for 2 and 5 GHz respectively (and the high power SXTsq-5HPnD) have about 50% more processing power than the Nanostations.  I use one here for 2 GHz and it worked fine.  Its load average was .40  and it displayed the mesh status page about twice as fast as the Ubiquiti gear.  

Cheaper, too! Recommended as upgrades! 

Approximate prices (didn't shop around a lot):

NSM2:  ($89 Amazon)
SXTsq-2nD: $39 (ispsupplies.com)

NSM5:  $130  (Amazon)
SXTsq-5nD: $80 (Walmart :-D  )
SXTsq-5HPnD ~$49 (ispsupplies.com)

If your experience made you think you'll need more gain in the future, you could go for one of the Ubiquiti or Mikrotik dishes.  Both have faster CPUs than the Nanostations - Ubiquiti Powerbeam: 560 MHz, Mikrotik LHG products: 650 MHz).

We'll do this test again, maybe semi-annually.  Anyone up for the Fourth of MESHuly?  :-)  We'll certainly wait until at least after the AREDN production build that's been rebased on the most recent OpenWrt code (updated drivers, security fixes, etc.) is available.  That will give time for people to re-evaluate their hardware situation and decide what upgrades, if any, they want to implement.

The AREDN devs thank you - this was a very illuminating exercise.

And thanks to all that participated - willingly or otherwise :-D