You are here

Use of Raspberry Pi3 as MeshWebServer

15 posts / 0 new
Last post
AA7AU
AA7AU's picture
Use of Raspberry Pi3 as MeshWebServer

We are working on the early build-out phases of a proof-of-concept (inexpensive) AREDN multi-mode mesh network here in the mountains of East Central Idaho. If all goes well, we will find better funding and be able to afford better hardware etc and build out more locations. In the meantime, it's all on a shoe-string budget - but we want to demo a well-featured integrated alternate AREDN network to our potential local users.

We have a Raspberry PI3 running Rasparian *Jesse* configured to use for webservice (etc) on our local mesh. It currently works well over the mesh when plugged with an ethernet cable directly into the LAN port of one of our Bullets at the POE and/or thru a dumb (non-VLAN) switch. It also will respond to traffic coming in over it's WIFI side and enable me to connect to it over public internet and then thru a residential WISP router (which has had the proper ports opened/protected for SSH/etc) etc to the WIFI side of the PI3. But the PI3 won't work on both interfaces at the same time!

In the best of all possible worlds, I would like that Pi3 to be active and responsive over both eth0 to the Mesh as well as be active and responsive to traffic coming over the wlan0 WIFI entry-point so that I can work remotely on the Pi development/maintenance as needed at the same time.

While I've worked professionally in computer systems/software since 1969 (until retired), I still get a bad headache when I try to understand advanced network routing. The recent changes within the Pi/Rasparian implementation arc only make this worse, such that much of the on-line help is now totally and hopelessly out of date for Jesse (and god forbid, Stretch).

My question is: how do I set my Pi3/Jesse configuration up to handle the in/out traffic properly from both eth0 and wlan0 as described above?

Thanks for any help, encouragement, advice, etc - Don - AA7AU
 

KE0EYT
KE0EYT's picture
Have you made sure that
Have you made sure that hotplugging is disabled?

 
kg7bz
Disable Default Route

This isn't a problem on the Pi itself, but a configuration on the MESH node.

On the Basic Setup page for the node you have the Pi connected to, uncheck the "Disable Default Route" box under Advanced. This stops the node from advertising that it's the route to everywhere, not just the 10.x.x.x MESH. This route is disrupting your Pi Internet traffic out the WiFi interface when you have both interfaces (Ethernet for MESH and WiFi for Internet) connected.

AA7AU
AA7AU's picture
Yes, that was it, thanks!

Six moths later as I finally try to catch back up here in the forum and get back to work on our local Mesh Island now that spring is here, I realize that I neglected to post here in this thread for anyone else with this issue: Yes, thank you to KG7BZ, that was the exact answer I needed. It was right in front of me all the time - Disable Default Route. Much appreciated. - Don

AA7AU
AA7AU's picture
Rpi3 server update

We now have our [solar/battery-powered] Raspberry Pi3 "server" hard-wire (CAT5) connected to one of our mesh nodes thru a dedicated/dumb switch. That mesh node has "Disable Default Route" checked. The PI is also connected thru its WiFi link to a local LAN which does have regular internet service to it. We are able to use controlled access to the PI thru both the internet and the mesh. For maintenance, I can use SSH and/or RDP to remotely connect to the Pi on the LAN side.

Under "DHCP Reservations" we have reserved a Hostname with a static IP for the lan0 MAC address of the Pi and then defined two services on that Hostname. The first is a general http webservice which is advertised *with* a link (more on that in a later post); the second is the NTP service which is advertised *without* a link. I have configured a couple of our mesh nodes to use that NTP service and it works well. I do note that it takes some time after reboot for the NTP update to occur for that node.

We are starting out using external NTP resources thru the LAN side, however there are other options available later on to use GPS-based time or even manually set it for WTSHTF worst cases.

My questions this time are: 1) is it normal for the individual mesh node to take so long to get the time updated after reboot, 2) what is the refresh frequency after that (bandwidth use implications)?

TIA - Don

KG6JEI
The NTP sync runs by

The NTP sync runs by attempting to ping the host in question once a minute until a ping succeeds.

This means until a route is available to the NTP server the test will fail and an attempt to sync will not occur. Depending how many hops away, how good of quality the links are and depending on if you are using a name or an IP this may take some time to attempt a sync.

After the system has determine NTP server is reachable the ntpclient is launched and syncs one time and does not attempt to sync again.

Edit:
A previous version of this post stated a 600s retry but that is not accurate as we do only a “simple”  check which has a count limit of 1 instead of infinite tries.

AA7AU
AA7AU's picture
Drift? Or, perhaps bad original sync?

Thanks for the update. Do I understand that once a mesh node has synced sucessfully, it does not ever do a refresh again? That would seem to imply that a node might get a bad time (bad setup etc) once or it might drift over time such that it would never get corrected until a subsequent reboot?

Another question would be: does this mean that all mesh nodes which have not sucessfully synced are continually trying to ping whatever NTP server is setup for it once a minute forever while powered up?

TIA, - Don

KG6JEI
"That would seem to imply
"That would seem to imply that a node might get a bad time (bad setup etc) once or it might drift over time such that it would never get corrected until a subsequent reboot?"

I wouldn't expect a node to get a bad time, timeservers are generally very reliable, they even know internally how out of sync they are from their master source when they are syncing from other servers.   Its of course possible if the server is badly misconfigured but if that is the case there are bigger issues.     Drift is a possibility but I haven't personably seen that occur and haven't had anyone report seeing time drift.

"Another question would be: does this mean that all mesh nodes which have not sucessfully synced are continually trying to ping whatever NTP server is setup for it once a minute forever while powered up?"

They will send a single ping packet once every 60 seconds until they get a response after that they will try and start the time sync which will run under its own routine.  It will not try and restart the ping routine.

Strictly speaking nothing on the node is currently time sensitive. It really only impacts the log files and the main page where it shows you the time, everything else on the node currently doesn't care about the time.  This might change if we ever enable DNSSEC (even if we do this it woulnd't break the mesh side only the  internet side) or if some of the download sites ever move to being on SSL (personally I don't have any plans for that but I can't speak for everyone else) but otherwise nothing on the node is time sensitive currently and it really is just for display purposes.
 
AA7AU
AA7AU's picture
NTP really is just for display purposes ???

I don't mean to be argumentative, however If NTP is only for display purposes then why the heck is it there?

Also, "I wouldn't expect a node to get a bad time, timeservers are generally very reliable" - it is our intent to run our mesh as an emergency digital network - which would seem to imply that public internet-based services like NTP will possibly not be reliable/available. That's why I put up our own NTP server. We are a mesh *island* and do not use tunnels or direct internet connections. The only way we inter-connect is thru a device like our PI as described above.

Above you wrote "After the system has determine NTP server is reachable the ntpclient is launched and syncs one time and does not attempt to sync again." However this seems like the problem where the mesh remembers forever an abandonded tactical node name without rechecking it until that node drops off-network for a time-out, instead of reverifying the tactical name (and/or the NTP time) on some sort of regular timely basis. This does not seem like what I would consider a resilient approach to network design. Neither does having every node which does not have an NTP connection available just keep pinging out into the darkness forever.

I did run across a help file which seemed to imply that NTP *was* checked and kept current on the mesh. I remain confused on this. Had I realized all this beforehand, I might lowered the priority of setting up our own NTP service and moved on to other tasks. Was the original default NTP server setup as blank?

Thanks for all the on-going help. More perspective would be much appreciated. We're struggling with used M2 Bullets (all we could afford) and inexpensive antennas and trying to make our proof-of-concept AREDN network something to demo to our local support community and upon which to base future funding requests. I still can't get up to our local mountain top M2 install due to snow etc. Maybe next month. We're trying hard here ...

TIA,
- Don

KG6JEI
Note: If were going to
Note: If were going to continue discussing this we might want to open a new forum thread as we have really moved away from the original topic of "Use of Raspberry Pi3 as MeshWebServer" which means this discussion is become very much buried where it will not be of much use to other users.

"If NTP is only for display purposes then why the heck is it there?"
A fair question,   Part of this is historical, and this has to be traced back into our history under the BBHN name.  Back under those days (and even a bit under AREDN a bit with MeshChat for example) it was considered acceptable to run additional software on a node. Protocol Version 0 and protocol Version 2 might of had some minor TIME affinity related to the message signature algorithm, but it didn't have to have accurate time but it could benefit from it. Some of these additional software that were historically talked about running on  node could very much need time for their purposes even though they were never officially supported as part of the project.  This history continues to spill over and the view of the devices continues to adjust as time continues, but a lot of historical  'its always been' and "it may still some day be needed"  remains.    Just because today we have nothing onboard the node that is time decadent doesn't mean we won't have a new feature that is somehow time dependent tomorrow.  A Feature Request to remove the capability hasn't ever been submitted so the core group hasn't ever discussed if it comes out.  I can't say if it would be taken out or not if requested as i could see arguments for and against but ultimately it makes everyone happy to see accurate time on their node so a NTP server setting has to stick around though equaly (untill something becomes time dependet) it a similar level could be served for a large majority of users by the team just removing the date/tiime fields from the GUI and no one would have a clue about it but that didn't get much traction at the time I had mentioned it.

"That's why I put up our own NTP server. We are a mesh *island* and do not use tunnels or direct internet connections. The only way we inter-connect is thru a device like our PI as described above"
Even then I would expect the timeserver to be fairly accurate, not knowing your exact layout, timeserver daemons know to degrade themselves  if they loose their sync source. GPS Synced systems will degree to local clock and indicate as such inside their protocol and may even refuse to serve time.  Of course its always possible to setup a timeserver that syncs from its local clock w/o syncing it to a secondary source (GPS, GPSDO, Rubidium, etc) in which case you could have error yes.  Ultimatley this sort of reliability and accuracy would be up to the timeserver admin to configure in their system. A node will always have to trust its getting valid time data as it can not indepdnetly verify it.  How accurate you want that time source to be is a choice of the local network. I know some groups that will be happy with +/- several minutes time across their network, and I know some groups (maybe not so much in ham but in the IT world where I work its a lot more valid) where +/-100ms is acceptable only as long as the clock is in the process of skewing to accurate to the nanosecond. 

"I did run across a help file which seemed to imply that NTP *was* checked and kept current on the mesh." Its possible we have some bad documention somewhere,  if you see it a bugfix ticket is always appreciated so we can track it down. A lot of code went in before I even started working with it under BBHN,   a lot of code then went in after I started working with it but before I was able to get a formal code review process in place.  During both those windows small items like documentation could have been easily overlooked. Its very easy to get focused on one small part and forget about the rest not realizing it exists.. 

"Was the original default NTP server setup as blank?"  The original should be us.pool.ntp.org

I'll note until the 3.16.1.x release versions this wasn't even a setting a user could change, it was set in the backend and wasn't a suer choice. I can with hindsight reading your message see how someone may think a timeserver actually serves a purpose to a node for some functional reason based on how that page is laid out.

"We're struggling with  used M2 Bullets (all we could afford) and inexpensive antennas and trying to make our proof-of-concept AREDN network something to demo to our local support community and upon which to base future funding requests."

I can understand trying to work within what you have set as a budget, honestly though the bullet is one of those devices that is just a shot in the foot.  Depending what you mean cheap antennas (for example those yagi's where the square boom is 50% of the width) you could be ending up with a dummy load for an antennas or higher cost than it would of been to just wait a little bit and go with a NanoStation or some other setup. There is a good article from Clint who I worked with back in the BBHN days http://ae5ca.com/your-first-bbhn-node-should-be-a-nanostationm2/ where much like him I thought the bullets would be great devices and I too found out while they were better than the WRT54's they were nothign compared to the NanoStations for both ease, cost, and quality.   In my experience from looking at various deployments for the Proof of Concept its really important to go with good solid gear vs what is cheaper.   You can always bring people in later with some of the less ideal gear, but if you can't get the concept to work  you loose the interest of individuals and may never get off the ground.   I've heard of this with WRT54's from back in the day, they got the community talking, but they were so limited they would actually push people away from trying got setup a mesh.     To do a good POC for my city and get them interested I invested in a Rocket and Sector antenna, it wasn't cheap but it created a reliable network that got users on, the network has grown for my city to the point where once my node was core critical that now it can drop off in a failure and the network has enough redundancy to continue on without it..  This however took having a strong presence that users could reliable connect into  first to get them momentum growing.

"However this seems like the problem where the mesh remembers forever an abandoned tactical node name without rechecking it until that node drops off-network for a time-out, instead of reverifying the tactical name"  This is a known flaw, I put some partial fixes in  under a previous version but they were found not to fix everything.  Unfortunately the team is small there is limited time to track down and go after issues. I find a lot of times my codeveloreps seem to want to hit fancy and new feature vs take time to work on a known problem and seem to want to focus on  'how can I get some code in quickly even if it has flaw and we can fix them some undetermined time later" (A heated discussion item on our last held weekly call on the 8th)  And unfortunately sometimes even larger issues come up that force us to move away from dealing with the more obvious items (like this flaw) to dealing with them for longer term stability.  Its a juggeling match of resources and enthusiasm that is hard to fulfill every desire.


"Neither does having every node which does not have an NTP connection available just keep pinging out into the darkness forever."  From a technical standpoint I wouldn't expect this to be much different from just blindly checking for NTP server. Getting very technical,  they both use UDP packets.  ECHO Packets have a shorter lifetime vs a NTP packet which means less socket open time which means less exposure and less resource consumption.  In the case of  a DNS name like the default us.pool.ntp.org  unless one has internet access the domain name won't even resolve which means a ping will exit out immediately.  The same is true of a hostname based on the local mesh, until the node has the hostname in its data file its not going to even send any packets off the node so its not like any extra traffic. This also holds ture to a static IP unless there is some route to that IP or a default (internet) route its not going to generate any packets off node anyways (they will all be dropped inside the kernel)   Why the code was written this way originally I don't know, I believe this was common of  OpenWRT firmware at the time the project (under the BBHN name) was started and its held on, I'm not sure if its still the common method or not.  Noting that embedded routers often do things much differently than a desktop would because of how they operate in much more confined environments.  Could there be better ways to do it, possibly.  Without a deep technical dive onto every line of code I can't say for sure if this is actually that bad of a method to deal with it  since the NTPClient is a very simple application it may not handle non resolvable hosts and packet loss as well as a ping could can as just one example of maybe why it was done this way.
AA7AU
AA7AU's picture
WOW! Thanks, will start a new thread soon!

Thanks very much to KG6JEI for such a terrific post. I totally agree ..."If were going to continue discussing this we might want to open a new forum thread" .  I will do so soon, and will post a link back to this discussion above as well. Your help and historical perspective is very much appreciated.

73,
- Don - AA7AU

ps: I may actually be down in Oceanside for a brief trip later on this summer, perhaps we can visit with each other while I'm there. It is a long way from East Central Idaho, but I did spend 20+ years in San Diego way back when working with computer software/systems so I know the area. I do missing blue-water sailing/racing very much. We used to claw our way up the coast occasionally and stay in Oceanside harbor on our way to/from Catalina etc.

W8GUC
W8GUC's picture
Use of Raspberry Pi3 as MeshWebServer

Don.... I just purchased a Raspberry Pi3, and am wanting to build a MeshWebServer also.
Could I please get a copy of the software(s) that you have built/put-together to build the same.
I would not like to start from scratch, and would very much, enlist your supports to me, and my start of the same endeavors.
Thanks,and 73,
Reuben,
W8GUC,
Dayton, Ohio.

AA7AU
AA7AU's picture
Copy from a PI3 as server

>Could I please get a copy of the software(s) that you have built/put-together to build the same.

While it sounds like a good idea, it's not, sorry.

As almost everything in this involves some sort of configuration (custom tailoring) and decision process, it would be much more problematic than beneficial to start with an exact copy of someone else's currently operating system (with all their config decisions, passwords, databases, etc etc). This is definitely not a one-size-fits-all kind of thing. There in a learning curve that a system operator should go thru in creating a customized server, else the very first maintenance/support issue is likely to derail you completely.

And, there are "gotchas" even in the difference between PI3B and PI3B+ not to mention the earlier discussion above of which release of Debian (or other Linux) to choose and then which LAMP combination is best ... While there are "cookbook" approaches out there, as I wrote earlier - lots of stuff is out-of-date, so I can't even point you to an A-to-Z cookbook plan to building one. Perhaps someone else here can help further ...

Sorry. Good luck,
- Don - AA7AU

W8GUC
W8GUC's picture
Use of Raspberry Pi3 as MeshWebServer
Don... thanks for the reply anyway... I will plug alon with with my ideas, and will keep you/the group, posted as to my progress here.
It's a novel idea, and I believe it can/will work.
Best regards,
Reuben, W8GUC.
AA7AU
AA7AU's picture
End thread here?

Here's a brief list of where I'd suggest you start for LAMP if you are NOT using the newer Pi3B+:
Use a PI3B (not B+) with a new (I buy from Amazon) SanDisk class10 (or better) 16gb MicroSD card; get a Logitech KB/Mouse that works with their "Unify" (single dongle supports multiple devices) USB receiver, and a nice long HDMI cable to connect to your monitor and/or TV (some day I'll post my recommended hardware parts list) - then -
Download and install Raspian/*Jesse* (full version, not "lite") from the Raspberry Pi resources;
then install Apache; Mariadb; PHP5 (wish I could be mores specific but this is not the forum for PI3 support).
Once you get that running, learn about "headless". enable SSH on the PI, and learn to connect with something like Putty (but you have to learn CLI for Linux). Then install XRDP for Remote Desktop ... but I digress.

Have fun - lots to learn!
- Don - AA7AU

End Thread here - sorry, but this is not a forum for Raspberry Pi usage [-Don]

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer