I have a MicroTik hAP lite designated as "KN6RVU-BASE" and connected to port 5 is a Ubiquiti Rocket M3 designated as "KN6RVU-DISH3". Plugged into port 2 of KN6RVU-BASE is a switch with a GXP1630 IP Phone designated as KN6RVU-IPPHONE connected. Also, my laptop is plugged into the switch.
- The KN6RVU-BASE WAN in connected to a static IP address provided by AT&T.
- The KN6RVU-DISH3 is connected AE6XE-PleasantsPk-RM3 with LQ:100% NL Q:92% TxMbps: 26.0
- There is a tunnel to another MikroTik hAP lite.
The system is working good on the AREDN mesh.
I am trying to create redundancy into the system. In an emergency, if the internet goes down and only the repeater is available, I would like the system to still work.
I have run three experiments using trace routes to 3 different domains:
Test 1: The full system is up with internet, with tunnel and repeater connected.
Test 2: The full system is up with internet, without tunnel and repeater connected.
Test 3: Internet down (WAN) and repeater connected.
Parameters
KN6RVU-BASE (MikroTik uAP ac lite)
Mesh RF: 10.242.252.115/8
LAN IP : 10.151.227.153/29
WAN IP : 104.1.254.XX/29
DNS1 : 68.94.156.8
DNS2 : 68.94.157.8
Default Gateway: 104.1.254.XX
dtdlink/md2: 10.242.252.110
KN6RVU-DISH3 (Ubiquiti Rocket M3)
Mesh RF: 10.14.32.221/8
LAN IP : 10.113.6.233/29
WAN IP : none
DNS1 : 8.8.8.8
DNS2 : 8.8.4.4
Default Gateway: depending on test
dtdlink : 10.15.32.221
AE6XE-PleasantsPk-RM3
WiFi : 10.14.32.134/8
LAN IP : 10.113.4.49/29
WAN IP : none
default gateway: 10.139.72.156
Test 1
KN6RVU-DISH3 default gateway: 10.242.252.110 dtdlink / mid2
- DNS always resolved the domain name's IP addresses.
- Trace routes have two paths depending on the destination.
- Thru the tunnel
- Thru the WAN to an AT&T node (64.148.105.174).
Test 2
KN6RVU-DISH3 default gateway: 10.14.32.134 AE6XE-PleasantsPk-RM3
- DNS always resolved the domain name's IP addresses.
- Trace routes have two paths depending on the destination.
- dtdlink.KN6RVU-DISH 10.15.32.221 ---> 10.14.32.134
- Thru the WAN to an AT&T node (64.148.105.174).
Test 3
KN6RVU-DISH3 default gateway: 10.14.32.134 AE6XE-PleasantsPk-RM3
- DNS rarely resolved the domain name's IP addresses.
- Trace routes tries two paths depending on the destination. Sometimes IP addresses works.
- dtdlink.KN6RVU-DISH 10.15.32.221 ---> 10.14.32.134
- Tries thru the WAN without success.
Problem
How to configure the system to continue working if the internet (WAN) goes down.
1) on what device did you execute the traceroute command? What is this device's IP address?
2) on KN6RVU-BASE, how is the option in basic setup (checked or un-checked)? "Allow others to use my WAN"
I suspect for #2, this option is not set. I suspect for option 1, you are not running traceroute on a device on the LAN of KN5RVU-BASE. Here's a couple of relevant data points:
1) devices on the LAN of KN6RVU-BASE will always use the internet or WAN interface, should the ethernet link be up.
2) devices on the LAN of KN6RVU-DISH3 will not use the internet-WAN from KN6RVU-BASE unless "allow others to use my WAN" is checked on KN6RVU-BASE. (hence it will default to Pleasant Pk) Since your traceroute went to Pleasants Pk, you must be running traceroute on a device on DISH3 (not BASE).
3) It is possible for the Ethernet WAN link to be "up" to your home router, but the internet access is down. The mesh node can not see or detect past your home router to know if something is broken. Consequently, there is no automatic switch over to route through Pleasants Pk if your internet service is down. You'd have to manually pull the ethernet cable. (possible we could add more smarts to "ping" something on the internet to insure connectivity and then trigger the routing to the internet to change as a future enhancement.)
Joe AE6XE
Joe, thanks for the response.
1) Attached is a diagram of the system.
2) I have check "allow others to use my WAN" on KN6RVU-BASE.
3) Traceroute was run on KN6RUV-BASE for all three of the test.
4) The WAN is connect to a static IP address on internet. When I ran test 3, I pulled the WAN cable.
I liked your idea of checking connectivity of both the internet and repeater and adjusting the routing table accordingly.
Would there be any advantage to run the computer and phone on the KN6RVU-DISH3 LAN?
Can you confirm with test3. I would expect if there is no Ethernet link, with cat5 not connected, that the node would not try to route out a non-existent interface (or down interface). It should change the routing to go to the nearest gateway via Pleasants Pk.
Need to confirm the steps to manually get the node to use an alternative mesh internet gateway, should the WAN path not be functional.
Joe AE6XE
I ran a number of tests using traceroute to better define what is happening. Attached is a table of results.
When the WAN was disconnected, it took about 20 mins of testing for the route to find the ae6xe path.
The problem was determining the DNS for the domains.
One site never resolved.
2) You have 2 paths from DISH3 to Pleasants Pk. The route calculations give a cost of ~"1" direct RF between your DISH3 and Pleasants Pk. The dtdlink and tunnel connections are going over 5 hops to get from DISH3 to Pleasants PK. Each of these hops has a cost of "0.1" or a total cost of "0.5". Since the cost is lower, it is routing over 5 non-RF hops, instead of 1 direct high quality RF link. Suggest not to use the tunnel, it's not needed (and is sort of defeating the purpose).
Reviewing the system. I have a MicroTik hAP lite designated as "KN6RVU-BASE" and connected to port 5 is a Ubiquiti Rocket M3 designated as "KN6RVU-DISH3". Plugged into port 2 of KN6RVU-BASE is a switch with a GXP1630 IP Phone designated as KN6RVU-IPPHONE connected. Also, my laptop is plugged into the switch.
I have done some additional testing and found the following:
a) After changing the system configuration, it takes minutes for the router to change gateways and resolve DNS lookups.
b) If the WAN, tunnel and dish are connected, the DNS and route go through the tunnel.
c) If the tunnel is turned off, the DNS and route take time to resolve to the dish. At first the DNS come back with a 23.XX.XX.XX address which routes to an akamaitechnologies.com static IP. Sometime to the same IP for different domains and sometimes different 23.XX.XX.XX IPs. The 23.XX.XX.XX tries to route through the WAN. Once the DNS returns a 10.XX.XX.XX, the dish becomes the gateway.
d) If the dish goes down, the data is routed through the tunnel. If the tunnel is down, there is no path for the AREDN mesh because the internet doesn't route 10.XX.XX.XX IP's.
Conclusions
1) For system redundancy, you need a reliable path through a tunnel on the WAN to the AREDN mesh and a RF connections to the AREDN mesh.
2) The chosen route may not be the best. Example, the dish is connected to the AE6XE-PleasantsPk-RM3 repeater, but the data route goes through the tunnel to the repeater.
3) It takes time for the system to adapt to the system changes if one of the paths goes down.
Joe AE6XE, thank you for your help.
"c) If the tunnel is turned off, the DNS and route take time to resolve to the dish. At first the DNS come back with a 23.XX.XX.XX address which routes to an akamaitechnologies.com static IP. Sometime to the same IP for different domains and sometimes different 23.XX.XX.XX IPs. The 23.XX.XX.XX tries to route through the WAN. Once the DNS returns a 10.XX.XX.XX, the dish becomes the gateway."
This behavior is specific to the path for DNS to resolve hostname and obtain an IP address. When a hostname is NOT known on the AREDN mesh network, DNS will forward go out a WAN somewhere to ask the ISP to resolve. Some DNS servers, when no resoultion can be found, will forward to a default page. this can be to an advertisement page, etc.
This is what is happening when the hostname is resolved to the 23.XX.XX.XX address. Then, routing to the 23.x.x.x address, follows the path out to the internet. If the internet is not reachable, the DNS path may not be able to resolve the hostname to begin with, and obtain this non-mesh 10.x.x.x address.
Some may see this behavior, some may not.
Joe AE6XE
update: added the NOT above, big difference :)