You are here

Unexpected Rise in Noise Floor

21 posts / 0 new
Last post
n7ie
Unexpected Rise in Noise Floor

After a storm came through, we've experienced about a 10 dB rise in the noise floor, which has extremely limited the range of most individual nodes (2.4 GHz, Channel -2).  Your ideas as to possible causes/solutions are appreciated.

AE6XE
AE6XE's picture
N7IE,  I'm wondering if water
N7IE,  I'm wondering if water got into antenna cables or connectors?    I've monitored tower sites with buckets of rain coming down and not seen this behavior.   Is the SNR 10dB lower and noise floor showing -95dBm, or literally the noise floor now shows -85dBm?    reboot of the node help?

Joe AE6XE
n7ie
Joe, thanks for your input. 
Joe, thanks for your input.  The noise floor is now consistently at about -85 dB.  I've looked at the S/N Ratio charts of various nodes on our 2.4 GHz mesh (by backdooring in on 3.4 gigs).  Most around town show about -85 dB now.  Whereas I used to see multiple nodes, I now see only one who is less than a city block away from me.  Whereas he had a S/N Ratio of 59 dB before, his S/N Ratio is now 5 dB at my QTH.  When looking at the Realtime S/N charts, I see a "Strongest Signal" at random on several nodes that goes up to 0 (zero) dB, then drops off to below the -95 dB level.  But the noise floor stays at about the -85 dB level.  I've checked Channel -1 and Channel 1; both have noise floors at -95 dB.  Other ideas?  Thanks!
n7ie
BTW, yes, we are contacting
BTW, yes, we are contacting those on our system to power down their nodes to see if we can isolate the issue to a specific node.  73 .. .
AE6XE
AE6XE's picture
This is interesting.   I
This is interesting.   I would expect this to bounce around, to not be stable or static at -85dBm noise floor.   The firmware-driver has logic to periodically calculate what it believes is the noise floor.   This is a dynamic real time process.  When data error rates go up, or it misses xmitting ~10 beacons (AREDN rate is 2 per sec), because the channel is not clear (clear channel assessment CCA logic), then it will bump up the calculation  of the NF.  This is to drown out the interfering signals.    From memory I think it steps up/down by 10dB -- need to look at the code again.    

There really isn't such a thing as an absolute signal value -- the hardware's ability to do this is not accurate.  The 802.11 chip implementation basically say the received signal strength is relative based on a guesstimate of the noise floor.   Showing the NF at -95dBm or -85dBm is somewhat arbitrary.   A lot of this behavior is hidden in the qualcomm NDA, or not published anywhere.  But some is readily reverse engineered from the code.

Long story short, keep an eye on it to see what happens. There's noise form somewhere or a hardware-water problem.   If noise, a rock steady -85dBm would suggest somebody has a stuck interfering transmitter that the AREDN node is trying to get on top of.  

Joe AE6XE
  
n7ie
The algorithm moving the
The algorithm moving the noise floor makes sense.  The in-area node not seeing the issue is a nanobridge M2 pointed away from our local area.  The other more distant nodes, which do see the problem, have directional antennas pointed our way.  Wish we had Fox Hunt equipment for 2.4 GHz.  Thanks for all the help.

​73 de bill n7ie .. .
n7ie
Joe, another question:  How
Joe, another question:  How long would it take the system to recalculate/repopulate data across the mesh after the offending node is powered down? 
k1ky
k1ky's picture
Noise Floor "Masked" 2.4Ghz
We are experiencing some sort of signal blocking around the downtown Nashville area that affects the LQ/NLQ (one end has a good LQ and the other end varies down to 0 or just stays at 0 NLQ.  Signal strengths are good in the 15-30 db spread and the interesting thing is that the noise floor is still at -95!
There are brief periods where some links are usable, which indicates that whatever the offending source is coming from goes off the air or changes from time to time.  We hae not attempted to really get serious about tracking this down, but will do some more investigating soon.  This is observed from many stations that are 130 feet in the air pointed in all directions (3 sectors on each site).  I'm thinking possibly some high power 2-way or broadcast system that could be spraying trash all over the band.  This affects stations within 3 miles of the downtown area.
n7ie
I'm just hoping that a
I'm just hoping that a similar problem is not happening here!!  Good luck on resolving your issue.  73, bill .. .
k1ky
k1ky's picture
FoxHunt equipment for 2.4 Ghz
Wouldn't the band scanning tools (the graphical waterfall type) that come with the UBNT equipment work?  Are there any additional modules that could be added to our MESH nodes that would perform the same function to analyze signals on the band?
n7ie
Tom, I'm not familiar with
Tom, I'm not familiar with that particular band scanning tool.  Info would be helpful.  Thanks.

​bill n7ie .. .
k1ky
k1ky's picture
AirView
The app is called AirView - available to run on a UBNT device from within the factory firmware.
Here is a link describing the operation:
https://help.ubnt.com/hc/en-us/articles/204950584-airMAX-How-to-use-airV...
AE6XE
AE6XE's picture
Short answer, it should be

"How long would it take the system to recalculate/repopulate data across the mesh after the offending node is powered down"

Short answer, it should be order of seconds, not minutes.   A "wifi scan" will force a noise calibration to be triggered. 

Comments in the code that may shed light on this:

       /*
         * If beacons are stuck, the most likely cause is interference.
         * Triggering a noise floor calibration at this point helps the
         * hardware adapt to a noisy environment much faster.
         * To ensure that we recover from stuck beacons quickly, let
         * the baseband update the internal NF value itself, similar to
         * what is being done after a full reset.
         */

In a 20Mhz channel, the hardware maximum noise is ~-95dBm.  In 10Mhz it would be ~-98dBm (but I don't think the code compensates for channel width).  See next comment on why we nearly always see -95dBm as the noise floor).    To get a noise floor higher than -95dBm means the outgoing beacons are stuck, thinks the channel is always busy and can't xmit them =  a lot of noise on the channel OR you're doing 5Mhz channel with firmware prior to 3.16.1.0 (was doing 10 beacons per second) in combination with a lot of nodes = too much contention for the channel.  We lowered beacon rate in 3.16.1.0 to 2 per second.
                        /*
                         * Normally we limit the average noise floor by the
                         * hardware specific maximum here. However if we have
                         * encountered stuck beacons because of interference,
                         * we bypass this limit here in order to better deal
                         * with our environment.
                         */

        /*
         * If the noise floor seems normal for all chains, assume that
         * there is no significant interference in the environment anymore.
         * Re-enable the enforcement of the NF maximum again.
         */

n7ie
A "wifi scan" will force a noise calibration to be triggered. 
Joe, we're back to normal.  Doing the "wifi scan" forced a noise calibration (you'd know; I wouldn't, HI HI) and our mesh is back at -95 dBm (arbitrary) and I'm again seeing the nodes I should be seeing.  KUDOS and our profound gratitude!!  Tnx es 73 de bill n7ie .. .
AE6XE
AE6XE's picture
N7IE,   can you go to the
N7IE,   can you go to the setup->Admin page, at the bottom and click on the support download button?    Upload that file back here.     There's a SNR monitor program looking for unusual signal jumps among all the neighbors, which will then trigger this slight of hand channel change and back in the background to force these Noise Floor calibrations.  I want to see if we can capture the history to determine if there is a way to improve this program.  It was written, because we found under noise conditions, the mesh nodes could stop attempting NF calibrations thinking everything is tuned and optimized.    If the signal strength changes very slowly, this program won't detect the conditions and trigger the calibration.  I might need to force NF calibrations every 1 or 2 hours weather it needs it or not.

Given that things are back to normal after the wifi scan, this is probably what happened.  This behavior may be good for client only connecting to a single access point--likely the scenario the original writers were thinking of most.  But for multiple point connections, it's not desirable to tune out half the neighbors.   The zillion $$ question is how to detect when something is noise vs a legitimate neighbor, particularly when there are a lot of varying signal strengths from all the neighbors.

Joe AE6XE
k1ky
k1ky's picture
Support Files for Noise Floor Calibration
Do you want them here or is there a specific Ticket # that you would like for them to go to?  I have a node that has been working most of the day that is now NLQ0 for 2.8 hours back to the hospital site.  There is a neighboring unit in the same office that is still communicating with the hospital site.  I have support files from all 3 nodes available.  I believe I'm using a mewer rssi program on all of these nodes.
AE6XE
AE6XE's picture
K1KY,   The scenario for the
K1KY,   The scenario for the test version of the rssi program you have has been determined to be a different problem -- rssi is one of multiple triggers contributing to the root cause and showing symptoms.  There is a ticket on that issue.  Thanks, no further data is necessary as we have a test case setup to readily reproduce and are digging into it further.  

Joe AE6XE
k1ky
k1ky's picture
OK - standing by....
Very well Joe.  If ever you need further testing in a "harsh" environment, I have just the place for a "trial by fire"!!
n7ie
Roger, Joe.  Sorry I've been
Roger, Joe.  Sorry I've been busy this evening.  I'll see if I can capture the file (new ground for me) and post it here.  Catch you tomorrow.  bill n7ie .. .
n7ie
Joe, please check your email.
Joe, please check your email.  Thanks, bill n7ie .. .
n7ie
Rats!
Spoke too soon.  For a minute or so things were normal; now the problem has recurred.

Theme by Danetsoft and Danang Probo Sayekti inspired by Maksimer