Had an issue the last few weeks where DNS across the network would randomly just drop/timeout for a period of a few minutes to an hour or so.
Across the entire network (at around lunctime) users would suddenly be unable to access certain websites but strangely applications like skype / outlook etc would still function normally.
I quickly diagnosed that it was a DNS issue as normal web browsing would fail but if i entered the domain’s IP address (using my phone and www.ip-lookup.net to find the domain ip) it would load instantly.
I tried bypassing the local DNS server by changing DNS manually to Google DNS (18.104.22.168 and 22.214.171.124) but this also didn’t work while the “blackout” was in progress.
Summary of symptoms…
- Connection not lost – existing connections stay alive (until bounced/rebooted)
- Pings to 126.96.36.199 (Google DNS) all successful
- Unable to resolve DNS through any DNS servers (Local DNS Server, Google DNS, Open DNS, ISP DNS etc) on any machine, laptop, server within the network.
- nslookup fails with Local DNS server, Google DNS, Open DNS and ISP DNS
- On router reboot DNS goes back to working OK for a period of time.
Restarting the router (a Draytek Vigor 2820 in this case) would fix the issue but usually DNS resolution would just drop off again sometime after (sometimes a few minutes, sometimes an hour or so).
After much frustration, and the realisation that it had to be something to do with the router, I rooted around in the Draytek Vigor 2820 settings to see what could be causing the DNS to drop off.
The penny dropped when looking at the DOS defence firewall settings. (Firewall > DOS Defence) in particular UDP Flood Defence
The threshold value was set to 150 packets/sec. Draytek recommend 2000 packets/sec for a 20mb connection. So roughly 100 packets/sec for each 1mb
For a more specific calculation see here: http://www.draytek.com/index.php?option=com_k2&view=item&id=5315&Itemid=293&lang=en
This particular connection is at best 8mb to i set this threshold value to 800. (See screengrab below)
I’m guessing everyone on their lunch breaks was hitting the connection hard with youtube, facebook, twitter, random browsing etc triggering the DoS defence filter.
This has now completely solved this incredibly annoying DNS issue.