Affected Services: Los Angeles, California - Incoming DNS traffic
Incident Times(s): November 27, 2015 ~12:00PM PST to 3:00PM PST (approx. 3 Hours of DNS traffic downtime)
Incident Response Team: All Global Frag Networks' Staff, in particular: The Network Engineering & System Administrative Team
Global Frag Networks received initial complaints of website/domain connectivity issues around 12:00PM PST. Part of our due diligence we immediately investigated the situation. After a couple hours of troubleshooting and testing. We revealed that our router instance was specifically blocking port 53 on the entire LA network. We were able to re-enable all incoming traffic to port 53 and restored DNS traffic into our network around 3:00PM PST.
Action Summary for Los Angeles, CA:
- Checked for name server connectivity issues, but was not the issue.
- Checked for any DDoS attacks on our DNS port/servers, but was not the issue.
- Checked our rDNS server itself, but was not the issue.
- Found the issue, by determining that incoming DNS traffic was unable to connect into our network, publicly.
- Located the source of the problem on our routers.
- Deactivated the policy statement that was blocking the DNS port 53 on our routers.
- Confirmed DNS traffic was able to reconnect into our network.
- Deleted the policy statement that was blocking the DNS port 53 on our routers.
- We will be investigating on how that policy statement was initially added into our routers.
- How did that policy statement get activated in the first place.
- We will insure that this problem will not occur again on our network.
- Network Operations has been briefed of the situation and were alerted to monitor all activities on our networking gear.
This DNS outage effected customers who relied on DNS (rDNS/fDNS) traffic into our network. Anyone who has connected a domain name (or relate services) into our network were affected. This did not have any impact on server hardware, power, and network connectivity. We do apologies for any inconvenience we may have caused, but we do our best to insure your utmost satisfaction with us. We thank you for your understanding and support during the inconvenience we had today.
If you have any other questions or concerns, please submit a ticket at https://www.globalfrag.com/billing/
Global Frag Networks
Friday, November 27, 2015
Powered by WHMCompleteSolution