After a full day of troubleshooting, WiscNet engineers narrowed down the possible cause to two line cards at the core device in one of our POP locations in Madison. We proceeded with an emergency maintenance that started early evening on 09/6 and was completed around 9:00 pm that day. We have since then received confirmation from many members impacted that the steps we took solved the problem. Our engineers are working with the hardware vendor to determined the cause of the problem. We are actively working with members still experience issues.
Some of our members reported problems with their bandwidth graph - https://graphs.wiscnet.net/ - which is likely due to the changes made last night. Our engineers are on it too. We are working as quickly as possible to fix this problem.
We thank you all again for our cooperation and patience through this process. Contact us at support@wiscnet.net if you have any further issues.
Posted Sep 07, 2018 - 09:39 CDT
Resolved
This incident has been resolved.
Posted Sep 07, 2018 - 08:54 CDT
Update
We have received many calls from members who were impacted confirming they see much better speeds after last night's emergency backbone maintenance. We will continue to monitor for another couple of hours before resolving this status notification. Please email us at support@wiscnet.net if you are still seeing problems.
We thank you again for your patience!
Posted Sep 07, 2018 - 08:28 CDT
Monitoring
WiscNet engineers moved your connections to different line cards in one of the Madison core devices. Tests indicated the maintenance fixed the issue. Please let us know if you are still seeing problems.
Posted Sep 06, 2018 - 21:25 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 18:26 CDT
Update
WiscNet engineers have narrowed down the possible cause of the problem to two line cards in one of the core devices in Madison. They are now prepping to drive to the site and perform more tests. If tests confirm the problem, connections will be physically moved to another line card, causing a short outage.
Posted Sep 06, 2018 - 18:20 CDT
Update
At this point we have excluded Charter as the culprit. Our engineers are testing line cards at our core devices to identify a possible hardware problem in the WiscNet backbone. Thank you for your patience.
Posted Sep 06, 2018 - 17:57 CDT
Update
Charter has confirmed circuits appear to be running clean and the NNI (interface handoff) shows no errors. WiscNet engineers continue to investigate possible issues on the WiscNet core.
Posted Sep 06, 2018 - 16:26 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 15:25 CDT
Update
Charter has confirmed the NNI (interface hand off) in our POP shows no errors. We have requested they test the physical circuits that are being delivered over that aggregated interface. Pending update at this point.
Posted Sep 06, 2018 - 15:20 CDT
Update
The number of members impacted by the problem is increasing. We believe the issue is now impacting all members who connect to WiscNet via the Charter handoff in one of WiscNet's Madison POP. This update is to notify all members off that interface as they all could be experiencing problems until the issue is addressed. Thanks for your patience. Member with two connections to WiscNet have been updated so that all traffic is going to one connection until the issue with Charter is resolved.
Posted Sep 06, 2018 - 14:19 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 13:44 CDT
Update
WiscNet has collected troubleshooting information from members impacted and is now in the process of reaching Charter to open a ticket. The issue is impacting different members in different ways, so it took a little longer than usual for our techs to amass as much information as possible. Stay tuned for more updates as they become available. Thank you for your patience.
Posted Sep 06, 2018 - 13:44 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 13:26 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 13:13 CDT
Update
We are continuing to investigate this issue.
Posted Sep 06, 2018 - 13:08 CDT
Investigating
Some of our members who are connected to WiscNet to our Madison pop via Charter as their transport provider are experiencing latency issues. WiscNet is investing the cause and will provide updates as we learn more on what going on.
Posted Sep 06, 2018 - 12:40 CDT
This incident affected: TEACH - Madison (Blackhawk Technical College, Dodgeville School District, Oregon School District, Portage Community School District), Baraboo School District, Beloit College, Beloit School District, Berlin Area School District, Big Foot UHS School District, Clinton Community School District, Edgerton School District, Elkhorn Area School District, Jefferson School District, Monona Grove School District, Monroe School District, Palmyra-Eagle Area School District, Rosendale-Brandon School District, St. Augustine University Parish, Slinger School District, Verona Area School District, Waupun School District, Wisconsin Dells School District, Wisconsin Economic Development Corporation - Madison, and TEACH - Milwaukee (Dodgeland School District, Marian University of Fond du Lac, Stoughton Area School District).