INC0434638 - Large outage impacting multiple sites
Resolved
This incident has been resolved.
Posted Sep 23, 2024 - 10:45 CDT
Monitoring
Hello,
Here is the RFO provided for this event on 8-9-24:
"We had an outage on Friday, Aug. 9. It started at approximately 2:15pm and cleared at approximately 6:50pm. Following are sites that opened auto tickets. There may be others that were affected, but did not generate tickets. This was an impairment, not down hard. However, sites were getting packet loss of 80-99% which made them unusable. Up and recovery times for sites are not consistent with one another.
The root cause was a loop introduced into the network while provisioning the new ASEoD circuit for Hayward High School. The circuit was turned up, but not ready to be active in the network. Sites reported: (122)."
If there are any questions or concerns please reach out to WiscNet:
WiscNet Support
-------------------------------------
Email: support@wiscnet.net
Phone: (608)-442-6761 option 2
Hours: 7:45 AM - 4:45 PM, M-F
WiscNet NOC
-------------------------------------
Email: noc@wiscnet.net
Phone: (608)-442-6761 option 1
Hours: 24 x 7
Posted Aug 12, 2024 - 15:22 CDT
Update
We received update from the client that all alarms are clear and services should be restored for all sites. We confirmed that every site has good connectivity after the fix. This ticket will be closed in 24 hours if there are no further issues. Please contact us if you experience any more problems.
Posted Aug 09, 2024 - 20:26 CDT
Update
Wiscnet is continuing to work with our partner on this outage. At this tome we have no ETR. We will provide more details as they become available.
Posted Aug 09, 2024 - 18:54 CDT
Update
We are continuiing to work with our partner provider to resolve this issue.
Posted Aug 09, 2024 - 16:53 CDT
Update
The WiscNet NOC is aware of a network issue, causing an outage or significant packet loss for a large number of members. The provider is aware of and investigating the issue. We will be following up with them, and will provide updates as additional information is available.
Posted Aug 09, 2024 - 16:26 CDT
Update
The WiscNet NOC is aware of a network issue, causing an outage or significant packet loss for a large number of members. The provider is aware of and investigating the issue. We will be following up with them, and will provide updates as additional information is available.
Posted Aug 09, 2024 - 15:35 CDT
Investigating
The WiscNet NOC is aware of a network issue, causing an outage or significant packet loss for a large number of members. The provider is aware of and investigating the issue. We will be following up with them, and will provide updates as additional information is available.
Posted Aug 09, 2024 - 15:34 CDT
This incident affected: Adams-Friendship Area School District, Algoma School District, Antigo Unified School District, Arrowhead UHS School District, Bangor School District, Blackhawk Technical College, Brighton #1 School District, Catholic Memorial High School, CESA 1, CESA 7, CESA 9, Edgar School District, Edgewood High School, Elcho School District, Ellsworth Community School District, Erin School District, Freedom Area School District, Goodman-Armstrong Creek School District, Grafton School District, Green Lake School District, Hartford J1 School District, Horicon School District, Ithaca School District, Kewaunee School District, Kimberly Area School District, Kohler School District, Lac du Flambeau #1 School District, Lake Country School District, Lake Mills Area School District, Linn J6 School District, Living Hope Lutheran School, Marquette University High School, Mequon-Thiensville School District, Merton Community School District, Milton School District, New London School District, Norris Academy, Northland Pines School District, Norwalk-Ontario-Wilton School District, Paris J1 School District, Peshtigo School District, Phelps School District, Racine Lutheran High School, Rib Lake School District, Rosholt School District, Sacred Heart of Jesus and Mary School, St. Catherines High School, Sheboygan Falls School District, Shoreland Lutheran High School, Sturgeon Bay School District, Swallow School District, Tomorrow River School District, Union Grove J1 School District, Union Grove UHS School District, Walworth J1 School District, Waterloo School District, Waupaca School District, West Salem School District, White Lake School District, Whitefish Bay Dominican High School, School District of Wild Rose, Wisconsin Center for the Blind and Visually Impaired, Wisconsin Heights School District, Wisconsin Lutheran College, Wisconsin School for the Deaf, and Wrightstown Community School District.