INC0354667 Large scale outage impacting multiple members

Incident Report for WiscNet

Resolved

All members members connections remain stable. The outage was caused by a broadcast storm caused by a loop in the network. Once this was resolved, we also saw improvements on widespread packet loss issue, which has been an ongoing issue for a small subset of members - we are tracking the packet loss issue on this Status Page: https://status.wiscnet.net/incidents/t7408mmn5gmq
Posted Apr 11, 2024 - 16:10 CDT

Monitoring

The transport provider isolated multiple potential sources of the problem and corrected each. Their team and WiscNet observed sustained recovery of all affected sites. We will continue to monitor for stability and will re-engage with the provider if issues resurface.
Posted Apr 05, 2024 - 23:30 CDT

Identified

The transport provider has identified an issue on their network causing a core interface to shutdown, they are attempting to resolve the root cause.
Posted Apr 05, 2024 - 19:27 CDT

Update

Changed the status to critical, added sites which needed components created. Noting GLS (TEACH Network NOC) has ticket 10137563 open for this incident. We are working with GLS and AT&T to identify the issue and will provide updates as they become available.
Posted Apr 05, 2024 - 16:01 CDT

Investigating

WiscNet is investigating an outage impacting a large number of members. A cause has not yet been identified, but all affected members connect using the TEACH network. We will provide updates here as we get more information.
Posted Apr 05, 2024 - 15:45 CDT
This incident affected: TEACH - Milwaukee (Algoma School District, All Saints Catholic School - Berlin, Arrowhead UHS School District, Brighton #1 School District, Catholic Memorial High School, CESA 1, CESA 6, Elcho School District, Erin School District, Freedom Area School District, Geneva J4 School District, Hartford J1 School District, Horicon School District, Hortonville Area School District, Kewaunee School District, Kimberly Area School District, Kohler School District, Lac du Flambeau #1 School District, Lake Country School District, Linn J6 School District, Living Hope Lutheran School, Good Shepherd Lutheran School, Marquette University High School, Merton Community School District, Milton School District, New London School District, Nicolet UHS School District, Northland Pines School District, Paris J1 School District, Port Washington-Saukville School District, Racine Lutheran High School, Sacred Heart School of Theology, Sheboygan Falls School District, Shoreland Lutheran High School, St. Catherines High School, Sturgeon Bay School District, Swallow School District, Union Grove J1 School District, Union Grove UHS School District, White Lake School District, Whitefish Bay Dominican High School, Wisconsin School for the Deaf, Wrightstown Community School District), TEACH - Madison (Beecher-Dunbar-Pembine School District, Cambria-Friesland School District, CESA 9, Crivitz School District, Deerfield Community School District, East Troy School District, Edgar School District, Edgewood High School, Ellsworth Community School District, Goodman-Armstrong Creek School District, Green Lake School District, Lake Mills Area School District, Norris Academy, North Lakeland School District, Phelps School District, Poynette School District, Princeton School District, Rio Community School District, Rock Prairie Montessori, Sacred Heart of Jesus and Mary School, School District of Wild Rose, Tigerton School District, Waterloo School District, Wisconsin Center for the Blind and Visually Impaired), and Manitowoc School District, Richmond School District.