Postmortem
Date & Time of Incident: 11:03 AM 8/25/2026
Duration: 5 minutes
Technician(s): Carter Jensen
Review Date: 8/26/2026
Brief Summary: Alarms came in for sit at 11:03 AM and cleared at 10:08 AM. Three router line cards, two in Milwaukee and one in Madison failed.
Root Cause: Software bug on equipment
Posted Aug 26, 2026 - 11:34 CDT
Resolved
This incident has been resolved.
Posted Aug 26, 2026 - 11:26 CDT
Update
The router vendor has identified a software bug and has recommended a configuration workaround that we have applied. We will be evaluating alternative code trains while this mitigation is in place. There is no expected impact at this time.
2026-08-25 11:03:14 CDT - outage start time
2026-08-25 11:07:37 CDT - outage end time
2026-08-25 11:24 CDT - high priority ticket created with the router vendor
2026-08-25 15:45 CDT - ticket escalated and set to critical due to lack of response from the vendor
2026-08-25 15:54 CDT - second router added to the router vendor ticket
2026-08-25 17:25 CDT - vendor confirmed a software bug and a mitigation
2026-08-25 17:35 CDT - mitigation applied to the all routers impacted by this bug
Posted Aug 25, 2026 - 17:36 CDT
Update
An additional router line card in Madison is impacted and we've updated the case to critical with the router vendor for next steps.
Summary: three router line cards, two in Milwaukee and one in Madison, have crashed and rebooted today. They automatically recovered and our technicians are working with the vendor to determine the root cause of the outage.
2026-08-25 11:03:14 CDT - outage start time
2026-08-25 11:07:37 CDT - outage end time
2026-08-25 11:24 CDT - high priority ticket created with the router vendor
2026-08-25 15:45 CDT - ticket escalated and set to critical due to lack of response from the vendor
2026-08-25 15:54 CDT - second router added to the router vendor ticket
Posted Aug 25, 2026 - 16:13 CDT
Monitoring
Summary: two line cards in a downtown Milwaukee core router crashed at the same time. They automatically recovered and our technicians are working with the vendor to determine the root cause of the outage.
2026-08-25 11:03:14 CDT - outage start time
2026-08-25 11:07:37 CDT - outage end time
2026-08-25 11:24 CDT - high priority ticket created with the router vendor
Posted Aug 25, 2026 - 13:16 CDT
This incident affected: TEACH - Milwaukee (Algoma School District, All Saints Catholic School - Berlin, Arrowhead UHS School District, Bellin College Inc, Bonduel School District, Bowler School District, Waukesha County Federated Library System (Bridges Library System), Bridges Library System, Brighton #1 School District, Bristol School District #1, Burlington Area School District, Catholic Memorial High School, CESA 1, CESA 6, CESA 7, Chilton School District, Concordia University, Elcho School District, Elkhart Lake-Glenbeulah School District, Erin School District, Fox Point J2 School District, Freedom Area School District, Geneva J4 School District, Genoa City J2 School District, Grafton School District, Granton Area School District, Grantsburg School District, Holmen School District, Horicon School District, Hortonville Area School District, Howards Grove School District, Kettle Moraine School District, Kewaunee School District, Kiel Area School District, Kimberly Area School District, Kohler School District, Lac du Flambeau #1 School District, Lake Country School District, Lakeshore Technical College, Linn J6 School District, Living Hope Lutheran School, Maple Dale-Indian Hill School District, Marian University of Fond du Lac, Medford Area Public School District, Menominee Indian School District, Mequon-Thiensville School District, Merton Community School District, Milton School District, Milwaukee County Federated Library System, Mishicot School District, Monarch Library System, New London School District, Nicolet UHS School District, Northcentral Technical College, Northern Ozaukee School District, Northland Pines School District, Outagamie Waupaca Library System, Paris J1 School District, Pewaukee School District, Phillips School District, Pittsville School District, Plymouth Joint School District, Port Washington-Saukville School District, Prairie Lakes Library System, Prairie Farm Public School District, Prentice School District, Racine Lutheran High School, Racine Unified School District, Randall J1 School District, Raymond #14 School District, Reedsville School District, Rib Lake School District, Sharon J11 School District, Shawano School District, Sheboygan Falls School District, Shoreland Lutheran High School, South Milwaukee School District, Spring Valley School District, St. Mary Catholic Schools, Stone Bank School District, Sturgeon Bay School District, Swallow School District, Three Lakes School District, Union Grove J1 School District, Union Grove UHS School District, Washington-Caldwell School District, Waterford Graded J1 School District, Waterford UHS School District, Waupaca School District, Whitefish Bay Dominican High School, Wisconsin Lutheran College, Wisconsin School for the Deaf, Wrightstown Community School District), TEACH - Madison (Dodgeville School District), and Hartford UHS School District, Marquette University, Medical College of Wisconsin, Moraine Park Technical College, Ripon Area School District, Sacred Heart Seminary and School of Theology.