Monitoring
WiscNet's monitoring indicates that the last TEACH circuit was fixed at approximately 11:52AM. If you are still experiencing connectivity issues please reach out to WiscNet support. WiscNet will continue to work with TEACH to provide an RFO.
Posted Oct 25, 2024 - 12:16 CDT
Update
Further analysis on the collateral damage to non-TEACH services on the WiscNet network. The TEACH outage this morning was causing a large number of circuits to flap on two WiscNet core routers. Our initial analysis is pointing to these two core routers not being able to keep up with the protocol churn from circuit flaps and packet loss on this many circuits. A large number of members transit these two routers to get to the internet as these routers are members of WiscNet's 2x100Gbps core backbone ring. These routers should be using their ASICs for forward most traffic not destined to the control plane. We'll be talking about mitigations internally to limit impact in the future.
Posted Oct 25, 2024 - 09:52 CDT
Update
The related TEACH NOC GLS ticket number for this event is 10685278.
Posted Oct 25, 2024 - 09:38 CDT
Update
The TEACH configuration rollback is complete. The majority of TEACH circuits are back online. Four remain, and WiscNet and TEACH are actively troubleshooting.
Posted Oct 25, 2024 - 09:19 CDT
Update
WiscNet has received multiple reports from non-TEACH connected members of performance issues and outages. WiscNet is investigating high CPU utilization on core routers that land TEACH circuits in both Madison and Milwaukee as protocol churn from circuits flapping appear to be causing performance issues.
Posted Oct 25, 2024 - 08:50 CDT
Identified
WiscNet is aware of and monitoring widespread circuit outages and packet loss remaining after a TEACH maintenance that occurred early in the morning on October 25th. WiscNet has been notified that TEACH is working on rolling back the changes at this time. WiscNet will update this Status Page incident with further information when we receive it.
Posted Oct 25, 2024 - 07:42 CDT
This incident affects: Abbotsford School District, Adams-Friendship Area School District, Albany School District, Algoma School District, All Saints Catholic School - Berlin, Village of Allouez, Almond-Bancroft School District, Altra Federal Credit Union, Altoona City, Altoona School District, Alverno College, Antigo Unified School District, Appleton Area School District, Appleton Boys and Girls Club, City of Appleton, Appleton Performing Arts Center, Argyle School District, Arrowhead UHS School District, Ashwaubenon School District, Village of Ashwaubenon, Auburndale School District, Aurora University, Augusta School District, Bangor School District, Baraboo School District, Barneveld School District, Barron Area School District, Beecher-Dunbar-Pembine School District, Belleville School District, Village of Bellevue, Bellin College Inc, Belmont Community School District, City of Beloit, Beloit College, Beloit School District, Benton School District, Berlin Area School District, Big Foot UHS School District, Birchwood School District, Black Hawk School District, Blackhawk Technical College, BloodCenter of Wisconsin, Bonduel School District, Boscobel Area School District, Bowler School District, Boyceville Community School District, Brighton #1 School District, Brillion School District, Bristol School District #1, City of Brookfield, Brown County, Brown County 2 Remote Site, Brown County 3 Remote Site, Brown County Jail, Brown County Law Enforcement, Brown County New Eye, Brown County PDCT, Brown County PDCT 2, Brown County Public Service Commission, Burlington Area School District, Cadott Community School District, Calumet County, Cambria-Friesland School District, Cambridge School District, Cameron School District, Campbellsport School District, Carroll University, Carthage College, Cassville School District, Catholic Memorial High School, CESA 1, CESA 2, CESA 3, CESA 4, CESA 5, CESA 6, CESA 7, CESA 8, CESA 9, CESA 10 - Chippewa Falls, CESA 10 - Madison, Chetek-Weyerhaeuser School District, Children Hospital and Health System, Chilton School District, Chippewa County, Chippewa Valley Technical College, Chippewa Falls Area Unified School District, Clear Lake School District, Clinton Community School District, Cochrane-Fountain City School District, Colby School District, Coleman School District, Colfax School District, Concordia University, Cornell School District, Crivitz School District, Cuba City School District, Cudahy School District, D C Everest Area School District, Dane County, Darlington Community School District, Deerfield Community School District, De Forest Area School District, City of De Pere, De Pere School District, Delavan-Darien School District, Denmark School District, Dodgeland School District, Dodgeville School District, Door County, Dove Health Care, Dunn County, East Troy School District, Eastern Shores Library System - Monarch Library System, Eau Claire Area School District, Eau Claire Catholic Schools, City of Eau Claire, Eau Claire County, Edgar School District, Edgerton School District, Edgewood High School, Educators Credit Union, Elcho School District, Elk Mound Area School District, Elkhart Lake-Glenbeulah School District, Elkhorn Area School District, Ellsworth Community School District, Elmbrook School District, Elmwood School District, Erin School District, Fall Creek School District, Fall River School District, Family Peace Center, Flambeau School District, Florence County School District, Fond du Lac Band Reservation, City of Fond du Lac, Fond du Lac County, Fond du Lac School District, Fontana J8 School District, Fort Atkinson School District, Fox City Performing Arts Center, Fox Point J2 School District, Fox Valley Lutheran High School, Fox Valley Technical College, City of Franklin, Franklin Public School District, Frederic School District, Freedom Area School District, Geneva J4 School District, General Mitchell International Airport, Genoa City J2 School District, Germantown School District, Gillett School District, Gilmanton School District, Glenwood City School District, Good Shepherd Lutheran School, Goodman-Armstrong Creek School District, Grafton School District, Town of Grand Chute, Granton Area School District, Grantsburg School District, City of Green Bay, Green Bay Area School District, Green Bay Community Church, Green Lake County, Green Lake School District, City of Greenfield, Greenfield School District, Greenwood School District, Gresham School District, Hartford J1 School District, Hartford UHS School District, Hartland-Lakeside J3 School District, Hayward Community School District, Highland School District, Hilbert School District, HNR School District-Neosho, HNR School District- Rubicon, Holmen School District, Holy Apostles School, Holy Hill Area School District, Horicon School District, Howard Village, Hortonville Area School District, Howard-Suamico School District, Howards Grove School District, Hudson School District, Hurley School District, Hustisford School District, Inclusa, Independence First, Indianhead Federated Library System, Iola-Scandinavia School District, Iowa-Grant School District, Ithaca School District, Jackson County, City of Janesville, Janesville School District, Jefferson County, Jefferson School District, Johnson Creek Elementary, Johnson Creek School District, Kaukauna Area School District, Kenosha County, Kenosha County Public Library System, Kenosha Unified School District, Kettle Moraine School District, Kewaskum School District, Kewaunee School District, Kiel Area School District, Kimberly Area School District, Kohler School District, City of La Crosse, La Crosse School District, Lac Courte Oreilles Ojibwa Community College, Lac du Flambeau #1 School District, School District of Ladysmith, Lake Country School District, Lake Holcombe School District, Lake Mills Area School District, Lakeland College, Lakeland UHS School District, Lakeshore Technical College, Lakeshores Library System, Lancaster Community Schools, Laona School District, Lawrence University, Lena School District, Lincoln County, Linn J6 School District, Little Chute Area School District, Village of Little Chute, Living Hope Lutheran School, Lodi School District, Lomira School District, Loyal School District, Luther High School, Luxemburg-Casco School District, Madison Area Technical College, City of Madison, Madison Metropolitan School District, Manawa School District, Manitowoc Public Utilities, Manitowoc School District, Maple Dale-Indian Hill School District, Marathon City School District, Marathon County, Marian University of Fond du Lac, Marinette City, Marinette County, Marinette School District, Marion School District, Markesan School District, Marquette University, Marquette University High School, Marshall School District, City of Marshfield, Marshfield Unified School District, Mayville School District, McDonell Area Catholic School District, McFarland School District, Medford Area Public School District, Medical College of Wisconsin, City of Menasha, Menasha Joint School District, Town of Menasha, Menasha Utilities, Menominee Indian School District, Menomonie Area School District, Mequon-Thiensville School District, Mercer School District, Meriter Hospital, Inc., Merton Community School District, Messmer Catholic, Mid-State Technical College, Mid-State Technical College - Stevens Point, Mid-Wisconsin Federated Library System, Middleton-Cross Plains Area School District, Milton School District, Milwaukee Area Technical College, Milwaukee Art Museum, Milwaukee City Housing Authority, City of Milwaukee, Milwaukee County Federated Library System, Milwaukee Institute of Art and Design, Milwaukee Public Museum, Milwaukee School of Engineering, Mineral Point Unified School District, Minocqua J1 School District, Mishicot School District, Mondovi School District, Monona Grove School District, Monroe School District, Montello School District, Monticello School District, Moraine Park Technical College, Mosinee School District, Mukwonago School District, Muskego-Norway School District, Necedah Area School District, City of Neenah, Neenah Joint School District, Neillsville School District, City of New Berlin, New Berlin School District, New Glarus School District, New London School District, Niagara School District, Nicolet Area Technical College, Nicolet Federated Library System, Nicolet UHS School District, Norris Academy, North Central Health Care, North Fond du Lac School District, North Lakeland School District, Northcentral Technical College, Northeast Wisconsin Technical College, Northern Ozaukee School District, Northern Waters Library Service, Northland College, Northland International University, Northland Pines School District, Northwood School District, Norwalk-Ontario-Wilton School District, Oconomowoc Area School District, Oconto County, Oconto Falls Public School District, Oconto Unified School District, Omro School District, Oregon School District, Osceola School District, Oshkosh Area School District, City of Oshkosh, Osseo-Fairchild School District, Outagamie Waupaca Library System, Owen-Withee School District, Pabst Theater Organization, Palmyra-Eagle Area School District, Parkview School District, Paris J1 School District, Pecatonica Area School District, Peshtigo School District, PESI, Pewaukee School District, Phelps School District, Phillips School District, Pier Wisconsin, Pittsville School District, City of Platteville, Platteville School District, Plum City School District, Plymouth Joint School District, Port Edwards School District, Port Washington-Saukville School District, Portage Community School District, Portage County, Poynette School District, Prairie du Chien Area School District, Prairie Farm Public School District, Prentice School District, Prescott School District, Price County, Princeton School District, Pulaski Community School District, Racine Lutheran High School, Racine Unified School District, Randall J1 School District, Randolph School District, Raymond #14 School District, Reedsville School District, Rhinelander School District, Rib Lake School District, Rice Lake Area School District, Richmond School District, Rio Community School District, Ripon Area School District, Ripon College, River Falls School District, River Ridge School District, River Valley School District, Riverdale School District, Rock County, Rock Prairie Montessori, Rosendale-Brandon School District, Rosholt School District, Royal Credit Union, Rubicon J6 School District, Rural Health Care, Sacred Heart Hospital, Sacred Heart of Jesus and Mary School, Sacred Heart School of Theology, St. Augustine University Parish, St. Catherines High School, Saint Croix Central School District, Saint Croix Falls School District, St. Francis Xavier Catholic School Systems, St. Joseph's Hospital, St. Mary Catholic Schools, St Mary's Hospital Medical Center, St. Mary's Springs Academy, St. Norbert College, St. Thomas Aquinas Parish School, St. Vincent Hospital, Salem School District, Sauk Prairie School District, Seneca School District, Sevastopol School District, Sharon J11 School District, Shawano School District, Sheboygan Area School District, City of Sheboygan, Sheboygan County, Sheboygan Falls School District, Shell Lake School District, Shiocton School District, Shoreland Lutheran High School, Shullsburg School District, Silver Lake College, Siren School District, Slinger School District, Solon Springs School District, South Central Library System, Southern Door County School District, Southwest Health Center, Southwest Library System, Southwest Wisconsin Technical College, Southwestern Wisconsin School District, Sparta Area School District, Spencer School District, Spooner Area School District, Spooner Health System, Spring Valley School District, Stanley-Boyd Area School District, Stevens Point Area Public School District, City of Stevens Point, Stone Bank School District, Stoughton Area School District, Stratford School District, Sturgeon Bay School District, City of Sturgeon Bay, Village of Suamico, Sun Prairie Area School District, City of Superior, Suring Public School District, Swallow School District, SWITCH - Southeastern Wisc Info Tech Exchange, Thorp School District, Three Lakes School District, Tigerton School District, Tomahawk School District, Tomorrow River School District, Tri-County Area School District, Trout Lake, Turtle Lake School District, Two Rivers Public School District, Union Grove J1 School District, Union Grove UHS School District, UW-Whitewater, Verona Area School District, Victory Christian Academy, Victory Christian Academy - 2, Viterbo University, Wabeno Area School District, Walworth J1 School District, Washburn County, Washington-Caldwell School District, Waterford Graded J1 School District, Waterford UHS School District, Waterloo School District, Watertown Unified School District, City of Waukesha, Waukesha County, Waukesha County Federated Library System (Bridges Library System), Bridges Library System, Waukesha County Technical College, Waukesha School District, Waunakee Community School District, Waupaca County, Waupaca School District, Waupun School District, City of Wausau, Wausau School District, Wausaukee School District, Wautoma Area School District, Wauwatosa City, Wauzeka-Steuben School District, Webster School District, City of West Allis, West Allis-West Milwaukee School District, West De Pere School District, West Salem School District, Western Technical College, Westfield School District, Westosha Special Ed, Weyauwega-Fremont School District, White Lake School District, Whitefish Bay Dominican High School, Whitefish Bay School District, City of Whitewater, Whitewater Unified School District, Whitnall School District, School District of Wild Rose, Winding Rivers Library System, Winnebago County, Winnebago Lutheran Academy, Winneconne Community School District, Winnefox Library System, Wisconsin Center District, Wisconsin Center for the Blind and Visually Impaired, Wisconsin Dells School District, Wisconsin Department of Military Affairs, State of Wisconsin Dept of Public Instruction, Wisconsin Department of Transportation, Wisconsin Economic Development Corporation - Madison, Wisconsin Economic Development Corporation - Milwaukee, Wisconsin Heights School District, Northwood Technical College, State of Wisconsin Investment Board, Wisconsin Lutheran College, Wisconsin Lutheran High School, Wisconsin School for the Deaf, State of Wisconsin, Wisconsin Valley Library Service, Wonewoc-Union Center School District, Woodruff J1 School District, Wrightstown Community School District, WiscNet Headquarters, City of Watertown and Wisconsin Regional Peering (MERIT Network, Pennsylvania State University, SysNet, Indiana Giga POP, Northwestern University, University of Chicago, University of Iowa, University of Illinois, NorthernLights GigaPOP, UW SysNet, Illinois State University, Penn State University, KINBER, MOREnet, Iowa State University, University of Nebraska CSN, University of Nebraska Lincoln, University of Missouri, OneNet, GPN).