461223 - Multiple Backbone Links showing issues in Chicago - Equinix
Incident
Report for WiscNet
Postmortem
Summary
On Thursday morning, the WiscNet peering router in Chicago unexpectedly became unstable. This instability caused logical and physical interfaces to go down at unexpected times. After 24 hours of troubleshooting with the vendor, re-slotting the midplane control boards resolved the issue.
Impact
Router issues caused intermittent traffic loss in 5-10 minute durations from 9:30 am on Thursday, November 11 Until 11:54 am on Friday November 12th. This event affected connectivity to the Internet and to WRIPS peers.
Router Ping loss
Router CPU
Resolution
The Chicago peering has been stable since the midplane Control Boards were reseated. WiscNet engineers are monitoring the router closely and continue to work with the vendor to determine the root cause of this issue.
An RMA for the possibly faulty line card is in process and two replacement line cards have been installed in the router.
Timeline
- 2018-11-01 09:32 Chicago peering router begins experiencing issues
- 2018-11-01 09:40 Members begin reporting network issues
- 2018-11-01 09:45 WiscNet engineers begin troubleshooting the issue; it was determined that a router line card with backbone interfaces was rebooting sporadically
- 2018-11-01 10:20 WiscNet engages Emergency Remote Hands to move fiber jumpers away from the impacted line card
- 2018-11-01 10:21 WiscNet opens a Priority 1 ticket with the vendor to troubleshoot the problematic line card; vendor determines the cause is a physically faulty line card and begins RMA process
- 2018-11-01 11:40 Emergency Remote Hands responds and interfaces are moved from problematic line card interfaces; WiscNet engineers power off the line card
- 2018-11-01 13:08 & 2018-11-01 13:41 All LAG (Link Aggregation) interfaces on the router flap, impacting traffic; WiscNet engineers re-engage troubleshooting efforts with the vendor; engineers discover a recurring syslog entry that first appeared when issues began
- 2018-11-01 14:00 WiscNet ships two replacement line cards to Chicago from WiscNet’s office; one to replace faulty card, one to provide additional redundancy and capacity
- 2018-11-01 16:51 LAG interfaces flap again while engineers are engaged in a troubleshooting session; engineers continue to troubleshoot with the vendor
- 2018-11-01 21:51 LAG interfaces flap again while the vendor is troubleshooting the router
- 2018-11-02 03:13 Vendor still has no resolution; WiscNet engineers decide to schedule an emergency router reboot at 05:00
- 2018-11-02 05:00 Router reboots successfully; problematic errors still appear in syslog; WiscNet engages remote hands to physically remove original problematic line card from the router
- 2018-11-02 06:20 Problematic line card removed, syslog messages still occuring; WiscNet re-engages the vendor
- 2018-11-02 09:29 Vendor requests reseating midplane Control Boards on router as next step and indicates that procedure is impactful; WiscNet engineers begin testing procedure in lab which shows that impact is less than when LAG interfaces flap unexpectedly
- 2018-11-02 11:30 WiscNet engages remote hands to perform Control Board reseating procedure
- 2018-11-02 11:54 After reseating 3rd and final Control Board, problematic syslog messages stop and the router stabilizes
Resolved
WiscNet backbone equipment has been stable. We will send a detailed report via this page shortly.
Update
Our engineers at WiscNet reported no issues in Chicago over the weekend. This is welcome news.
The team is still working out details with Juniper and monitoring changes along the way. We plan to send a report to you mid to end of the week pending any further adjustments to the backbone. Thank you
The team is still working out details with Juniper and monitoring changes along the way. We plan to send a report to you mid to end of the week pending any further adjustments to the backbone. Thank you
Monitoring
WiscNet engineers, along with Juniper and hands and eyes assistance at Equinix in Chicago have worked together to identify and correct the recent issues with instability in the WiscNet network. At the time of this update, the network has been stable for the last 60 minutes. WiscNet engineers are cautiously optimistic that the problem has been resolved, but will monitor the issue over the weekend. We will send a more detailed reason for outage report next week explaining what we learned about the problem and how we fixed it.
Once again, thank you for your patience. Give us a call if you have any questions or concerns. Here's to a good weekend!
Once again, thank you for your patience. Give us a call if you have any questions or concerns. Here's to a good weekend!
Update
WiscNet continues to troubleshoot and work with our vendor on a permanent fix.
Update
WiscNet continues to troubleshoot and work with our vendor on a permanent fix. WiscNet will be performing emergency maintenance that will be service impacting at 5:00AM local time on Friday, November 2nd on the core router at Equinix. There will be brief interruptions in service as traffic redirects.
Update
It's been a long day. WiscNet continues to troubleshoot issues with a core router in Chicago which is causing intermittent connectivity. WiscNet is actively troubleshooting and working with the vendor now and we will continue to provide updates as they become available.
Update
WiscNet continues to troubleshoot issues with a core router in Chicago which is causing intermittent connectivity. WiscNet is actively troubleshooting and working with the vendor now.
Update
WiscNet engineers are still investigating. We have a work-around in place for now in order to restore connectivity.
Update
We are continuing to work on a fix for this issue.
Update
Our engineers are all hands on deck regarding this issue. Working with Juniper, Equinix helping out with H&E. Re-routing of traffic was done manually so performance is much better at this point. We will hold off sending updates until case is resolved. If anything changes we will let you know ASAP. Thank you again!
Update
Our engineers are on the phone with vendor to correct the issue with the card in the device. We are also re-routing traffic between cards to alleviate the performance issues.
We will continue to update this page as more information becomes available.
Thank you for your patience
We will continue to update this page as more information becomes available.
Thank you for your patience
Identified
We are still seeing issues with the line card in Chicago at this point. Please accept our apologies for this unexpected problems. We are working as fast as we can to remedy the problem.
Investigating
We are continuing to investigate this issue.
Identified
Our engineers identified a possible cause for this issue. One of the line cards in our ISP router rebooted unexpectedly. Cards is coming back on line
Investigating
We are receiving a number of phone calls from our members with reports of slowness.
Our backbone shows an issue in the Equinix POP in Chicago. Our engineers are investigating.
Please stay tuned for updates as they become available
Our backbone shows an issue in the Equinix POP in Chicago. Our engineers are investigating.
Please stay tuned for updates as they become available
This incident affected: Abbotsford School District, Adams-Friendship Area School District, Albany School District, Algoma School District, Village of Allouez, Almond-Bancroft School District, Altoona City, Altoona School District, Alverno College, Antigo Unified School District, Appleton Area School District, Appleton Boys and Girls Club, City of Appleton, Appleton Performing Arts Center, Argyle School District, Arrowhead UHS School District, Ashwaubenon School District, Village of Ashwaubenon, Auburndale School District, Aurora University, Augusta School District, Bangor School District, Baraboo School District, Barneveld School District, Barron Area School District, Beecher-Dunbar-Pembine School District, Belleville School District, Bellin College Inc, Belmont Community School District, City of Beloit, Beloit College, Beloit School District, Benton School District, Berlin Area School District, Big Foot UHS School District, Birchwood School District, Black Hawk School District, Blackhawk Technical College, BloodCenter of Wisconsin, Bonduel School District, Boscobel Area School District, Bowler School District, Boyceville Community School District, Brighton #1 School District, Brillion School District, Bristol School District #1, City of Brookfield, Brown County, Brown County 2 Remote Site, Brown County 3 Remote Site, Brown County Jail, Brown County Law Enforcement, Brown County New Eye, Brown County PDCT, Brown County PDCT 2, Brown County Public Service Commission, Burlington Area School District, Cadott Community School District, Calumet County, Cambria-Friesland School District, Cambridge School District, Cameron School District, Campbellsport School District, Carroll University, Carthage College, Cassville School District, Catholic Memorial High School, CESA 1, CESA 2, CESA 3, CESA 4, CESA 5, CESA 6, CESA 7, CESA 8, CESA 9, CESA 10 - Chippewa Falls, CESA 10 - Madison, CESA 11, Chetek-Weyerhaeuser School District, Children Hospital and Health System, Chilton School District, Chippewa County, Chippewa Valley Technical College, Chippewa Falls Area Unified School District, Clear Lake School District, Clinton Community School District, Cochrane-Fountain City School District, Colby School District, Coleman School District, Colfax School District, Concordia University, Cornell School District, Crivitz School District, Cuba City School District, Cudahy School District, D C Everest Area School District, Dane County, Darlington Community School District, Deerfield Community School District, De Forest Area School District, De Pere School District, Delavan-Darien School District, Denmark School District, Dodgeland School District, Dodgeville School District, Door County, Dove Health Care, Dunn County, East Troy School District, Eastern Shores Library System - Monarch Library System, Eau Claire Area School District, Eau Claire Catholic Schools, City of Eau Claire, Eau Claire County, Edgar School District, Edgerton School District, Edgewood High School, Educators Credit Union, Elcho School District, Elk Mound Area School District, Elkhart Lake-Glenbeulah School District, Elkhorn Area School District, Ellsworth Community School District, Elmbrook School District, Elmwood School District, Erin School District, Fall Creek School District, Fall River School District, Family Peace Center, Flambeau School District, Florence County School District, City of Fond du Lac, Fond du Lac County, Fond du Lac School District, Fontana J8 School District, Fort Atkinson School District, Fox Point J2 School District, Fox Valley Lutheran High School, Fox Valley Technical College, Frederic School District, Freedom Area School District, Geneva J4 School District, Genoa City J2 School District, Germantown School District, Gillett School District, Gilmanton School District, Glenwood City School District, Good Shepherd Lutheran School, Goodman-Armstrong Creek School District, Grafton School District, Town of Grand Chute, Granton Area School District, Grantsburg School District, City of Green Bay, Green Bay Area School District, Green Bay Community Church, Green Lake County, Green Lake School District, City of Greenfield, Greenfield School District, Greenwood School District, Gresham School District, Hartford J1 School District, Hartford UHS School District, Hartland-Lakeside J3 School District, Hayward Community School District, Highland School District, HNR School District-Neosho, HNR School District- Rubicon, Holmen School District, Holy Apostles School, Holy Hill Area School District, Horicon School District, Howard Village, Hortonville Area School District, Howard-Suamico School District, Howards Grove School District, Hudson School District, Hurley School District, Hustisford School District, Inclusa, Independence First, Indianhead Federated Library System, Iola-Scandinavia School District, Iowa-Grant School District, Ithaca School District, Jackson County, City of Janesville, Jefferson County, Jefferson School District, Johnson Creek Elementary, Johnson Creek School District, Kaukauna Area School District, Kenosha County, Kenosha County Public Library System, Kenosha Unified School District, Kettle Moraine School District, Kewaunee School District, Kiel Area School District, Kimberly Area School District, Kohler School District, City of La Crosse, La Crosse School District, Lac Courte Oreilles Ojibwa Community College, Lac du Flambeau #1 School District, School District of Ladysmith, Lake Country School District, Lake Holcombe School District, Lake Mills Area School District, Lakeland College, Lakeland UHS School District, Lakeshore Technical College, Lakeshores Library System, Laona School District, Lawrence University, Lena School District, Lincoln County, Linn J6 School District, Little Chute Area School District, Village of Little Chute, Lodi School District, Lomira School District, Loyal School District, Luther High School, Luxemburg-Casco School District, Madison Area Technical College, City of Madison, Madison Metropolitan School District, Manawa School District, Manitowoc Public Utilities, Manitowoc School District, Maple Dale-Indian Hill School District, Marathon City School District, Marathon County, Marian University of Fond du Lac, Marinette City, Marinette County, Marinette School District, Marion School District, Markesan School District, Marquette University, Marquette University High School, Marshall School District, City of Marshfield, Marshfield Unified School District, Mayville School District, McDonell Area Catholic School District, McFarland School District, Medford Area Public School District, Medical College of Wisconsin, City of Menasha, Menasha Joint School District, Town of Menasha, Menasha Utilities, Menominee Indian School District, Menomonie Area School District, Mequon-Thiensville School District, Mercer School District, Meriter Hospital, Inc., Merton Community School District, Messmer Catholic, Mid-State Technical College, Mid-Wisconsin Federated Library System, Middleton-Cross Plains Area School District, Milton School District, Milwaukee Area Technical College, Milwaukee Art Museum, Milwaukee City Housing Authority, City of Milwaukee, Milwaukee County Federated Library System, Milwaukee Institute of Art and Design, Milwaukee Public Museum, Milwaukee School of Engineering, Mineral Point Unified School District, Minocqua J1 School District, Mishicot School District, Mondovi School District, Monona Grove School District, Monroe School District, Montello School District, Monticello School District, Moraine Park Technical College, Mosinee School District, Mukwonago School District, Muskego-Norway School District, Necedah Area School District, City of Neenah, Neenah Joint School District, Neillsville School District, City of New Berlin, New Berlin School District, New Glarus School District, New London School District, Niagara School District, Nicolet Area Technical College, Nicolet Federated Library System, Nicolet UHS School District, Norris Academy, North Central Health Care, North Fond du Lac School District, North Lakeland School District, Northcentral Technical College, Northeast Wisconsin Technical College, Northern Ozaukee School District, Northern Waters Library Service, Northland College, Northland International University, Northland Pines School District, Northwood School District, Norwalk-Ontario-Wilton School District, Oconto Falls Public School District, Oconto Unified School District, Omro School District, Oregon School District, Osceola School District, Oshkosh Area School District, City of Oshkosh, Osseo-Fairchild School District, Outagamie Waupaca Library System, Owen-Withee School District, Palmyra-Eagle Area School District, Parkview School District, Paris J1 School District, Pecatonica Area School District, Peshtigo School District, PESI, Pewaukee School District, Phelps School District, Phillips School District, Pier Wisconsin, Pittsville School District, City of Platteville, Platteville School District, Plum City School District, Plymouth Joint School District, Port Edwards School District, Port Washington-Saukville School District, Portage Community School District, Portage County, Poynette School District, Prairie du Chien Area School District, Prairie Farm Public School District, Prentice School District, Prescott School District, Price County, Princeton School District, Pulaski Community School District, Racine Lutheran High School, Racine Unified School District, Randall J1 School District, Raymond #14 School District, Reedsville School District, Rhinelander School District, Rib Lake School District, Rice Lake Area School District, Richmond School District, Rio Community School District, Ripon Area School District, Ripon College, River Falls School District, River Ridge School District, River Valley School District, Riverdale School District, Rock County, Rock Prairie Montessori, Rosendale-Brandon School District, Rosholt School District, Royal Credit Union, Rubicon J6 School District, Rural Health Care, Sacred Heart Hospital, Sacred Heart of Jesus and Mary School, Sacred Heart School of Theology, St. Augustine University Parish, St. Catherines High School, Saint Croix Central School District, Saint Croix Falls School District, St. Francis Xavier Catholic School Systems, St. Joseph's Hospital, St. Mary Catholic Schools, St. Mary's Springs Academy, St. Norbert College, St. Thomas Aquinas Parish School, St. Vincent Hospital, Salem School District, Sauk Prairie School District, Sevastopol School District, Sharon J11 School District, Shawano School District, Sheboygan Area School District, City of Sheboygan, Sheboygan County, Sheboygan Falls School District, Shell Lake School District, Shiocton School District, Shoreland Lutheran High School, Shullsburg School District, Silver Lake College, Siren School District, Slinger School District, Solon Springs School District, South Central Library System, Southern Door County School District, Southwest Health Center, Southwest Library System, Southwest Wisconsin Technical College, Southwestern Wisconsin School District, Spencer School District, Spooner Area School District, Spooner Health System, Spring Valley School District, Stanley-Boyd Area School District, Stevens Point Area Public School District, City of Stevens Point, Stone Bank School District, Stoughton Area School District, Stratford School District, Sturgeon Bay School District, Village of Suamico, Sun Prairie Area School District, City of Superior, Suring Public School District, Swallow School District, SWITCH - Southeastern Wisc Info Tech Exchange, Thorp School District, Three Lakes School District, Tigerton School District, Tomahawk School District, Tomorrow River School District, Tri-County Area School District, Trout Lake, Turtle Lake School District, Two Rivers Public School District, Union Grove J1 School District, Union Grove UHS School District, UW Platteville Real Estate Foundation, UW-Whitewater, Verona Area School District, Victory Christian Academy, Viterbo University, Wabeno Area School District, Walworth J1 School District, Washburn County, Washington-Caldwell School District, Waterford Graded J1 School District, Waterford UHS School District, Waterloo School District, Watertown Unified School District, City of Waukesha, Waukesha County, Waukesha County Federated Library System (Bridges Library System), Bridges Library System, Waukesha County Technical College, Waukesha School District, Waunakee Community School District, Waupaca County, Waupaca School District, Waupun School District, City of Wausau, Wausau School District, Wausaukee School District, Wautoma Area School District, Wauwatosa City, Wauzeka-Steuben School District, Webster School District, City of West Allis, West Allis-West Milwaukee School District, West De Pere School District, West Salem School District, Western Technical College, Westfield School District, Westosha Special Ed, Weyauwega-Fremont School District, White Lake School District, Whitefish Bay Dominican High School, Whitefish Bay School District, City of Whitewater, Whitewater Unified School District, Whitnall School District, School District of Wild Rose, Winding Rivers Library System, Winnebago County, Winnebago Lutheran Academy, Winneconne Community School District, Winnefox Library System, Winter School District, Wisconsin Center for the Blind and Visually Impaired, Wisconsin Dells School District, Wisconsin Department of Military Affairs, State of Wisconsin Dept of Public Instruction, Wisconsin Department of Transportation, Wisconsin Economic Development Corporation - Madison, Wisconsin Economic Development Corporation - Milwaukee, Wisconsin Heights School District, Northwood Technical College, State of Wisconsin Investment Board, Wisconsin Lutheran College, Wisconsin Lutheran High School, Wisconsin School for the Deaf, State of Wisconsin, Wisconsin Technical College System, Wisconsin Valley Library Service, Wonewoc-Union Center School District, Woodruff J1 School District, Wrightstown Community School District, Wisconsin Regional Peering (MERIT Network, Pennsylvania State University, SysNet, Indiana Giga POP, Northwestern University, University of Chicago, University of Iowa, University of Illinois, NorthernLights GigaPOP, UW SysNet, Illinois State University, Penn State University, KINBER, MOREnet, Iowa State University, University of Nebraska CSN, University of Nebraska Lincoln, University of Missouri, OneNet, GPN), and Transit Share (University of Wisconsin SysNet, University of Iowa, Penn State, University of Minnesota, Indiana University, MERIT, KINBER, Iowa State University).