This incident has been resolved. We are tracking the widespread packet loss issue in this status page - INC0341032 | WiscNet | Multiple sites Impacted | PacketLoss - https://status.wiscnet.net/incidents/hg7p7wn7m4c4. Please see that incident for further updates on the packet loss issue.
Posted Apr 01, 2024 - 16:47 CDT
Monitoring
After the reboot of the Ciena equipment by AT&T engineers, most sites regained connectivity, but we are still seeing packet loss at various degrees. The packet loss is widespread and it involves more schools. The work will continue and updates will become available on a regular basis.
Posted Apr 01, 2024 - 11:51 CDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 01, 2024 - 10:37 CDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 01, 2024 - 10:30 CDT
Update
Just as we asked you to prepare for the worst-case scenario, there is a significant update. Alarms started to clear after AT&T rebooted Ciena equipment. However, monitoring and troubleshooting of ongoing packet loss and members who are still down will continue through Monday. We will provide updates throughout the day.
If you are experiencing issues or have any questions, please contact our Support Team at support@wiscnet.net or call 608-442-6761 option 2.
Thank you again for your patience The WiscNet Team
Posted Mar 31, 2024 - 16:56 CDT
Update
Dear WiscNet Members
GLS and multiple ATT teams continue to narrow down the source of the problem. At this point they see broadcast ARP packets being lost between two Juniper routers deep within the TEACH/ATT network. There are reportedly Ciena devices on that path also. Both vendor’s support desks have been engaged. As troubleshooting continues today, there may be additional instability in the TEACH network as more drastic measures are attempted, such as rebooting core devices.
While the symptoms shared in prior updates remain consistently repeatable, there is no clear idea as of Sunday at 1:30 pm what the solution will be or how long it will take to find it. At this point we want affected members to know we suspect the problem may well continue into Monday and to prepare accordingly.
Please know that WiscNet is doing everything we can to assist ATT and GLS in finding a resolution. All parties have been on day long calls well into the night looking at enormous logs from several different points in the network. Members have been on-site during their spring break to let technicians in to offer onsite visibility. All angles are being covered. Yet here we are. This is certainly an unprecedented situation.
We will continue to keep you posted throughout the day, even if there is no progress so you know where the situation stands. Thank you for your understanding.
If you have any questions between now and Monday when we are back in the office, please don't hesitate to call our 24x7 NOC and leave a message with so we can get back to you as soon as possible.
Thank You The WiscNet Team
Posted Mar 31, 2024 - 14:16 CDT
Update
Dear WiscNet Members
GLS and multiple ATT teams continue to narrow down the source of the problem. At this point they see broadcast ARP packets being lost between two Juniper routers deep within the TEACH/ATT network. There are reportedly Ciena devices on that path also. Both vendor’s support desks have been engaged. As troubleshooting continues today, there may be additional instability in the TEACH network as more drastic measures are attempted, such as rebooting core devices.
While the symptoms shared in prior updates remain consistently repeatable, there is no clear idea as of Sunday at 1:30 pm what the solution will be or how long it will take to find it. At this point we want affected members to know we suspect the problem may well continue into Monday and to prepare accordingly.
Please know that WiscNet is doing everything we can to assist ATT and GLS in finding a resolution. All parties have been on day long calls well into the night looking at enormous logs from several different points in the network. Members have been on-site during their spring break to let technicians in to offer onsite visibility. All angles are being covered. Yet here we are. This is certainly an unprecedented situation.
We will continue to keep you posted throughout the day, even if there is no progress so you know where the situation stands. Thank you for your understanding.
If you have any questions between now and Monday when we are back in the office, please don't hesitate to call our 24x7 NOC and leave a message with so we can get back to you as soon as possible.
Thank You The WiscNet Team
Posted Mar 31, 2024 - 14:10 CDT
Update
AT&T and GLS engineers have discovered that some unicast traffic is unable to traverse the entire transport network and that some MAC addresses are flapping between VPLS paths. These are symptoms and the root cause is still being investigated. Packet captures are being performed across the path in an attempt to isolate the issue.
Posted Mar 30, 2024 - 20:00 CDT
Update
Tier 3 AT&T, GLS, router vendor support staff, and WiscNet teams continue to work to identify the root cause of the issue. Down circuits are still experiencing a lack of end-to-end connectivity. Packet captures are being run on each piece of equipment along the paths to attempt to identify the root cause. Troubleshooting with all players will continue Saturday morning.
Posted Mar 29, 2024 - 21:47 CDT
Update
GLS & AT&T engineers have further escalated the situation in their organizations and opened a high-severity ticket with the equipment manufacturer. All parties continue to troubleshoot the situation. Thank you for your patience during this difficult situation. Please contact our NOC at 608-442-6761, option 1 if you have questions over the weekend.
Posted Mar 29, 2024 - 17:33 CDT
Update
GLS and AT&T engineers continue to dive into the network configuration to find where the issue with irregular packet forwarding resides. A technician has been dispatched to one of the impacted schools to add a layer of onsite troubleshooting to the investigation. We will continue to provide updates throughout this process until resolution. Thank you for your patience through this difficult situation. Please contact us at support@wiscnet.net or call us at 608-442-6761 option 2 if you have any questions.
Posted Mar 29, 2024 - 10:29 CDT
Update
.
Posted Mar 28, 2024 - 20:19 CDT
Update
Greetings,
Investigation into this outage remains ongoing at this time. The NOC has engaged WiscNet's Engineering team for further assistance / review, and we continue to follow up- and escalate with our partner provider regularly for updates on their investigation.
Thank you for your continued patience as we work to resolve this issue.
Please contact the WiscNet NOC if you have any further questions.
GLS and AT&T technicians have identified irregular packet forwarding behavior on TEACH circuits and have been able to replicate the issue. Technicians continue to troubleshoot to isolate the root cause of the issue. We are looking for volunteers in rebooting equipment on site and/or meeting an ATT technician for further troubleshooting Please contact our support team 608-442-6761 option 2 so we can help coordinate. Thank you!
Posted Mar 28, 2024 - 19:17 CDT
Update
The conference call is still in progress. The engineers continue to take deep dives into many configuration layers of the network in search of the reason for the loss of service. At this time, the outage is still ongoing with no ETA for resolution. We are deeply sorry for the impact to our members and their users, staff, faculty and users. We are doing the best we can to assist all parties in solving this problem. Again, please contact our Support Team at support@wiscnet.net or call us at 608-442-6761, option two with questions and concerns. We are truly grateful for your understanding and patience.
Posted Mar 28, 2024 - 16:47 CDT
Update
Several engineers meeting on a conference call, 2:30 this afternoon. This post will continue to be updated. Thank you for much again, for your patience. Please contact our Support Team at 608-442-6761 option 2 or email us at support@wiscnet.net with questions or comments about this service interruption.
Posted Mar 28, 2024 - 14:17 CDT
Update
Engineers from GLS and WiscNet have been systematically checking the several points within the network. Presently, moving towards the AT&T Core network. We will continue to provide updates. Thank you for your patience. You may also all email or call us at 608-442-6761 option 2.
Posted Mar 28, 2024 - 13:09 CDT
Update
The list of sites was updated to include all of the sites affected by this incident, WiscNet staff are investigating and will update this incident as we have more information.
Posted Mar 28, 2024 - 10:08 CDT
Update
Greetings,
The NOC has been made aware of an active outage taking place which is affecting your service. At this time, the NOC is working directly with our local provider to restore your services as quickly as possible with Investigation still underway. Once further details are provided, the NOC will ensure to forward those over for your review. Please continue to stand-by as we Investigate cause of outage.
Respectfully, WiscNet NOC
Posted Mar 28, 2024 - 09:55 CDT
Update
Greetings,
The NOC has been made aware of an active outage taking place which is affecting your service. At this time, the NOC is working directly with our local provider to restore your services as quickly as possible with Investigation still underway. Once further details are provided, the NOC will ensure to forward those over for your review. Please continue to stand-by as we Investigate cause of outage.
Respectfully, WiscNet NOC
Posted Mar 28, 2024 - 09:53 CDT
Investigating
Greetings,
The NOC has been made aware of an active outage taking place which is affecting your service. At this time, the NOC is working directly with our local provider to restore your services as quickly as possible with Investigation still underway. Once further details are provided, the NOC will ensure to forward those over for your review. Please continue to stand-by as we Investigate cause of outage.
Respectfully, WiscNet NOC
Posted Mar 28, 2024 - 09:36 CDT
This incident affected: Algoma School District, Arrowhead UHS School District, Brighton #1 School District, Good Shepherd Lutheran School, Horicon School District, Lake Country School District, Linn J6 School District, Merton Community School District, Nicolet UHS School District, Paris J1 School District, Racine Lutheran High School, Sacred Heart School of Theology, St. Catherines High School, Shoreland Lutheran High School, Swallow School District, Union Grove J1 School District, Union Grove UHS School District, Whitefish Bay Dominican High School, Wisconsin Lutheran College, and Wisconsin School for the Deaf.