CrowdStrike Issue

Friday, July 19 - 4:33 PM

Update - Global Software Issue Impacting Devices at Western

We would like to provide an update to the global software issue with Crowdstrike that impacted some Western devices and services today. Please share this update widely within your units.

We have made significant progress in restoring devices and services across the university. We are pleased to inform you that the majority of impacts from the Crowdstrike issue have been resolved. 

There are some areas that are still experiencing some local residual issues, and efforts continue to address those. Some may continue into early next week.

We have continued to provide university leadership, key staff, and the community with updates through email and social media. 

If your units are still experiencing issues, please contact the WTS Helpdesk. Our team continues to be available to address any further issues that may arise.

Overall, we are in a good position, we are thankful to our Western Technology Services teams and campus IT partners who have been working diligently today to ensure a smooth recovery. They have done some excellent work in responding to the issue on behalf of the Western community.

Friday, July 19, 2024 - 12:56 PM

Update - Global Software Issue Impacting Devices at Western

We would like to provide an update to the global software issue with Crowdstrike impacting some Western devices and services. Please share this update widely within your units.

As communicated, an issue within an update to Crowdstrike’s software is causing inoperability in some Windows devices across campus. Many industries across the world are experiencing the same outages today.

Efforts are progressing well towards restoration of devices and services across campus. All Western Technology Services (WTS)-supported central services are operational, including Peoplesoft, Brightspace, and others. We recognize that some units on campus may still be experiencing service impacts, and we are continuing to support those units in their remediation efforts.

Impact across campus is varied:

  • Some reports of impact to local devices, servers, computer labs, and other technology have been reported.
  • 3rd party software like payment processors may have also been impacted by the global outage, causing service disruptions at Western for units using them.
  • General Use classrooms across campus have also been impacted differentially, and our Classroom Technology Team is working to restore devices in these spaces. Devices have been restored in spaces that had events booked today.

Administrators should check on devices in their units to ensure they are all functioning correctly. WTS has developed documentation for resolving issues locally, which has been shared with the campus IT community. WTS has also provided campus IT staff with a list of machines running Crowdstrike in their units to help with remediation efforts.

Users are encouraged to report ongoing issues to the WTS Helpdesk or contact their local IT support teams for further investigation.

Communications have been shared with leadership and key staff across campus, and community updates have been shared on social media and on the WTS website.

Updates will continue to be provided as they are available.

Friday, July 19, 2024 - 5:46 AM

Global Software Outage Impacting Many Devices at Western

We want to inform you of an issue impacting many devices and services across campus overnight. 

The latest version of Crowdstrike security software, which is installed on thousands of workstations and servers at Western, has a bug which causes those devices to become temporarily inoperable. 

This issue is not specific to Western and is not a security incident or cyberattack.

Our Actions:

  • The provider of the software has provided instructions on remediation. Since we became aware of the issue around 2:30 a.m., Western Technology Services has been working to identify and remediate affected devices under our direct management and restore related services.
  • We are working to communicate this issue more broadly, including to our faculty and distributed IT colleagues, and to provide support to bring other impacted devices back online as soon as possible.

Updates will be provided periodically as details become available.

Friday, July 19, 2024 - 5:41 am

WTS is continuing efforts to get central servers restored to operation. CrowdStrike have posted updates to their Tech Alert which have been included below.

There will be an 8 AM TUMS call to provide updates and answer any questions, details on the call will be provided prior to the start time. 

Summary

CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor.

Details

  • Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor.
  • This issue is not impacting Mac- or Linux-based hosts
  • Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.

Current Action

  • CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.
  • If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue:

Workaround Steps for individual hosts:

  • Reboot the host to give it an opportunity to download the reverted channel file.  If the host crashes again, then:Boot Windows into Safe Mode or the Windows Recovery Environment
    • Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
    • Locate the file matching “C-00000291*.sys”, and delete it. 
    • Boot the host normally.

Note: Bitlocker-encrypted hosts may require a recovery key.

Workaround Steps for public cloud or similar environment:

  • Detach the operating system disk volume from the impacted virtual server
  • Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes
  • Attach/mount the volume to to a new virtual server
  • Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  • Locate the file matching “C-00000291*.sys”, and delete it. 
  • Detach the volume from the new virtual server
  • Reattach the fixed volume to the impacted virtual server

Friday, July 19, 2024 - 2:41 AM

WTS is currently investigating an issue with CrowdStrike that is affecting Windows hosts. A bug has been introduced in the latest update that is causing hosts to enter a blue-screen reboot loop. CrowdStrike have posted a workaround to recover affected systems.

Summary

CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor. 

Details

Symptoms include hosts experiencing a bugcheck\blue screen error related to the Falcon Sensor. 

Current Action

CrowdStrike Engineering has identified a content deployment related to this issue and reverted those changes.

If hosts are still crashing and unable to stay online to receive the Channel File Changes, the following steps can be used to workaround this issue: 

Workaround Steps:

  1. Boot Windows into Safe Mode or the Windows Recovery Environment
  2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
  3. Locate the file matching “C-00000291*.sys”, and delete it. 
  4. Boot the host normally. 

Published on  and maintained in Cascade.