{"id":23073,"date":"2024-07-19T18:56:08","date_gmt":"2024-07-19T14:56:08","guid":{"rendered":"https:\/\/me-en.kaspersky.com\/blog\/crowdstrike-global-cyber-outages\/23073\/"},"modified":"2024-07-19T18:56:11","modified_gmt":"2024-07-19T14:56:11","slug":"crowdstrike-global-cyber-outages","status":"publish","type":"post","link":"https:\/\/me-en.kaspersky.com\/blog\/crowdstrike-global-cyber-outages\/23073\/","title":{"rendered":"Global outage due to Friday&#8217;s release of CrowdStrike"},"content":{"rendered":"<p>Ever heard the unspoken rule: \u201cNever release on Friday\u201d? We have, but CrowdStrike hasn\u2019t. They released a tiny driver on an ordinary Friday morning, which became the cause of a huge outage all over the world.<\/p>\n<p>An incorrect update for CrowdStrike\u2019s EDR (Endpoint Detection and Response) solution has affected Windows devices around the world \u2014 <a href=\"https:\/\/x.com\/_johnhammond\/status\/1814178288220479565?s=46&amp;t=NOPDBSpOvyQ5d8ELQ2eNjw\" target=\"_blank\" rel=\"nofollow noopener\">giving<\/a> corporate users the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Blue_screen_of_death\" target=\"_blank\" rel=\"nofollow noopener\">Blue Screen of Death (BSOD)<\/a>. The failure has affected, for example, airport information systems in the US, Spain, Germany, the Netherlands and other countries.<\/p>\n<p>Who else was affected by CrowdStrike\u2019s Friday release and how to roll back bricked computers \u2014 all in this post\u2026\n<\/p>\n<h2>What happened<\/h2>\n<p>\nIt all started early Friday morning with corporate users around the world reporting problems with Windows. At first, a glitch in Microsoft Azure was blamed, but later CrowdStrike <a href=\"https:\/\/www.cnbc.com\/2024\/07\/19\/crowdstrike-suffers-major-outage-affecting-businesses-around-the-world.html\" target=\"_blank\" rel=\"nofollow noopener\">confirmed<\/a> that the root cause was in the <em>csagent.sys<\/em> or <em>C-00000291*.sys<\/em> driver for its CrowdStrike EDR. And it was this driver that caused an abundance of silly office photos showing off the (dreaded) blue screens.<\/p>\n<div id=\"attachment_51752\" style=\"width: 970px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/media.kasperskydaily.com\/wp-content\/uploads\/sites\/37\/2024\/07\/19185553\/crowdstrike-global-cyber-outages-01.jpeg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-51752\" class=\"wp-image-51752 size-full\" src=\"https:\/\/media.kasperskydaily.com\/wp-content\/uploads\/sites\/37\/2024\/07\/19185553\/crowdstrike-global-cyber-outages-01.jpeg\" alt=\"Blue screen of death on all computers = a day off for airport linemen\" width=\"960\" height=\"1280\"><\/a><p id=\"caption-attachment-51752\" class=\"wp-caption-text\">Blue screen of death on all computers = a day off for airport linemen<\/p><\/div>\n<p>If we wanted to list everyone affected by this outage, such a list sure wouldn\u2019t fit into this post \u2013 or dozens of them. So instead we\u2019ll briefly cover the main victims of CrowdStrike\u2019s negligence. Airline companies, airports, and people who want to either go home or go off on a long-awaited vacation were the most affected:\n<\/p>\n<ul>\n<li>London\u2019s Heathrow Airport, like many others, announced flight delays due to a technology glitch;<\/li>\n<li>Scandinavian Airlines posted a notice on its website saying, \u201cSome customers may experience difficulties with their bookings due to an IT issue affecting several countries. SAS is fully operational but delays are expected\u201d;<\/li>\n<li>In New Zealand, banking, communications and transportation systems are experiencing problems.<\/li>\n<\/ul>\n<p>\nVarious medical centers, chain stores, the New York subway, the largest bank in South Africa and many other organizations that make lives more comfortable and convenient on a daily basis were affected. The fullest list of those affected by the outage we can find is <a href=\"https:\/\/x.com\/spectatorindex\" target=\"_blank\" rel=\"nofollow noopener\">here<\/a> \u2014 and it\u2019s growing by the minute.\n<\/p>\n<h2>How to fix it<\/h2>\n<p>\nAt this stage, it\u2019s rather problematic estimating how long it\u2019ll take to fully restore the affected computers around the world. Things are complicated by the fact that users need to manually reboot their computers in Safe Mode. And in large corporations, this is usually impossible to do on your own without the help of a system administrator.<\/p>\n<p>Nevertheless, <strong>here are the instructions for how to get rid of the blue screen of death caused by the CrowdStrike driver update:<\/strong>\n<\/p>\n<ol>\n<li>Boot your computer in Safe Mode;<\/li>\n<li>Go to <em>C:WindowsSystem32driversCrowdStrike<\/em>;<\/li>\n<li>Locate and delete the <em>csagent.sys<\/em> or <em>C-00000291*.sys<\/em> file;<\/li>\n<li>Restart your computer in normal mode.<\/li>\n<\/ol>\n<p>\nAnd while your sysadmins are doing this, you could use a hack that\u2019s come out of India today: employees of one of the country\u2019s airports have started filling out boarding passes\u2026 manually.<\/p>\n<div id=\"attachment_51753\" style=\"width: 684px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/media.kasperskydaily.com\/wp-content\/uploads\/sites\/37\/2024\/07\/19185602\/crowdstrike-global-cyber-outages-02.jpeg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-51753\" class=\"size-full wp-image-51753\" src=\"https:\/\/media.kasperskydaily.com\/wp-content\/uploads\/sites\/37\/2024\/07\/19185602\/crowdstrike-global-cyber-outages-02.jpeg\" alt=\"India isn't too worried about the global disruption\" width=\"674\" height=\"588\"><\/a><p id=\"caption-attachment-51753\" class=\"wp-caption-text\">India isn\u2019t too worried about the global disruption. <a href=\"https:\/\/x.com\/akothari\/status\/1814202068531552666?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Etweet\" target=\"_blank\" rel=\"nofollow noopener\">Source<\/a><\/p><\/div>\n<h2>How the failure could have been avoided<\/h2>\n<p>\nAvoiding this situation should have been straightforward. First, the update shouldn\u2019t have been released on a Friday. This is as per a rule that\u2019s been known to all in the industry since the year dot: if an error occurs, there\u2019s too little time to fix it before the weekend, so the system administrators at all companies affected need to work over the weekend to fix things.<\/p>\n<p>It\u2019s important to be as responsible as possible about the quality of updates released. We at Kaspersky launched a program back in 2009 to prevent mass failures such as this one at our customers, and <a href=\"https:\/\/www.kaspersky.com\/blog\/soc2-audit\/28001\/\" target=\"_blank\" rel=\"nofollow noopener\">passed an SOC 2 audit<\/a>, which confirms the security of our internal processes. For 15 years now, every update has been subjected to multi-level performance testing on various configurations and operating system versions. This allows us to identify potential problems in advance and resolve them on the spot.<\/p>\n<p>The principle of granular releases should be followed. Updates should be distributed gradually, not all at once to all customers. This approach allows us to react instantly and stop an update if necessary. If our users have a problem, we register it, and its solution becomes a priority at all levels of the company.<\/p>\n<p>As with cybersecurity incidents, in addition to fixing the visible damage, you need to find the root cause to prevent these types of problems repeating in the future. It\u2019s necessary to <a href=\"https:\/\/support.kaspersky.com\/help\/KSC\/14\/en-US\/181095.htm\" target=\"_blank\" rel=\"noopener\">check software updates on test infrastructure<\/a> for operability and errors before rolling them out to the company\u2019s \u201ccombat\u201d infrastructure, and to implement changes gradually \u2014 continually monitoring for possible failures.<\/p>\n<p>Incident handling should be based on an integrated approach to building protection from a trusted supplier with the strictest internal requirements for the security, quality and availability of its services. The basis for this work can be the <a href=\"https:\/\/go.kaspersky.com\/next?icid=gl_kdailyplacehold_acq_ona_smm__onl_b2b_kdaily_lnk_sm-team_______8bef525ada73b6ec\" target=\"_blank\" rel=\"noopener nofollow\">Kaspersky Next<\/a> line of solutions. This will help your company not only stay afloat \u2014 but also increase the efficiency of your information security system. This can be done either gradually \u2014 increasing protection step by step \u2014 or all in one go. Protect your infrastructure today with us so that the next <a href=\"https:\/\/en.wikipedia.org\/wiki\/2024_CrowdStrike_incident\" target=\"_blank\" rel=\"nofollow noopener\">global outage<\/a> doesn\u2019t affect your customers.<\/p>\n<p>And we, for our part, can help you make this decision: <a href=\"https:\/\/go.kaspersky.com\/Kaspersky_Next_EDRO_promo?icid=gl_kdailyplacehold_acq_ona_smm__onl_b2b_kdaily_lnk_sm-team_______5f972f9b8894e0fe\" target=\"_blank\" rel=\"noopener nofollow\">switch to Kaspersky and unlock two years of Kaspersky Next EDR Optimum for the price of one<\/a>. Experience the pinnacle of robust, reliable cybersecurity protection!<\/p>\n<input type=\"hidden\" class=\"category_for_banner\" value=\"kaspersky-next\">\n","protected":false},"excerpt":{"rendered":"<p>The story of how CrowdStrike released an update on a Friday and brought down thousands, tens of thousands, or maybe even hundreds of thousands of computers around the world.<\/p>\n","protected":false},"author":2706,"featured_media":23075,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1318,1916,1917,1486,9],"tags":[2493,2581,2775,38,521,121,113],"class_list":{"0":"post-23073","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-business","8":"category-enterprise","9":"category-smb","10":"category-threats","11":"category-tips","12":"tag-azure","13":"tag-driver","14":"tag-failure","15":"tag-microsoft","16":"tag-threats","17":"tag-updates","18":"tag-windows"},"hreflang":[{"hreflang":"en-ae","url":"https:\/\/me-en.kaspersky.com\/blog\/crowdstrike-global-cyber-outages\/23073\/"},{"hreflang":"en-in","url":"https:\/\/www.kaspersky.co.in\/blog\/crowdstrike-global-cyber-outages\/27743\/"},{"hreflang":"en-us","url":"https:\/\/usa.kaspersky.com\/blog\/crowdstrike-global-cyber-outages\/30423\/"},{"hreflang":"en-gb","url":"https:\/\/www.kaspersky.co.uk\/blog\/crowdstrike-global-cyber-outages\/27954\/"},{"hreflang":"ru","url":"https:\/\/www.kaspersky.ru\/blog\/crowdstrike-global-cyber-outages\/37918\/"},{"hreflang":"x-default","url":"https:\/\/www.kaspersky.com\/blog\/crowdstrike-global-cyber-outages\/51748\/"},{"hreflang":"ru-kz","url":"https:\/\/blog.kaspersky.kz\/crowdstrike-global-cyber-outages\/28089\/"},{"hreflang":"en-au","url":"https:\/\/www.kaspersky.com.au\/blog\/crowdstrike-global-cyber-outages\/33886\/"},{"hreflang":"en-za","url":"https:\/\/www.kaspersky.co.za\/blog\/crowdstrike-global-cyber-outages\/33551\/"}],"acf":[],"banners":"","maintag":{"url":"https:\/\/me-en.kaspersky.com\/blog\/tag\/failure\/","name":"failure"},"_links":{"self":[{"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/posts\/23073","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/users\/2706"}],"replies":[{"embeddable":true,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/comments?post=23073"}],"version-history":[{"count":1,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/posts\/23073\/revisions"}],"predecessor-version":[{"id":23074,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/posts\/23073\/revisions\/23074"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/media\/23075"}],"wp:attachment":[{"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/media?parent=23073"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/categories?post=23073"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/me-en.kaspersky.com\/blog\/wp-json\/wp\/v2\/tags?post=23073"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}