By now, you all know the story of the CrowdStrike incident. On July 19, millions of computers around the world entered a Blue Screen of Death (BSOD) loop after CrowdStrike pushed a faulty update to their flagship product, CrowdStrike Falcon. Airlines, hospitals, and all manners of institutions suffered outages ranging from minor to crippling. Thousands of travelers were stuck at Atlanta International Airport alone. The insurance industry estimates that total damages from the incident amount to billions of dollars. 

CrowdStrike carries a huge portion of the responsibility for failing to test their update adequately and not rolling it out in a safer, more incremental manner following industry best practices. But Microsoft has also come under scrutiny for allowing 3rd party software to bring down the entire operating system in the first place. Microsoft is no stranger to criticism, but I’ve got to say that it had an interesting response this time around: The EU made me do it.  

A Microsoft spokesperson told the Wall Street Journal that it “cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint.” The understanding is a 2009 antitrust settlement agreement between Microsoft and the EU, best known for introducing the Browser Choice Screen.

The clear implication of Microsoft’s statement is that it could technically have prevented the CrowdStrike incident, but its hands were tied by EU red tape. What’s more, it follows that this misguided regulation prevents them from plugging this gaping security hole to stop future incidents of equal or even greater magnitude from happening. Is EU bureaucracy leaving the world vulnerable to crippling IT outages?

If true, I would expect world governments, including the many EU member states that rely on Microsoft products, to pressure the EU to release Microsoft from such a backward regulation. And why isn’t Microsoft shouting this from the rooftop, instead of quietly casting aspersions through an anonymous spokesperson? I disapprove of government overreach as much as the next person, but something doesn’t smell right here. What role did the EU really play in the CrowdStrike incident? 

To better understand the incident, we first need to understand what CrowdStrike Falcon is. CrowdStrike Falcon is an Endpoint Detection and Response (EDR) solution, the modern evolution of antivirus products. While Antivirus software generally compares any executed binary to a list of known malicious ones, EDR takes the slightly smarter approach of looking for known malicious behaviors. For example, EDR could look for binaries trying to execute scripts from temporary directories, something legitimate programs rarely do but is a known malware pattern.

EDR software needs special privileges to monitor other processes for malicious behavior. However, a normal process cannot monitor other processes’ behaviors. The operating system provides processes with the illusion that they are the only thing running on the machine. This is a vital security mechanism in modern operating systems. You don’t want Candy Crush to be able to read what you’re writing in your email client. The one process that can actually monitor others is the operating system kernel itself.

The kernel runs with full privileges, able to do just about anything it wants, to provide basic operating system services: process isolation, multitask scheduling, hardware abstraction, and so on. To borrow some of that privilege, software vendors can write kernel modules, which are essentially small extensions to the operating system kernel. The typical use case is to write device drivers for hardware, but it can also be useful for specialized types of software like EDRs and VPNs. Kernel modules run as part of the kernel and inherit its awesome privileges and responsibilities. This enables EDR software to be notified when new processes are started or engage in certain types of behavior.

One big problem with this approach is that it makes the whole system less resilient. When a process running in user space (as opposed to kernel space) encounters a critical error, the process crashes but the kernel and other processes keep running just fine. But when a kernel extension encounters a critical error, it crashes the entire kernel, which in turn crashes the entire system. On Windows machines, this leads to the infamous Blue Screen of Death. Making matters worse, kernel modules tend to be loaded automatically on startup, so a kernel module bug could lead to the computer booting straight into BSOD, rendering the machine entirely useless. 

This is exactly what happened on July 19. CrowdStrike issued an update to their EDR agent which had a nasty bug in it known as a page fault. For some reason, CrowdStrike did not subject this class of updates to automated testing or incremental rollout, leading to the undiscovered bug being rolled out to millions of machines at once and promptly sending those machines into a BSOD loop.

Incidents like this are not only possible but probable because of EDR’s need to run with kernel privileges. It’s an inherently dangerous thing to give full kernel privileges to non-kernel software. The people behind MacOS and Linux realized long ago that requiring full kernel privileges to implement security software is untenable. Apple has deprecated its kernel modules (KEXT) in favor of System Extensions. MacOS System Extensions enable you to access some of the kernel’s privileges without running as part of the kernel, a much more secure and resilient solution. System Extensions allow you to more closely tailor the privilege afforded to EDR to just the ones it needs to do its job. And if a System Extension crashes, it crashes just itself instead of the entire system. Linux has a similar solution called eBPF, which runs in a sandboxed environment inside the kernel.

This is what Microsoft is referring to when they say they can’t “wall off its operating system in the same way Apple does”. Supposedly, the EU agreement is preventing them from doing so. But this is what the agreement has to say about security:

Microsoft shall ensure on an ongoing basis and in a Timely Manner that the APIs in the Windows Client PC Operating System and the Windows Server Operating System that are called on by Microsoft Security Software Products are documented and available for use by third-party security software products that run on the Windows Client PC Operating System and/or the Windows Server Operating System. 

Once you get past the legalese, this paragraph says that Microsoft cannot give its own security products preferential access to operating system features and deny its competitors the same. It needs to give equal access to its own security products and 3rd party ones. 

This makes perfect sense for an antitrust agreement: Don’t abuse your monopoly position in one market (operating systems) to give yourself an unfair leg up in another market (security solutions). Doing so is known as monopoly leveraging and antitrust legislation around the world is designed to prevent that type of anti-competitive behavior. The US government sued Microsoft over this very same practice in the browser market in 1998. Australia fined VISA for leveraging its dominant position in the credit card market to stifle competition in the dynamic currency conversion market. People don’t want companies to engage in leveraging. They want products to win based on their merits and not because they could force their competitors to compete with one hand tied behind their backs. 

What’s important to note here is that the agreement says nothing about kernel access. It’s just about equal access to the APIs that Microsoft’s own security products use. There is nothing to prevent Microsoft from implementing a solution like MacOS’ System Extensions or Linux’s eBPF and having its security products use those instead. Microsoft chose the easy way out by instead opening its kernel to all security products and the world’s IT infrastructure is less safe for it.

Seen in this light, it’s clear that the WSJ statement is not just misleading. It’s complete BS. Microsoft can, and should, wall of access to its operating system in the same way Apple does. The fact that they haven’t gotten around to it at any point in the past 15 years while spending more than $180 billion in stock buybacks is kind of shocking, even though it fits the company’s long-standing pattern of neglecting security, which it has repeatedly been called out for.

Yes, CrowdStrike should have followed industry best practices for safe and responsible rollout, especially when it comes to kernel extension code. But Microsoft gave CrowdStrike and countless other vendors no option but to write kernel extensions to implement their products. The EU did not make Microsoft do that. It was just a matter of time before something like the CrowdStrike incident happened. If Microsoft doesn’t provide a safer path for security vendors, it’s also just a matter of time before it happens again.

Let’s put modern computer vision to the slightest of tests. Having dabbled in computer vision research, I figured I could fool the Google Cloud Vision API in about five attempts. Google has been a leader in AI for decades, so it should be a fairly good representation of the current state of the art. I stopped trying after three attempts where Google misclassified every single one of the images, which, by the way, my three-year-old had no trouble with. Not an impressive performance so far. So why is Nature, the most prestigious scientific publication in the world, saying that AI beats humans in image classification?

The Experiment That Started it All

The Nature News article is actually referencing the AI Index Report 2024 by the Stanford HAI Institute, a report whose ostensible aim is to “provide unbiased, rigorously vetted, broadly sourced data in order for policymakers, researchers, executives, journalists, and the general public to develop a more thorough and nuanced understanding of the complex field of AI”. It says that “AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding” and that “AI has surpassed human baselines on a handful of benchmarks, such as image classification in 2015”. So what was this computer vision singularity event back in 2015 where AI supposedly outpaced humans at image classification?

To answer that, I first have to tell you about Imagenet. In 2012, the first annual ImageNet Large Scale Visual Recognition Challenge was held, a competition where researchers submitted their best models to compete for the highest accuracy score on the Imagenet dataset. The dataset has 1.2 million images scraped from the photography-focused social network Flickr. Researchers collected images by searching Flickr images using search words like “can opener” or “swing” and then validating that the image contained said search terms using workers hired through Amazon Mechanical Turk. Each image has exactly one out of 1000 possible labels. Teams train their algorithm on all 1.2 million images and test it on a separate set of 100 thousand images. You can submit 5 guesses for each image and if one of them is true, you get full points. In essence, each image in the test set is a multiple-choice question with 1000 possible answers but you get to check five boxes.

In 2014, Andrej Karpathy, then a Stanford graduate student, decided to compare himself against the latest winning algorithm. He painstakingly labeled 1500 images according to the rules of the competition and achieved an error rate of 5.1%, beating that year’s winner, GoogLeNet, by 1.7 percentage points. This brings us to the “singularity event”: the following year, Microsoft developed a model that narrowly beat Karpathy’s result with a 4.94% error rate. This result was widely reported in both the mainstream and tech media as the advent of computers surpassing humans in image recognition. And so it would seem. A human was pitted against an AI algorithm and the algorithm won. Right?

Human Errors Are Not Really Errors At All

When you take a closer look at the results, it turns out that humans and computers make different kinds of mistakes on the challenge. When the computer fails, it fails miserably. It thinks that a figurine is a coffee pot, that a king crab is a strawberry, and that a salt shaker is hairspray. Even seven years later, Google researchers noted that 40% of mistakes made by state-of-the-art algorithms were “major mistakes” that “most humans would likely not make”.

Meanwhile, humans make different types of mistakes. They can’t keep all 1000 classes in their head, so they forget or don’t know that a given class is a possible answer. Some images contain multiple objects, none of which is more salient than the others, and only one of which is the correct answer (to be fair, the computer makes those mistakes as well). And the most common mistakes humans make are caused by what are called fine-grained classes. 

Fine-grained classes are classes like “Indigo Bunting” or “Bedlington Terrier” as opposed to more coarse-grained classes like “bird” or “dog”. The label set contains a mix of coarse-grained and fine-grained classes, including over 120 distinct breeds of dogs. I’ll admit that AI might be better than the typical human at these fine-grained labeling tasks simply because the typical human can’t name 120 different dog breeds. Humans can still easily recognize images of dogs and even describe them in great detail, whether or not they are familiar with the breed name. Meanwhile, the computer will get the breed right some of the time, and some of the time think it’s an image of a flatweave rug. A subsequent study found that once you add multiple labels and divide the test set into “organics” and “objects”, effectively separating the fine-grained classes from the coarse-grained classes, the median human accuracy is 99% while the best AI model achieved 95%. 

All of this is to say, when you take a closer look at the errors, it’s not clear that AI beat humans in image recognition based on the Imagenet data. Humans can easily recognize what the images depict, even though sometimes they struggle with fine-grained ontological classification. Meanwhile, the AI archives does alright for itself but is way, way off the mark when it’s wrong.

The Results Don’t Replicate

Even if AI had truly outperformed humans on the Imagenet challenge, the more egregious problem is that it doesn’t replicate. In 2019, a group of Berkeley researchers decided to replicate the AI portion of the experiment. They collected a new dataset of the same size using the same methodology as Imagenet: scraping Flickr using keywords, using Amazon Turk workers to verify, using the Imagenet classes, and even limiting their collection to the same time period as when the original Imagenet data was collected. Then they took previously published AI models, trained and tested them on the new data, and compared the results to how the models performed on the original data. Every one of the models had at least an 11 percentage point drop in accuracy. So even when you recreate the experiment using data that is as close to the original as possible, the AI results don’t hold any longer. This means that these models are not so much good at recognizing images as they are good at recognizing the specific images in the Imagenet dataset alone.

AI needs a higher scientific standard

To be fair, the field of computer vision has continued to improve on Microsoft’s 2015 result and has continued to refine the experiment by adding multiple labels to images, using expert labelers to comb out wrong labels, and testing more humans (Shankar et al. had a whopping 5 human participants). But, not only do trained humans continue to outperform AI models on the Imagenet challenge, but the field has also failed to address the biggest shortcoming of using the Imagenet challenge to evaluate human vs AI performance: you can’t draw a sweeping conclusion about the relative abilities of AI and humans based on a single experimental setup. 

This limited experiment contains images that are nothing like the broader spectrum of images humans encounter in their daily lives. We don’t experience the world in a series of well-lit and well-composed photographs like the ones you find on Flickr. Real-world images show objects from lots of weird angles, often poorly lit, and with all kinds of occlusions. And we certainly don’t divide the world into 1000 seemingly arbitrary classes. This experimental setup is so limited that the only conclusion you could draw from its results is “Huh, interesting - we should investigate further”. 

Other fields of study, such as the medical sciences, have developed rigorous testing methodologies such as randomized double-blind placebo-controlled trials, meta-analyses combining multiple studies to achieve greater statistical significance, and systematic reviews to assess the totality of evidence to answer hard questions.“We tried this drug on this one guy and it seems to work” would not exactly fly in pharmacology. 

Meanwhile, the field of AI is still using Karpathy’s Imagenet results as a human baseline a decade later. Not that there was anything wrong with Karpathy’s experiment. It’s an interesting result. The problem is that the field of AI is still using it as a baseline. Which is a symptom of the systemic lack of empirical rigor in the field of AI. It focuses too often on producing a higher accuracy on a benchmark than actually improving our collective understanding of computer vision algorithms or specific contours of AI image recognition capability.

The Bottom Line

So no, AI does not beat humans at image recognition. The scientific evidence crumbles under scrutiny. There’s no new tech that leverages superhuman image recognition. Google doesn’t recognize an upside-down bus, self-driving cars are terrible, and your radiologist remains very much human. And still, this claim continues to be repeated, uncritically, to this day in the media.

To be fair, the claim shows up in the media because of reports like the Stanford AI Index. And you could say the report just says AI beats “a human baseline” on a “benchmark” and that is technically true. But without providing any additional context, the entirely predictable result was that even the scientific media just saw “AI beats humans”. 

Researchers in this incredibly well-funded field could easily devise and carry out a series of rigorous experiments to shed clear light on the question of AI vs human image recognition. The reason they haven’t is probably because they know the answer. AI has made a ton of progress but is far, far from human performance. So why can’t they just say that in their reports?

Subscribe now