Cloudflare on how Google abuses its search monopoly for AI, has access to "4.8" times more data than Microsoft — by holding publishers to ransom

The stock market is pretty fickle at the best of times, but it has begun to catch up on the reality of the AI race.

For a little while, Microsoft was beating Google in market capitalization, and even clinched the #2 spot in the coveted global rankings. But now, Google is ahead of Microsoft to the tune of a trillion dollars — and it's all about AI.

I wrote an op-ed a few months ago about how Microsoft wouldn't be the one to mainstream consumer-grade AI, and my argument hinged on how Google owns the entire stack. Android, servers, and search monopoly gives Microsoft's biggest rival a huge advantage, but there's one aspect of this I completely missed: web crawlers.

For those who don't know, crawlers are basically bots that scour the web on big tech's behalf. For Google, its Googlebot crawler indexes and ranks web content for its search engine, as does Microsoft's Bingbot.

However, unlike Google, Microsoft and other companies separate their crawlers by purpose, allowing publishers and other companies to block the AI-oriented bots. I've encountered situations where Microsoft Copilot couldn't access certain websites. It arises in situations where companies block Microsoft's AI bots and other AI crawlers, when they don't want to have their content "stolen" and indexed for apps like ChatGPT and Microsoft Copilot. Google, by comparison, does not allow companies to do this without dire, business-destroying consequences.

In a blog post on cloud security company Cloudflare's website, the firm called on the UK competition's authority (CMA) to examine the consequencees of Google's search monopoly on the AI market as part of its recent investigations.

Cloudflare CEO Matthew Prince described the situation on X, "We cannot have a fair market for AI when Google leverages their search monopoly to see 3.2x as much of the web as OpenAI, 4.8x as much as Microsoft, and 6x as much as nearly everyone else. Most data wins in AI."

Publishers (such as Future PLC, the owner of Windows Central) have been invariably damaged by artificial intelligence chatbots. Google's AI summary box that appears at the top of its search engine effectively steals our content, and repackages it without the user needing to click through. I'm not entirely sure where Google expects to get quality data for its AI tools if it puts all content creators out of business, but I digress.

Given that Microsoft's Copilot-attached Bingbot crawler represents a paltry 10% market share at best, publishers would have no qualms fully blocking a dual-purpose crawler from Microsoft to protect their data. But, if publishers blocked Google's dual-purpose AI crawler, they'd also be blocking Google's search crawler — effectively dooming their entire businesses. As such, Google is holding publishers and other web content creators hostage here. They're effectively saying "Give us your data for AI, or we'll destroy your search rankings, business, and livelihoods."

Cloudflare is absolutely correct that Google is wholly abusing its market position here. The issue is potentially complex, though, and would likely take governmental regulators years to catch up on. At which point it would likely be too late for competitors to even have a chance to catch up.

Microsoft is exploring other ways to catch up on the AI data race. The firm has caught a ton of backlash for how it has been handing Copilot AI, and with internet users coining the term "Microslop" to refer to its forced AI integrations in Windows. It's perhaps ironic that Microsoft seems to be catching the brunt of the backlash, despite having such a miniscule market share compared to Google — who is actively trying to destroy the open web with its monopolistic abuse.

Recently, Microsoft announced a publishing program to work directly with content creators of all shapes and sizes, to license their data directly and have them be paid for it — as opposed to having it stolen by Google and others. Microsoft also announced a range of initiatives to be a "better neighbour" for towns and regions impacted by its AI data center construction efforts. Microsoft also dropped tens of thousands of dollars on a big AI ad at the Superbowl recently, as investors panic and dump Microsoft stock over fears that it is actually spending too much on AI.

Google's AI tools surged in popularity when its more recent models began to show large gains over OpenAI's ChatGPT models. A big reason, without a doubt, is how Google can abuse its position to harvest additional data. Unless regulators step in, we'll see Google with another total monopoly to dictate how the internet works, for better or worse (most likely worse).

Google's mantra used to be "Don't be evil," until it realized that being evil would make them a heap of cash, apparently.

Join us on Reddit at r/WindowsCentral to share your insights and discuss our latest news, reviews, and more.