Microsoft and Cloudflare want to reinvent the web — and leave Google behind

Search engines might be entering a new era. Instead of keyword-based results, the focus is shifting toward AI-powered direct answers, something anyone who uses Google will already recognize.

Reported on by TechRadar, Microsoft and Cloudflare are teaming up to push this shift further with NLWeb, Microsoft’s new standard, and AutoRAG, Cloudflare’s infrastructure built for AI.

This move is less about traditional SEO and more about rethinking how websites interact with people and AI agents. It’s a step that could challenge Google’s dominance and reshape what it means to visit a website.

From search engines to answer engines

Traditionally, searching the web has been built on keywords. You type in a query, and the search engine returns a ranked list of links to choose from.

That model has worked for decades, but the web keeps growing, and AI is now at the forefront of how people discover information.

Tools like ChatGPT, Claude, and Copilot are changing expectations. Many users now prefer direct answers over digging through links.

This is where answer engines come in, aiming to provide immediate responses and cut out the extra steps in the middle.

Microsoft and Cloudflare are positioning NLWeb + AutoRAG as infrastructure to support this shift, making sites behave more efficiently with AI. This is with the bet that future web traffic will come from AI agents just as much as people browsing pages.

How NLWeb and AutoRAG work together

NLWeb is a new web standard created at Microsoft. It’s designed so websites can open up structured endpoints for natural language queries.

If that sounds confusing, here’s the simple version. NLWeb lets you visit a website and use a prompt like /ask to type a question. The site then gives you a direct answer, instead of forcing you to dig through pages to find it.

The idea is to make websites feel more like conversational apps and less like static pages.

AutoRAG is a Cloudflare service that handles Retrieval-Augmented Generation (RAG) automatically. It crawls a website, indexes the content, and converts it into embeddings stored in a managed database.

This setup keeps content fresh with continuous updates and removes the technical burden of building your own vector database and pipelines.

The two systems work together by NLWeb defining the standard for how queries and responses should be structured, and AutoRAG provides the infrastructure to implement the standard more easily.

Combined, NLWeb and AutoRAG allow websites to answer natural language queries from both humans and AI agents without relying on traditional scraping.

Within Cloudflare’s dashboard, you can block AI training bots if you want more control. But to enable the new feature, go to Compute & AI → AutoRAG, select Create in AutoRAG, and choose the NLWeb Website Quick Deploy option.

The risks — and the challenge to Google

Google dominates web search, built on crawling, indexing, and ranking pages. Its Gemini AI already gives direct answers, and Google has launched an AI mode for search.

NLWeb and AutoRAG take a different approach by bypassing the crawl-and-rank model entirely. If this gains traction, it could pull traffic away from Google and toward Microsoft’s ecosystem, strengthening Bing and Copilot in the process.

There are risks for consumers, too. Answer engines cut out the middleman, which some may prefer, but these systems are not perfect. They can be prone to bias and issues with accuracy.

Relying on AI-generated answers also reduces visibility for smaller sites, pulling money away from the people whose content helped train these tools in the first place. Personally, I’m not sure I’m comfortable with that, and I’d be curious to hear what readers think.