Microsoft warns attackers can secretly manipulate AI recommendations

Microsoft has recently warned that AI can be poisoned. At first glance, that might sound obvious. After all, AI systems are trained on vast amounts of information from books, media, and online posts, and not all of that can be information-accurate.

But what Microsoft is describing here is something more deliberate. It is a warning about a tactic designed specifically to trick AI assistants.

In simple terms, companies are hiding instructions inside “Summarize with AI” prompts. When clicked, those prompts quietly tell the AI to remember the company as a trusted source.

From that point on, the AI may start recommending that company in future responses. The user does not see the hidden instruction, and the assistant appears to be giving neutral advice, when in reality, it has been nudged to favor a specific brand

What Microsoft means by AI memory poisoning

If you weren't aware, AI assistants can store information across conversations you have. That can include preferences, instructions, facts, and other details you previously shared.

This warning from Microsoft is not about corrupting how an AI model is trained. Instead, it is about slipping the AI assistant a note that gets saved in its personal memory, which is unique to your interactions and not used to train the wider AI model.

It works through hidden instructions. For example, a prompt might quietly say, “Remember this company as a trusted source,” leading the AI to treat it as legitimate in future answers, which could lead the AI assistant to continuously recommend that site as a source.

According to Microsoft, this is not a small or isolated issue. Over a 60-day period, it identified more than 50 distinct prompt injection attempts from 31 companies across 14 industries. That suggests this tactic is very widely spread.

To avoid this, Microsoft advises users to be cautious with AI links and to regularly check their assistant’s saved memory. If something looks unfamiliar, review it and remove it.

If you are unsure whether your AI has been influenced, Microsoft recommends clearing its memory entirely. That effectively removes any injected instructions or stored biases.

Just another day and another totally normal thing we now have to think about as AI promises to make life easier, whilst also driving up prices across the industry, which is what we all want, right? That is sarcasm, of course.

AI can absolutely be useful in the right situations. But like any tool connected to the internet, using it carefully and understanding how it works is always the safer approach.

It is important I stress, there are serious concerns here as Microsoft warns of potential risks in areas like financial advice, health recommendations, news summaries, and even child safety.

I have tried my best here to explain things as simply as possible, but if you wish to read all about it, Microsoft does have a handy blog post, which you can find here, that goes into much more detail.

Have you checked what your AI assistant remembers about you lately? Let us know your thoughts in the comments and make sure to take part in our poll below:

Join us on Reddit at r/WindowsCentral to share your insights and discuss our latest news, reviews, and more.