Over the past few days, I've watched my productivity drop (don't tell my boss) a fairly huge amount owing to a sudden explosion of consumer-friendly and highly memeable artificial intelligence (AI) machine learning (ML) tools you might have seen on social media.

Services like Craiyon that automatically generate rough AI images based on virtually any input prompt have produced a range of wild and wacky collages across the net, which you can peruse yourself over on subreddits like /r/weirddalle or Twitter accounts like @Dalle2Pics. 

Chatbots and weird AI-generated art are widely known, but what's perhaps more shocking is this more recent wave of sophistication, and how it is already making its way into the general public's consciousness. 

And hey, I admit I have a very surface-level understanding of some of this stuff, but my goal here is to try to outline and extrapolate potential applications we could see appear over the course of the Xbox Series X|S console generation. 

After spending hours generating ridiculous AI stories and pictures to bug my friends and co-workers, I began to dig a little more into how it all works, and what the implications could be for practical applications that move beyond simple memery. 

Microsoft has been making massive investments in the space. We've all heard about the Nuance purchase, which gave Microsoft control over the conversational tools that powers Apple's Siri, but Redmond has been quietly building up its own internal tools and features as it looks to what will most likely end up becoming the next major shift in technology, both for businesses and consumers. 

How deep-learning AI will change gaming forever

Obviously, machine learning techniques have broad implications for almost every sector of life, but how they will intersect across gaming has potentially some of the broadest implications for Microsoft as a business. 

One problem the video game industry generally faces right now pertains to the gap between expectations and investment. Video games are becoming increasingly complex to make, fund, and manage, as they explode in exponential complexity and graphical fidelity. We've seen absolutely insane Unreal Engine demos that showcase near-photorealistic scenes and graphics, but the manual labor involved to produce a full game based on some of these principles is truly palpable both in terms of time, and expense. 

What is typically thought of as "AI" in a gaming context generally hasn't been AI in the true sense of the word. Video game non-player characters (NPCs) and enemies generally operate on a rules-based model that often has to be manually crafted by a programmer. Machine learning models are importantly far more fluid, able to produce their own rules within parameters, and respond dynamically to new information on the fly. This applies across all things, whether it's NPC behavior or content generation. 

I asked a GPT-3 ML text model to generate a quest for World of Warcraft, including objectives, and even NPC dialogue. This is a system that isn't even designed for it, leveraging random and unfiltered information from the internet to recontextualize and repackage the information based on the request. I also asked an AI to generate an evil sentient banana, with similarly fun results. 

Taking context from the internet, the AI model was able to produce a practically infinite quest dialogue, drawing down from information about World of Warcraft on websites and wiki entries to produce scarily realistic objectives that included real place names and enemy types. With direct access to more refined data and tailor-made context by a game developer, it's not hard to extrapolate how these models could be adapted to produce vast arrays of dynamic content. DALL-E and similar models are increasingly capable of producing an array of variants from any prompt, refining them with extraordinary levels of detail. It can also produce remixes of existing images. It's easy to imagine how that could extend to variations of monsters in games, or even variations of in-game behavior and animations. 

I suspect that for the foreseeable future, handmade content will remain the core of what makes a great video game truly great, but ML systems will become a primary source of supplementary content for a variety of game genres.

Imagine the next generation of Minecraft's world seed generation tools, using machine learning to include game rules, generate unique monsters, and new terrain. They could extend and improve tile-based procedurally generated levels in games like Diablo or Hades. They could generate supplementary quests for endgame play in MMO-type games like World of Warcraft or Fallout 76. 

Machine learning algorithms will supplement the construction and animation of creatures, level assets, and characters, at a speed and scale that simply isn't possible when you factor in labor costs. These systems won't (and shouldn't) replace real-life developers and artists, but should instead supplement alacrity in time-intensive work that can increase costs, lead to delays, and create pressure within a collaborative workflow. If you can have an AI handle the animation and behavior model of a simplistic background entity like a rat or a snake that is there purely for ambiance, that could free up dev time for more nuanced or complex tasks that have a greater impact on a player's experience. 

These are just a few very basic examples of how machine learning will enhance and elevate games across the Xbox Game Studios portfolio, but Microsoft will be keen to extend these services to the broader game development community as well under its Azure PlayFab brand. 

Microsoft is already poised to be a frontrunner 

Microsoft has been quietly (and not so quietly) investing quite heavily in machine learning for some time. I'd argue that the impact hasn't been fully felt yet in its gaming, but that could be about to change. 

Microsoft announced that it had acquired exclusive rights to some OpenAI models such as GPT-3 back in 2019 as part of a $1 billion investment. Since then, Microsoft has begun integrating aspects of machine learning it into some of its developer products. Copilot on GitHub and PowerApps Ideas are two services that leverage machine learning based on OpenAI models to supplement and speed up app development and programming. 

OpenAI is developing broad and extensive research into machine learning techniques loosely based on neural pathways in the human brain. You can browse some of its work on its website. Dalle 2 is producing truly insane lifelike artistic renders from simple text prompts with staggering results

Indeed, there are obvious concerns that are worth addressing when it comes to these machine learning models. And no, we're not talking about computers becoming sentient and Xbox Live becoming Skynet. 

Machine learning tools have already quite notoriously been abused to create all sorts of content that could be considered brand unsafe at best, to harmful at worst. Machine learning could lead to more sophisticated player-hostile bot behavior, for example, which could automate and trivialize gameplay. Imagine an aimbot that not only aimed for you, but also moved and dodged bullets for you as well. 

Indeed, I was able to bypass a lot of the simple filters attached to many of the current free tools to produce quite surprisingly dark content I'm sure publishers wouldn't want to appear within a game like Minecraft, for example, which is enjoyed by millions of kids across the world. Researchers are actively exploring ways to mitigate these kinds of issues, of course. Smarter, and faster context-sensitive content filters are obviously a heavy area of investment on the part of everyone involved. 

I distinctly remember a conversation I had with someone at Microsoft the year Minecraft Earth was announced. The canceled Minecraft spin-off was supposed to let people build augmented reality creations and place them in the real world, and I recall asking how Microsoft would be able to prevent naughty creations from unscrupulous 'crafters from appearing in other players' worlds. Machine learning was discussed as an option — training some kind of AI to recognize and moderate offensive content would obviously be paramount in a game like Minecraft Earth, which was all about sharing content. Microsoft also famously used early versions of ML models to moderate its chat channels on Mixer, its similarly canceled streaming service. 

Of course, human intervention will still be needed for the foreseeable in many cases, but these examples just further represent how pervasive machine learning will be across the entire gaming horizon. Content creation, content moderation, and even community management could be supplemented by AI.

Microsoft has already begun exploring how to implement machine learning into its games. Project Paidia has already demonstrated how AI can be trained to perform collaboratively, and dynamically with human players. The page there even has a HTML5 mini-game you can test out to demonstrate the basics of how it works, and mentions how they were exploring a potential implementation in the team-based brawler Bleeding Edge at one point. Researchers at MineDojo are also building similar ML models into Minecraft, building bots that can behave more intelligently and perform player-like actions. Microsoft also baked DirectML into the Xbox Series X and Xbox Series S to provide compute to generic machine learning models. Microsoft also leverages the tech to improve the delivery of streamed games on Xbox Game Pass' cloud gaming platform. 

Microsoft is hiring engineers for its gaming AI efforts, led by MSFT Research alumna and computer scientist Haiyan Zhang, with the goal of implementing machine learning into game development and gameplay. Microsoft is also one of the few companies with cloud computers pervasive enough to support the processing overhead that could emerge as a result of implementing some of these models directly into service games — where they're naturally going to see their full potential realized. 

Indeed, I suspect it won't be long before we start seeing more tangible player-driven interactions using machine learning models at a gameplay level in the coming years ahead. And Microsoft already has the infrastructure, investments, and ingenuity in place to see it all come to fruition.

This could be as big as the leap from 2D to 3D

I don't like making predictions because whenever I put text to paper like this, it opens me up to being WRONG, and how scary would that be?!

You don't have to go far to see failed editorial tech predictions, like the Daily Mail's infamous "The Internet is a passing fad" article from 20 years ago. Facebook (now known as Meta) has staked its entire future seemingly on VR and the so-called metaverse, which has yet to bear fruit and seems like something of a hype bubble poised to pop. Zuckerberg himself even admitted that the metaverse won't be a thing for at least a decade only a couple of years ago, which makes you wonder how long investors will accommodate such an expensive gamble. 

Microsoft has famously been relatively lukewarm on consumer VR, but it has gone deep on AI, which to me seems like a far safer, and far more lucrative bet. The benefits for businesses are obvious and don't necessarily require a huge amount of imagination. From speedier programming to intelligent customer service chatbots, the potential is massive, but the implementation in gaming is what excites me the most. 

Obviously, there are some serious problems to solve in certain implementations, but the potential is tantalizing. Vast and dense worlds populated with realistic and meaningful art and creatures, supplementary side objectives that free up dev hours for hand-crafted features and writing. Intensive tasks that take weeks reduced to mere minutes with cloud-powered machine learning algorithms, increasing the development speed of AAA games, and democratizing AAA quality assets and animation for smaller businesses and independent developers via ID@Azure. 

It won't happen overnight, but there's a revolution for gaming just around the corner that could be on par with the leap from 2D to 3D. 

What you need to know

• Grand Theft Auto V was released way back in 2013.
• It's recently been enhanced via machine learning to be nearly photorealistic.
• These enhancements are thanks to Intel Labs.

You would think that a game released on the Xbox 360 back in 2013 would be starting to show its age by now — and if you're only looking at the vanilla versions Rockstar has released, you'd be right and see that age in an instant. However, technological masterminds have used the game as a testing ground for all sorts of groundbreaking visual mods and graphics experiments over the years, with the latest bringing GTA V arguably as close to photorealism as any game's ever come.

Three members of Intel Labs have come up with "an approach to enhancing the realism of synthetic images." This trio consists of Stephan R. Richter, Vladlen Koltun, and Hassan Abu AlHaija (who worked on the project as an intern at Intel Labs).

Their work has been published in an in-depth report analyzing the utility of machine learning in improving synthetic visuals. That's a fancy way of saying that the three men have figured out how to produce hyper-realistic graphics in video games.

Mind you, the implications of this research are huge and extend well beyond GTA V or video games in general. But the video game applications for this technology shouldn't be undersold, either. Companies' dogged pursuits of photorealism in video games could be helped greatly if the research by Intel Labs is expanded upon.

There are caveats to the current demonstration of the potential of machine learning in gaming. Here's a key bit from the project's paper: "Inference with our approach in its current unoptimized implementation takes half a second on a Geforce RTX 3090 GPU." In other words, this technology isn't ready for mainstream gaming efforts. Still, the fact that the method works as well as it does on one of the current best graphics cards for gaming implies that the gaming industry could be close to making such methodology commercially viable in the near future.

You can see a lot of data from the project over on GitHub, where Intel Labs has published content related to the research. Perhaps there's enough info there for future iterations of this project to come to fruition at the hands of excited modders, well before the mainstream giants start tackling such processes. Who knows what the future of GTA V graphics mods looks like, now that this technological leap has been made public?

Different things come to mind for different people when they hear the words "artificial intelligence (AI)." Some envision Hollywood representations of A.I. as depicted in movies like The Terminator, The Matrix or I, Robot. Others conceive a more conservative image, such as A.I. players in video games or digital assistants like Cortana. Still, others envision the complex algorithms powering the intelligent cloud that provides helpful insights for decision-making in business. All of these are AI or, "intelligence exhibited by machines or software and the study of how to create computers and computer software that are capable of intelligent behavior."

Though Hollywood renditions of AI are extreme exaggerations of technology far beyond what we are capable of today, they offer a cautionary warning rooted in the ethical challenges AI currently presents. AI is fundamentally "made in our image." It is founded upon machine learning scenarios where humans provide systems with data to "create" its intelligence.

When human biases or limited perspectives forge the basis for how these artificial systems perceive the world, they invariably reflect the negative and often stigmatizing perceptions that plague human cultures. Thus, as AI becomes part of judicial, health, employment, government and other systems, it behooves us to temper its evolution with proactive guidance that preempts a dystopian manifestation of human prejudices.

1. AI, African Americans, gender and algorithmic bias

Many people strive to separate their consumption of tech news from sobering realities such as the sordid history of racism and the current biases African Americans and others still face. The interweaving of technologies that mimic human perceptions within our social structures, such as facial recognition, makes that an impossible task, however.

We must acknowledge that computer systems are only as reliable as the fallible humans who make them. And when it comes to AI's ability to perceive and distinguish between individuals of various skin colors and genders, some human biases can make it into AI systems. A study revealed that Microsoft's and IBM's facial-analysis services were frequently unable to accurately distinguish features of dark-skinned people, especially black women. The system's accuracy rate for white males was significantly better. Its "training" with a low representation of dark-skinned people contributed to the disparity. That "oversight" is likely a derivative of the deeper problem of low representation of blacks in tech. A more diverse workforce probably would have realized the data-pool deficit.

A more disconcerting (and admittedly uncomfortable to discuss) AI-and-race incident was when Google's photo-organizing service tagged black people as "monkeys," "chimps" or "gorillas." As of 2018, Google's workaround was removing gorillas and other primates from the system's vocabulary. Given the history where blacks have been compared to primates, the ethical impact of AI algorithms that echoed those prejudices is reflective of the need for a broad and diverse pool of data and people to preclude these problems.

2. Criminal justice

From government, shopping, education, transportation, business, defense, health care, and more, Microsoft and others are pushing AI into every aspect of our lives and culture. AI was used in the judicial system to determine if a criminal should be released. AI "decided" to release the man, who later killed someone. It was discovered that relevant criminal history data was not part of the data set AI used to make its decision.

3. Intelligent cameras

In 2016, Microsoft introduced AI-driven camera tech that recognizes people, activities and objects, and can access data about individuals and act autonomously. The potential misuse of this tech by governments, employers or individuals to track people, their habits, interactions, and routines is profound. Additionally, last year Google came under fire, even from its employees, for a Pentagon partnership that uses camera tech to analyze drone data.

4. Health care

In healthcare, multiple studies revealed physicians and residents believed blacks feel less pain than whites. Consequently, they prescribed painkillers less often for blacks than for whites with similar conditions. Consider the potential ethical and continued quality-of-care disparities if AI in healthcare is fed data from professionals who hold these and other biases.

5. Almost human

Last year Google demonstrated Google Duplex, an intelligent bot that could navigate phones calls, make appointments and was indistinguishable from a human. Ethical concerns abound when users are unaware they're talking to AI rather than a person.

Confronting the issues of bias and AI

Microsoft, Google, and others have begun addressing the ethical challenges AI presents. Internal boards have been formed and acknowledgments of AI's dangers have been included in companies' U.S. Securities and Exchange Commission (SEC) reports. Still, without external guidance, the lack of consistency and universally-applied standards remain, allowing an avenue for continued biases in AI.

Even with external boards, biases can remain an issue. Last year, Axon, the manufacturer of Taser, formed a board to review AI in body cameras used by police. In response, 40 civil rights, academic and community groups accused the company of excluding representatives from the communities most likely to be negatively impacted by the tech.

A.I. is increasingly part of our culture, and it and those creating it, governing its development and implementation should "look" like all of us. Groups like Blacks for AI and Women in Machine Learning are trying to ensure just that. Still, companies are pushing AI into products like smart speakers and facial-recognition checkpoints faster than adequate systems of accountabilty can be formed. It will take a collective effort from all of us, diligent oversight and an honest reflection on who we are to ensure the worse parts of us aren't part of the AI upon which we increasingly rely.

Microsoft today announced that it has acquired Berkeley, California-based AI company Semantic Machines. The acquisition, Microsoft says, will help it build upon its burgeoning AI efforts to make speaking with digital assistants like Cortana and bots more like carrying on a natural conversation.

Currently, interacting with bots and digital assistants involves issuing simple commands or queries, such as asking for the latest sports score or setting a reminder. Going forward, Microsoft wants to evolve these interactions to be much closer to how you would speak another human. That's where Semantic Machines comes in. "Their work uses the power of machine learning to enable users to discover, access and interact with information and services in a much more natural way, and with significantly less effort," Microsoft says.

Semantic Machines counts among its team several prominent natural language AI researchers, including UC Berkeley professor Dan Klein and Stanford University professor Percy Liang, along with former Apple chief speech scientist Larry Gillick. The company is led by technology entrepreneur Dan Roth.

As part of the acquisition, Microsoft says it will open a conversational AI center in Berkeley:

The acquisition comes not long after Microsoft touted a major breakthrough in conversational AI, which allows chatbots to talk and listen at the same time. This allows digital assistants and bots to have more of a flowing conversation with one another, operating much closer to how humans talk with one another. This week, the Microsoft's AI chief Harry Shum revealed that those capabilities, known as "full duplex," will be made available to developers and partners in Asia on June 1 for one of its chatbots, Xiaoice:

With Google recently showing off some similarly impressive tech of its own in this regard, it looks like competition in the conversational AI space is really heating up.

Eventually, we can likely expect these advancements to make their way to Microsoft's cognitive services for developers, Cortana, and chatbots like Zo in the U.S.

Another week and another batch of questions from you for me to try and answer.

This week I'll try to talk about Windows ML (it's OK if you don't know what it is), a Surface Laptop refresh, and what about getting more RAM into the Surface Book 2?

Grab your beverage of choice, check it out, and don't forget to share!

Audience questions for Episode 45

• Any rumors for Microsoft to offer increased RAM options in the Surface lineup, such as 32GB in Surface Book 2? - Joe B.
• Any sense if there will be a Surface Laptop refresh coming out of Build this year and any hope there will be a Kaby Lake R chipset or improved SSD tech in there? - @richmanab
• How does Microsoft plan to take advantage of #Windows ML? What kind of abilities could it have? -@tauheedul

Thanks to everyone for the questions!

Email me!

Don't use Twitter? After much feedback, we are happy to announce you can forward your questions to AskDan@WindowsCentral.com to submit your requests!

AskDan Forums!

Hate using email? Join me on our #AskDan Forums at Windows Central! I'll occasionally jump in there to try and answer questions or take ideas for new episodes of our web series.

Ask Dan Forums at Windows Central

You can also find previous episodes of #AskDanWindows here on the main landing page.

• Watch the #AskDanWindows playlist on Youtube
• Subscribe to #AskDanWindows in podcast format (RSS)

Called Gluon, the deep learning library is intended to give all developers an interface that allows them to build machine learning models using a Python API and "a range of pre-built, optimized neural network components." Both Microsoft and AWS are hoping Gluon can help all developers, regardless of skill level, build neural networks in a simple way without impacting performance.

As Microsoft explains, deep learning engines like Apache MXNet, Microsoft Cognitive Toolkit, and TensorFlow have managed to speed up a training process that used to take days or weeks, but they require developers to implement lengthy and complex code to build training models. Other tools make the model building easier, but come with a slower training period. Gluon aims to strike a balance between the two, giving developers an easy programming interface and quick training. From Microsoft:

Artificial Intelligence (AI) and machine learning are set to become an even bigger part of the technology and apps that we use on a day-to-day basis, and Microsoft and Amazon are hoping that Gluon will give developers a leg up. The Gluon interface is open source and is available today in Apache MXNet 0.11. Support for Microsoft Cognitive Toolkit is scheduled to arrive in an upcoming release. You can find out how to get started with Gluon and MXNet with tutorials for beginners and experts, or check out the Gluon interface on Github.

Microsoft yesterday announced commercial availability of their Microsoft Translator Hub. The service has been designed to allow developers and business to deliver tailored, real-time translation in up to 39 languages.  The real-time translation is available as an Azure Marketplace application, enabling users to leverage the Microsoft Cloud to deliver more relevant machine translation applications to their customers.

The applications vary for this service but as an example, it will allow for fluid, real time translation of live web chat interactions from customer services representatives. It goes beyond plain machine translation by allowing the output to be fully optimised for that particular application. A private set of data can be uploaded and using the Translator’s machine learning training system, output can then be tailored as required. Manipulating the output could allow exclusion of business specific nomenclature such as names of products or abbreviations. These customised translation scenarios can then be tested deployed so that they suit each application for which they are targeted. Once the solution is deployed, it can accessed through the familiar Microsoft Translator Service API, all the time being kept private or shared, depending on what you require.

Many of you may already be familiar with Microsoft’s translation engine. You can experience it though their Bing Translator Web page. Better still if you are a Windows Phone user (that’s why you’re here right?) you can simply download their excellent Translator app that allows for online as well as offline translation. The app uses a compressed version of the Bing Translation engine in conjunction with language packs to provide for offline translation. Currently it supports six languages Chinese, English, French, German, Italian and Spanish. It’s well worth a try.

What Microsoft are doing here is allowing its users a flexible way to combine their own customised translation models (using the Microsoft Translator Hub) with the solid existing MS Translator API. This in turn will allow those that use the service to build and deploy highly scalable solutions to fit their own needs rather than utilising a less than flexible off the shelf solution.

How does this stack up against Google’s Translation offering?

Whilst Google offer a similar set of Translation services for machine translation applications, they provide support for up to 60+ languages, which is almost double that of what Microsoft currently offers. Microsoft look to be offering users a way to leverage their machine learning engine to help customers maximise the relevance and accuracy of the translated output.

Providing the Translator Hub to users could just be compelling enough for its customers to choose Microsoft’s cloud based service over that of Google’s.  As the MS service is a linguistically informed statistical machine translation system, I would expect that its machine learning would only get better the more it is used. Perhaps Microsoft will, by proxy improve its service simply by seeing how their Cloud based system adapts to the various custom data sets that will be finding its way to them via the Translator Hub?

Machine Dreams

One of the things I find the most exciting with these cloud-based machine learning systems is the potential for improvement and progress for everyone. The collective input via millions of transactions of speech recognition and text translation queries via the web, combined with powerful machine learning, should lead to rapid improvements in the quality of results.

I’d certainly like to hope that this improved backend will benefit us directly with faster, more accurate speech recognition technology. Microsoft have revealed a few teasers of their proposed improvements to the Tellme speech engine. Currently it is fairly limited but its been demoed at the recent the Windows Phone 8 summit engaged in what they call conversational querys. So there will actually be a two way interaction with the service, this will be made available to 3^rd party developers to power their apps.

If you haven't tried Microsoft's Translator app for Windows Phone (it has augmented reality!) you can grab there here in the Marketplace for free.

Source Microsoft via ZDNet’s Mary Jo Foley