Microsoft wants to teach Cortana to 'do' (not just 'know') more

By Jason Ward
last updated

Microsoft is sending Cortana to school.

The Cortana Intelligence Institute (opens in new tab) is a collaboration between Microsoft Research, Cortana Research and RMIT University in Australia, focused on developing Cortana's next-generation capabilities.

Microsoft's approach with Cortana has always been different than what Google and Apple are doing with their digital assistants, Assistant (previously Google Now) and Siri, respectively. Cortana was intentionally patterned after real-world personal assistants, notebook and all.

The premise is that Cortana would, like real assistants, get to know a user better over time and subsequently be able to proactively serve that user. She also provides time-, and people-based reminders. For example, if a user needed to be reminded of something in relation to another individual, an interaction such as a phone call or email from that person would trigger the reminder. . Location-based reminders are another example of this type of personal support.

Cortana's (and other assistant's) ability to tie into a device's GPS and recognize interactions with specific people are examples of context-aware features the Cortana Intelligence Institute will build upon for next-generation capabilities.

Assistant and Siri tell what they know

This current generation of digital assistants, with all its promise, is in the embryonic stage of AI and ambient computing. As such, Alexa, Siri and even Assistant are glorified voice assistants with a mere touch of AI. Their current strength is providing information.

Google touts Assistant as a user's personal Google, and the company's mission is to "organize the world's information and to make it universally accessible and useful." As the world's leading search engine, upon which Assistant is based, information provision is Assistant's forte. Alexa's position via Amazon Echo is similar. It does, however, wade a little deeper into "doing" with its ties into Amazon's shopping experience and the breadth of Alexa Skills extending its abilities (opens in new tab) via third-party developers.

Ties into smart home appliances also make these assistants useful at performing certain tasks on command. Still, their abilities to perceive and directly support a user's specific activities is something that remains in the realm of science fiction. Microsoft's goal is that the Cortana Intelligence Institute's work will make that vision a reality.

Cortana will do more

"Bing and do," was Microsoft's tagline for its search engine during the product's early days. It was positioned intentionally as a decision engine rather than a search engine emphasizing the product's goal to help users "do" things rather than just provide information. This positioning could have been, in part, Microsoft's marketing strategy to avoid head-to-head comparisons with Google due to its much weaker search prowess.

It could also have been a reflection of the company's fundamental mission through its products and services to help individuals and businesses "do more." It is this foundation that coincides with the Cortana Intelligence Institute's goal to expand Cortana's context awareness and enable her to directly assist users (opens in new tab) with completing tasks rather than just providing information. From Microsoft:

In the future, Cortana may assist with chores from cooking to calendar management, which involve multiple steps, contextual awareness and a rich dialogue with users. …Their focus is largely around using context to better understand the tasks that people.

This goal to help users "do" differs greatly from Google's vision to help users "know."

Microsoft's digital "do more" assistant vision

Microsoft's AI vision is a cloud-based intelligent cross-platform agent that knows a user across work and life. Her integration with Office 365, Microsoft Graph, Windows and her presence on iOS and Android give the personal digital assistant the ability to know users quite well. Microsoft's goal is to make this seamless assistant capable of perceiving and supporting a user's needs no matter what device he is using. If Microsoft can ultimately realize its personal digital assistant goals with an assistant that can help users do more from any device it may have an edge over competitors.

Microsoft's great at research but weak with doing.

Of course, Cortana is behind its competitors in both mindshare and market presence. The lack of a smartphone platform and a weak smart speaker presence in comparison to Alexa, Assistant and Siri make Cortana, even with her Windows 10 presence, a virtual unknown in the consumer space.

Microsoft is excellent with cutting-edge research and development. The implementation of those ideas into a marketable product, however, has proven to be its shortcoming over the years.

Jason Ward
Jason Ward

Jason L Ward is a columnist at Windows Central. He provides unique big picture analysis of the complex world of Microsoft. Jason takes the small clues and gives you an insightful big picture perspective through storytelling that you won't find *anywhere* else. Seriously, this dude thinks outside the box. Follow him on Twitter at @JLTechWord. He's doing the "write" thing!

63 Comments
  • Cortana need to go global, I see it successful in that case and to be faster on response.. I know Microsoft can do it if they want to.
  • So people all over the world can ignore it & not use it.
  • It's becoming Microsoft's "IBM Watson".
  • The problem is, every time MS tries to make an AI bot it turns racist and has to be taken down. Cortana is no different, and certainly does not like anyone from outside the US. Currently she never explicitly states this bias, it's just made obvious by the sub-standard way she treats people who persistently remain outside the US but if they keep pushing she'll end up like the rest and will start to let everyone know exactly what she thinks.
  • That happened once because it was a public facing AI and people figured out how to train it. 
    Cortana is VERY different. It isnt learning conversation like that. It has a set of skills and language processing. Its not really AI. 
  • I think, and hope, that was a joke. Humanity is always the issue, not the robots. =P
  • Uh, people outside the US are  white too. It has nothing to do with racism. Cortana isnt racist. She is blue, after all.
  • The limitiation is not the technology or Microsofts willingness to provide the technology world wide.  It is ready! Unfortunately, many countries often have strict laws that prevent such products from quickly entering their markets.  It takes time and lots of legal paperwork and lots of money.  If there are only a small number of potential customers in a country, then it is just not worth it. Its politics my friends, not technology or Microsoft holding up these product rollouts.
  • Has that stopped other Amazon, Google or Apple?
  • I'm seeing a lot of language localization quests in this bug bash. At the end of this months Insider Mixer broadcast we were promised a big announcement that we've all been asking about. The event is listed on the Insider Blog as Mixer Broadcast: Cortana. I'd rather they slip Andromeda but I'd settle for a more worldly Cortana.
  • If you mean more region and language support, I agree.
  • Ironic they are working with an Australian University, here in Australia Cortana is an embarrassing joke, not a patch on what she was about 4 years ago under 8.1, so that makes her a decade or so now behind the current competition. Typical Microsoft, a decade late to the party......again.
  • Cortana isn't even fully enabled in English language countries like - England. Of course, translating American to English is a tough computer science issue.
  • Most of Microsoft management is big on vision but don't care much about execution. They still operate on their own timeline as though they're still in their monopoly days. These managers have to step aside and let the younger generation take over..until then it's going to be very painful and a waste of time to be in Microsoft camp. 
  • Cortana has no skills.  Who wants a digital assistant without any skills?
  • Cortana amazes me with all of her capabilities.  I am not sure what skills you need but she certainly meets 95% of my needs.
  • Insteon voice controls is one I would like. Insteon launched their hub supporting Cortana early on, but quickly dropped it. They now support Apple Home kit with siri, Google assistant and Alexa, but not Cortana. 
  • Insteon is in the Smart Home option for Cortana. What doesn't it do that you are wanting it to do? And for people touting Alexa's skills, what skills does Alexa have that Cortana doesn't. Instead of people spouting off numbers of skills, we should be specifying what is desired else it is the app store debate all over again.
  • I have both...I use cortana most of the time, but 1. REPONSIVE Alexa is so much more responsive and definetly better user experience...  2. MUSIC with alexa I can just use Prime music that's part of prime subscription, but I'm forced to get Spotify with cortana  3. PLATFORM :  Developer spending time on Alexa can get better exposure becuse of the network effect that Amazaon has aggressively pushed for...where's Microsoft?  
  • 1. Response on Invoke is just as fast if not better than echo.
    2. Amazon's free service is not the same as Spotify. Also, you could use iheartradio or TuneIn on Invoke.
    3. I agree with this point. But Microsoft seems to be riding the free boat here. E.g. Smart home devices that work with Alexa, in turn automatically work with Cortana.
  • Cortana works with my z Insteon. It didn't when I had the old 2224 model hub. You just need the newer hub that is advertised as Alexa capable.
  • sniperboywc is correct, as here in the UK at least, the vapourware known as skills doesn't exist!!!!
  • I think I know what that other 5% are. You perv!
  • Have you tried Cortana?
  • This. What Cortana does "she" does better than ever other digital assistant. The problem is that "she" can't do much.
  • The sad thing is ...microsoft has so much tech in the labs that can make Cortana unique...but they simply don't seem to have the capability to execute...there is no sense of urgency... The Hololens team created a poll on what app they should build  - most voted for choice was life size cortana.  When they just decided not to do it  - I realized Microsoft is a company run by people that doesn not have the Will, Desire or Capability to deliver on what the users want.   It's all about their genius vision and their own timelines that doesn't take into account what's happening in the market.  It looks like it's game over in samrt home and wearables...can they at least deliver on Surface Mobile and Mixed Reality...I don't know
  • Spot on this happened with the xbox one. But now they have changed that and made the xbox x from what gamers wanted so there maybe light at the end but a lot has to change. I was a big Microsoft user I used windows mobile 10, groove music, cortana, one drive, office xbox and others but I now have a Sony xz premium switched to Google home and Google music because Microsoft dropped the support for the devices and as a Microsoft fan that's not good as I was forced to take up other services.
  • Cortana is brilliant. I adore her. But I hate that she only lives in all her glory on my desktop. Just yesterday I asked Alexa for some simple factoid and she once again came up empty handed. I flipped open my Surface, invoked Cortana and got the information I needed in seconds.  At that point, my wife and I lamented that Cortana's biggest weakness is that she isn't on the delightfully powerful and cheap speakers that Alexa is. I can get every room in my home Echo'd up for FAR less than if I bought a bunch of Invoke speakers.  Entry level pricing and availability is key here. I can walk into my local Kroger/Smiths and grab an Echo Dot for $25 on sale right now. Cortana is late to the game. Alexa has my home, but Cortana has my heart.
  • Spot on. the $29 dollar Dot put Alexa in my home When I had a 950XL I was amazed at how well Cortana worked with my car system. I now have an iPhone and my car has Apple Car Play, or whatever it's called, and Siri is still behind what Cortana could do all by herself. Even my Icon with Window Phone 8 worked better than Siri with her dedicated Car Play...
  • Would this $30 Xiaomi YeeLight speaker work for you?  It is supposed to support both Cortana and Alexa. https://www.voicebot.ai/2017/12/29/xiaomi-yeelight-offers-30-cortana-pow...
  • This! Where was this 2 months ago? Missed the holiday season, but I do very much like the look of it.
  • Yes, I agree knowledge ops absolutely must also be accompanied by execution capability. But, as I've said many times, NONE of the assistants have cracked the omnipresence piece. That ABSOLUTELY must be accomplished if ANY of these assistants want to win.  Cortana is best postured to do this, believe it or not, but Microsoft goons are too ignorant to realize what they need to do. STOP trying to get Yet Another Speaker.  This is just beyond stupid.  It's bad enough they abandoned their own phones, but Cortana DOES exist on the other phones (which I'll never use).  Cortana exists on Xbox, PCs, tablets.  Cortana already has a presence virtually everywhere.  But Cortana is NOT omnipresent, which is the problem.  Cortana is a different entity on each of those device types.  Why the heck brag about Azure if you have yet to LEVERAGE that omnipresent backend to unify all instances of Cortana?  Cortana should not only follow you through work and life, but literally follow you any PLACE you go. Cortana should recognize YOU, individually, on ANY device that can hear or see you.  Immediate coordination and triangulation should be taking place.  Cortana should be able to hear you give a command, quickly discern which device picked you up the strongest (proximity) and then analyze what you've said.  Cortana should be able to decide either A) where the response for that command should logically be presented or B) ASK you when it can't determine.  When I say, "Hey, Cortana" I shouldn't get different responses from different devices, which is exactly what happens now.  When I say, "Hey, Cortana, turn on the Xbox" I hear three different devices "react" but then I also get three different full replies.  Cortana should be able to tell that the same command from the same person came across three devices at the same time, but based on the context of the command ONLY THE XBOX should execute it.  The phone and the tablet should simply have gone silent, handing responsibility for execution to the device that it made sense to do so. If I say, "Hey, Cortana, what's the weather like tomorrow?" all three devices could hear my command, but the device it received the command the strongest on would logically be the one to respond...OR Cortana could ask, "which device would you like that answer?" using the device closest to me.  Or, knowing multiple devices could hear me, I should be able to say, "Hey, Cortana, display the weather on my tablet".  All other devices would go silent and the weather would come up on my tablet. But to keep this idiotic idea for buying a special speaker needs to die.  We have 3 to 5 "speakers" and "microphones" in our house already.
  • You basically described how Google Assistant works. You would be much happier in a better ecosystem. Microsoft's consumer future if basically dead at this point. You are beating a dead horse.
  • No, that's not how Google Assistant works. I've tried that with my friends who lower themselves to using Android/Google stuff and the same problems show up. As I said, NONE of the assistants work the way I described.
  • You have great ideas.  I hope that they are implemented in the near future. I love Harmon Kardon Invoke.  The sound quality and the microphone array are excellent.  I am not interested in buying cheap junk.  I like quality products.  Should a speaker/microphone product run anyone's digial assistant?  That would be nice.  Unfortunately, that product doesn't exist yet.  Perhaps soon. Until such time, Harmon Kardon Invoke and Windows 10 are what I use at my home, office, phone, car etc. to interact with Cortana.  One solution that works everywhere for me.  I love it!  I look forward to new features comming soon.
  • Cortana should be more capable of helping me be productive as she has so much access to my data and my knowledge of the platform. It's just not really that useful.
  • "The premise is that Cortana would, like real assistants, get to know a user better over time and subsequently be able to proactively serve that user." Wasn't this exactly what Google Now did? It tried to anticipate the info you needed and put it into your feed so it would already be there when you looked for it.
  • No, Google Now was nothing more than a glorified web search. It still relied on your input into what you wanted to see in your feed, just like Cortana does at the moment, what that quote means is that Cortana would be able to look at what you are doing and serve up the information you need when you needed it without any input on the part of the user.
  • I always replace the warm and fuzzy assistant name with "the government" to put the feature in perspective and enable me to see if these features are something that I want anyone in the world (government, hackers, corporation)  to know about me.
  • You're already carrying a recording device that tracks your movements & has an always on data connection.
  • "Alexa, please dont tell NSA I just ordered Dominos"
  • Oh, good grief. 
  • Good article, Jason. I really want to see this come to fruition. Cortana is way more competitive than many would think, it has quiet a potential to 'Do' more and that's what I love about It (Her, perhaps? ;)). Ppl fail to realize it because of Cortana's limited reach worldwide.
  • I have computers, Xboxes, phones, and an Invoke  all with Cortana on them but still have to use an Echo to control a few things that Cortana can't. It's incredible that MS has sat on this goldmine for so long and watched the other guys go wild with features they could have implemented long ago. Being slow has cost them another market where they had unlimited potential but didn't act fast enough.
  • here we are 3 years after cortana was introduced for windows 10 and there has been essentially no improvement in functionality. sure, they added little things while trying to catch up to the competition, but not a single feature that harnesses the power of a pc. first and foremost i want to have a solution for cortana and video playback. i cannot fathom why cortana doesn't just emulate media playback keys when you say "hey cortana, play/pause video" or the like. there should also be a separate cortana ui when she is triggered while viewing video, the behavior of which can be customized:
    this should entail playback options like pausing the video while cortana is open OR keep playing as is OR keep playing while mute OR keep playing with reduced volume. There should also be options to set how cortana is visualized while watching, meaning having an overlay over the video OR having the video slide to the side/resize, making room for cortana OR having purely acoustic feedback with minimal if any visual aid. then, there should be a way to close cortana by voice. how is this not a thing yet? trigger with "hey cortana", close with "thanks cortana". these are just a few of the essential features they should implement asap. it's truly painful to see microsoft again squandering time and opportunity. cortana was one of the things i was most excited for in windows 10 because i can picture many scenarios in which this sort of assistant could be helpful, but even though i've read many articles like this over the years promising productivity improvements, there has been no real progress; and on top of it all, the cortana ui still looks really shoddily put together.
  • Do more? First learn other languages please!
  • your language not here https://support.microsoft.com/en-us/help/14236/language-packs  
  • Those don't relate to Cortana though. They relate more to the OS language.
  • "The Cortana Intelligence Institute"?  Sorry, but I can't read that without giggling.   But seriously, all of this pie in the sky stuff.  Hey Cortana, do this and that.  This all sounds very dystopian.  Are we REALLY to GD lazy to check the weather on the 5 weather channels on the 65" TV?  Or look at the tablet in our lap?  Pulling the phone out of your pocket is too much work?   Would you ask your wife/husband/brother/whoever what the weather is going to be?  "Mom, what's the average annual rainfall in Nova Scotia?"    If I was programming these things, I swear I would put in a randomized answer every now and then of "Go look it up yourself, I have better things to do".    I have never used any of these so-called "assistants".  I cannot envision any scenario where I would need to.  I can type a question faster.  
  • You underestimate many things.
  • No, I don't.  
  • From your ranting, are you against all AI as not specifically Cortana and the intelligence institue.  Cortana, as microsoft stated it's like assistant plus more.  Whether you use it or not, please it's not pie in the sky.  Don't hold back progress because of the pie you don't like.
  • 😶. "Bing and do more" oh where have I heard that before?
    Oh, right it.is.every.single.time speaking of bing all the amazing features in windows phone 8.x was locked to the US. Sound familiar?
    However bing's market share was hit and miss globally.
    I don't know what it is but risk aversion seems to run deep, now they are playing catch up once again in sector where they were once leading. However there is another factor in play, the magpie drive for short term profits combined with risk aversion, is the worst possible combination when it comes to company growth. The transitional play is absolutely flawed, as long it continues to be flawed they will lose ground to competitors in terms of profits, mindshare and users.
    At this inflection point they need to focus on their own ecosystem and making it accessible globally. Own ecosystem first before their competition, that is what Apple and Google have been doing as result they have Microsoft in a corner. Now Amazon is a contender in the Ai space with OEMs pushing Alexa over pre-included Cortana. That should tell you how flawed this chase of short term profits has become.
    At this rate Surface is not the only division that will flat line out.
    They need to focus on UWP, Polaris, Andromeda and other consumer focuser hardware. Surface growth won't be fueled by enterprise but by consumers. As enterprise customers don't buy new expensive hardware every year, consumers do.
    But... Which sector they decided to forego?
    Consumers.
    Sigh.... this is how become so circular that it's bordering on tedium. Microsoft's new motto should be "semper tedium" always boredom. The vibrancy and drive to be different and enable ALL people to do more was what made Microsoft what it was.
    Now I fear Microsoft will be a ghost of it's former self as long it continues to engage in the fallacy of short term profit creation.
  • They are no where near an ecosystem of their own. Cortana is yrs behind n Jason wants to convince us that its soon gng to bre great. 
  • What Microsoft is trying to achieve is making Cortana an PROACTIVE assistant rather than a REACTIVE assistant like Google Assistant and Siri. What this means is that Cortana can retrieve information based on where you are and what you are doing without any input from you.
  • As of now it lags even the reactive assistants
  • That's just playing with words. Alexa in particular can "do" a hell of a lot more than all the other voice assistants - and Cortana is at or near the bottom of that list.
  • @techiez, the thing is when Ballmer left he put in place an all ecompassing ecosystem pathway. But that was ripped to ribbons in the favour of rapid, short term profit driven growth as opposed to organic growth. Companies that grow organically tend to last decades if not centuries, the companies that engage in the fallacy of short term profit driven through greed do not last as long and fold within a few decades.
  • As I have commented more often and as you afirm now Jason, Microsoft reminds me of Philips from The Netherlands, very good in research, but marketing... Now they have completely sold of their consumer business and licensed their brand name for such products. Now it is a world famous player in advanced medical equipment. Let's see where Microsoft ends up. I think they should adhere to their dual use philosophy.
  • I don't think you should compare MS with Philips.  You need to know where MS strong point, enterprise, and desktop platform.  Think about google, the very first page you see on internet explorer "WWW.GOOGLE.COM", where do you access it from?  Where is google now?  Android?  How did google become household word?  So that's what MS is.  It's not apple or Android. To solidify my point, how do you access Steam if not windows by MS. MS is trying to capture consumer space, eventually they will get there, could be next year, 5 years?  Who's stopping them, everyone in this room?
  • Busytalk...
  • Given her low IQ and lack of linguistic talent maybe that's the way to go for her. Maybe a wakeup alarm function is on her skill set, beep beep beep ... or a flash light ... I guess a calculator would probably be a step to far
  • So we're just going to ignore all the things Google Assistant and Siri and, oh yeah, Alexa, can "do"? Is this a press release?