Microsoft is working Cortana with windshield projection into the connected car

By published

Microsoft is already working on getting Windows into the car with a new UI that's familiar to those rocking Windows 10. The company demoed what progress has been made thus far back at Build 2014, but now we've learned that Microsoft plans to add Cortana into the mix for some speech recognition.

During Microsoft's TechDays event in Taiwan, chief operating officer at Microsoft Asia-Pacific Research spoke about a concept that integrates the personal assistant with the windshield, better aligning information for quick glance viewing, rather than having the driver turn their head to face the center column. It's also reported the company is working with Taiwanese firms to further develop connected vehicles.

It's still early days for consumers, but it will certainly be interesting to see how a standardized Windows platform would go up against Apple CarPlay and Android Auto.

Source: Taipei Times, via: The Verge

Rich Edmonds
Rich Edmonds

Rich Edmonds is Senior Editor of PC hardware at Windows Central, covering everything related to PC components and NAS. He's been involved in technology for more than a decade and knows a thing or two about the magic inside a PC chassis. You can follow him over on Twitter at @RichEdmonds.
137 Comments
  • Tiles would work for cars ahold alot better than icons... Imo. However them flipping would be a bit distracting.
  • I rented a Camaro once that had the speed projected onto the windshield.  It actually looked as if it was hovering over the hood of the car and not 18 inches in front of my face.  It was a pretty cool feature, but I could imagine having too many pieces of information up there being a distraction.  I could see some pretty cool augmented reality possibilities, though.  Imagine having the GPS route overlaid on top of the streets.  Pretty cool possibilities.
  • As the owner of a camaro, this was one of the reasons i bought it. I kind of wish we had more that we could do with the hud overla though. Seeing GPS turn directions would be nice.
  • Mazda6 has this, a popup clear HUD with current speed and hooks into car navigation to show how many feet to next turn, arrow indicators, etc
  • Guess the factory fit camaro gps unit is tied in with the HUD because the models ive used have turn by turn directions pop up onto the HUD with distance to turn and destination among details displayed. Like niello said above, it would be cool for the route to be overlaid onto the streets once a destination is set. Lot of other possibilities like cortana displaying immediate events sceduled, onto the windscreen as pop up etc etc
  • he new BMW X3 and upwards have the HUD which lists the GPS directional arrows and speed on the HUD display. Only the driver can see it from the driving seat angle - the rest dont see it - thoughg the same window is also there on the dsh screen. Pretty cool and Useful.T
  • Yep, Camaros and corvettes have it. Corvettes started working on a version of the projected Speedometer in the late 80's
  • Grand Prix's had it even before the Corvettes worked on it.
  • And jet fighters had it in the 1960s...  How does this apply to vehicles consumers drive daily? Corvette introduced this technology, along with many technologies that have become standard in automobiles.  (The 1984 Corvette was the most technilogically advanced car made at the time, and still holds records.  Cherolet continued to add technology, keeping this title for the next 30 years.  Even today, there is debate of which is more technilogically advanced, the Tesla or the new Corvette. )  
  • There are these: http://m.ebay.com/itm/Head-Up-Display-Car-HUD-GPS-Dashboard-Mounted-Wind...
  • Wow. Someone has their painties in a bunch.
  • Cadillac does this for models equipped with HUD.  It's turn by turn though, so you only see upcoming turns/direction - not a whole map (just for clarity).  It is nifty and makes more sense than glancing over at the center console.
  • For tiles I would prefer slow fade out effect. It will be better than flipping
  • I think everyone is actually forgetting that the HERE team is doing pretty good on their own...91% of the cars apparently... http://360.here.com/2015/09/10/here-maps-lead-the-way-in-91-of-new-cars-...  
  • First work with our Facebook App Microsoft....Then Do That thing...
  • That's on facebook, not MS...
  • But it's Microsoft's fault.
  • Um how is it microsofts fault that facebook cant be bothered to make an app? Is it microsoft's fault that every other developer does not want to support WP? Theres only so much bribing they can do to devs to get them to start making apps.
  • Exactly...
  • Actually it is Microsofts fault that they have not managed to get more marketshare so that no app developers bothers to make good apps! There is no way to go around it...Microsoft marketing sucks!
  • Then it's your fault. You should have bought 10000 windows phones if you are too concerned.
  • lol if Facebook is more important than a safe functional computer in your car, you have some serious priority issues.
  • Considering how late to market this will be compared to Apple, Google, and even Blackberry's solution...  They probably should get a good Facebook App developed so that people can actually care about thier platform.  People won't care about this, if they have iPhones or Andorid phones and CarPlay and Android Auto are available.  They'll just get that instead.
  • Microsoft created the AutoPC industry. They set the standard and left it to the OEM's to take it from there. Now they have something nw to offer and are trying to set a higher benchmark. Apple and Google are doing nothing new. Look up the Clarion AutoPC.
  • It's really ignorant of you to think all of Microsoft's 100k+ employees should stop what they're doing and improve the Facebook app just so you can like some dumb post. There's a team for the Facebook app and they work on it, just like every product, service and department has teams of people to work on specific things. Take your head out of your ass and go do something useful rather than sitting around all day complaining.
  • Best comment reply that I've read so far... Thumbs up!
  • See comments on the store and you will learn who users go away from wp
    I guess Fb app is 90% of the reasons
  • The comment of a moron.
  • The only problem with your post was that Microsoft doesnt make the Windows Phone Facebook app... Facebook does....  
  • Yeah because the same people are defintiely working on both projects. Pointless comment.
  • Yes, because facebook is so important in peoples lives.  Get over it, I would rather have W10 in a car, than Microsft worrying about a silly facebook app.
  • Lol unfortunately yes, it is far more important in people lives, it's just numbers. # People using FB app every day >>> # people using more than a radio and GPS in a car :)
  • So, moving ahead in the future with a new technology is less important than you updating a status on Facebook?
  • Not to him maybe..... But it is to the majority of users. Also there is a lot more to fb than status updates.
  • Its waste of time to be on facebook..Its wast of time to talk about facebook....... Oh s****t I just wasted my time!!!!
  • LOL
  • Well they lost ford
  • They lost Ford with Sync. Sync was cool but aged quickly.
  • Yup
  • Would be nice to see this in Jags
  • I already did this with a $50 Bluetooth kit and my Windows Phone. One button to control everything by voice, pairs with multiple phones and doesn't flash crap up on the windshield which I would find incredibly distracting.
  • I find glancing at turn by turn on my phone distracting. I'd rather that be overlayed on my windshield.
  • It's not bad with a 1520. Of course, I'm not glancing that much. I just use the spoken turn by turn directions. Keeps all the attention on the road where it should be.
  • Its not too bad with an overlay. I even have a similar set up for ski goggles, and I've never had a problem.
  • Ok fine but Microsoft should do one thing at one time na...They update Cortana day by day but with the Facebook app they forget the app i think...So if they want to Success with Windows 10 mobile they need a good apps with all features like Android and Ios have...If they don't do this Then maybe The Company Wrap up things with their cars and with Cortana...
  • You make it sound like Microsoft has one engineer. Work on car stuff is so far removed from the team working on the Facebook phone app. I imagine both teams are working hard on their respective projects. Honestly, at this stage, Facebook has all the tools to easily bring their fully functional app to Windows. Sure, it may take some time for testing, but I cannot imagine that the Candy Crush developer (King?) would have access to these bridge tools before Facebook.
  • If they had more than one engineer(ing team) they would've been able to launch Windows 10, Xbox Win 10 and 10 mobile at the same time.
  • That's just silly. Of course they have more than one team. Projects run on their own schedules and some have dependencies on others. I assume (no inside info), it goes something like this: core OS (w/o UI) -> desktop and mobile-specific requirements (where the desktop-specific team probably does have a a lot more people than the mobile-specific team). For XBox, which needs a different UI, I suspect the requirement is the shared store and certain API's for developers to encourage universal apps with Windows 10.
  • Are you kidding me? Let's count the kind of changes that these OSs bring:
    1. A wholly refurbished visual look, which take a hell lotta time to finalize.
    2. A browser made from the ground up, which is no easy task.
    3. A newly designed, unified, one-stop store that is capable of providing users with all requisite information without making it look messy.
    4. Completely rewritten stock apps, which are much better than the solutions on 8.1.
    5. A digital assistant that is relevant on a bigger screen and to a different section of users.
    6. A completely revised dashboard for an already established system.
    7. A new app model.
    8. New development APIs.
    9. New porting software.
    Need I go on? And you want all that at the same time. Good luck. It's 2015, not 2025.
  • 1: A android-bastardized look that isn't exactly blowing peoples minds.
    2: A browser with the same problems as the former, with random webpages not working properly.
    3: True. Well, it will be eventually, at least...
    4: Except they're not. No pinch to zoom in Maps, etc.
    5: A digital assistant that only few have actual access to. Not exactly impressive, compared to the competition.
  • 1: Subjective response to a factual comment 2: Which problems do you mean. I never had a random page not work on IE 4: You provide a single example for an app on a beta version of an OS, not really the best argument 5: Again you provide an argument against something in Beta, with more users gaining access as time passes. You're also comparing everyhting to a mobile world, considering the OS's MS has developed holds the highest market share in the world in total devices, I'd say they are doing alright when compared to the competition.
  • Your logic is extremely flawed. Different hardware has different UI's different requirements, and different issues to deal with. Considering we don't have numbers to tell how big each team is, we can't even begin to guess the speed at which each products would be devloped, what hurdles they had to covercome, and what the developement cycle was like. Not to mention the fact that Windows Phone and Xbox were obviously being worked on during the time that Windows 10 was. In short, you're wrong.
  • Honestly, complaining in this forum isn't helping your cause.. You need to tweet and post your thoughts, etc on Gabe Aul, Joe Belfiore, and the head of Windows teams twitter and Facebook accounts. That might help
  • You already made this comment, why make it again a few minutes later?
  • Hopefully they can win over some big names but I bet they struggle after the negative crap people had to say about sync (personally I found it easier to use in the Ford my wife and I had then my friend Lexus or my other friends BMW infotainment systems but that's just me)
  • wow! windshield projection?!! like in mission impossible 4 BMW style?! if true I love MS for such innovative ideas but i also hate them for trivial reasons, anyway, waiting to see how it turns out
  • Windshield projection has been around for years.
  • Again, late to the game. Car play is already integrated into number of car models. Audi is already using Android for its systems. Microsoft? Oh, still testing.
  • Not late in this case as they were in the automotive space for years. Rather than drastically improve their systems they abandoned development and manufactures got fed up. They could have owned this space but typical MS.
  • Indeed, I'm perplexed why they could not work with Ford and fast track a WP7/8 based infotainment system years ago when sync was having issues. They had the OS, they had cheap hardware that booted fast, they had apps, they had maps. All kinds of fail.
  • An article I read blamed a lot of the problems on Ford dictating how things had to be - its been a while though and I dont remember the details.
  • Yep, they'll have this ready in 2 or 3 years when no one cares
  • Sadly Microsoft is very slow again this time. Though they indeed have embedded solutions for car infotainment systems, its only proprietary for each car models. Apple and Google already acting in this space and fast. Pretty soon they could saturate the market again, leaving Microsoft not enough room. They really have to act fast, since cars last for years and people don't normally buy cars every year. So that case Microsoft will miss a boat for a long time for many users. At least a good thing though, it uses MirrorLink which is much more open and available to many car manufacturers.
  • Does it really use MIRRORLINK??  That WOULD be COOL!  I and others have been BEGGING Microsoft to implement MirrorLink ASAP in WP8 and 10.   Cortana works a lot more intuitively with my 2015 Civic than the Android stuff in my HTC One M8, but MirrorLink gives me in-dash NAV through my phone's Sygic app and it really does work WELL!!  I've wanted a really good HUD in an affordable car for years, but MirrorLink almost trumps that.   So what's it going to take to get MS' new tech retrofitted???  
  • Cortana isnt avaiable in Brazil.
  • But it will be.
  • Microsoft needs to pursue the aftermarket options also. Plenty of people would be interested in adopting these systems if they didn't have to commit to a new car. Obviously, the feature set would be less than the integrated options but still puts MS in people vehicles.
  • Unfortunately aftermarket audio has been slowly dying out. That's because more and more automakers install high-end audio as standard nowadays.
  • Not to mention, changing out infotainment units is not as easy as it used to be. And many of the cars features are built into them, making removal problematic. 
  • Still Microsoft should pursue the aftermarket options also. Built in systems grow old fast. Are slow. Proper Miracast system could be enough already.  
  • I don't understand why they have to reveal this only for the copycats to implement it before MS own product even make it to market.
  • Because they dont have anything and they dont want people to completely forget about MS in this space. That said, they dont have anything and thats really all that matters.
  • There are several of these products on the market. There is limited danger of someone copying MS in this space.
  • Limited? check that Lumia living images... that the fruit morons thought they invented.
  • Too bad the interface won't be as nearly sexy as that mockup. Windows 10 UI is crap.
  • Have you seen the final product?
  • Yes, I'm from the future.
  • You need to go back to the future.
  • Cortana HUD mode would be awesome ...
    projecting directions, weather, notifications etc on windshield:)
  • I already have Cortana working in my car.
    http://m.youtube.com/watch?v=GSZAChzfCNQ
  • I commented on your youtube video as well, but what hardware are you running here?  I'm unfamiliar with CarPC.  I've installed my own headunits over the years and I'm curious if your setup is feasible in my vehicle.
  • I'd love this UI in my Car. MyTouch on Ford Sync is good, but this would be better.
  • This is what Google glass should have been from the beginning. The only place our daily lives need a HUD is in the car.
  • Based on the picture, looks like MS did it again. Better UI.
  • Sweet
  • While carplay n google auto are done
  • I have yet to see one car with either installed. Lots of promises, no results. 
  • I have a Pioneer system that does both... Really hoping MS would have something by now.
  • If apple makes a car,will it have windows?O_o
  • Apple, with all it's clout, is even now having problems getting this into new cars. What prayer does MS have. 
  • Probably because they keep poaching :P, can't keep their hands to themselves can they :P lol.
  • Because most Car systems are Windows CE based. So Microsoft has the connections already.
  • I'd buy that for a dollar.
  • Haha
  • What about support for more than just a few regions? Well not high hopes on that, most likely 'Cortana on a toaster' will come first.
  • Actually that might be sooner than you think :P.
  • For the life of me, I will never understand why MS didn't buy HERE. If MS are serious about getting into this space, having the Here inroad of already being in 4 out of every 5 cars would have been a massive advantage surely?
  • I have HERE maps in my Acura and it sucks (not to mention the $150 they want me to spend every year on map updates), so I just turn it off and prop my 1520 over the screen and use the HERE app instead.  It's so much better.
  • HERE provides data services for car makers. If car makers make crappy app and charge a lot for updates its their doing.
  • Yeah, what I mean is that Microsoft buying HERE doesn't mean that they (the car manufacturers) are going to use their app.
  • No of course not... But my point is if MS had bought Here, the in car entertainment system could be another product/service that MS could have offered car manufacturers that 'already' have a pre-existing agreement with them. It's much easier to pitch when you already have a foot in the door. I've no idea why Nokia didn't do this years ago either. I've had free Nav from Nokia in my car since the N95 days... That's a massive headstart they missed the opportunity on.
  • Well, it's all about their core focus with Satya Nadella which is why the AOL deal and Uber deal happened. Ballmer was a diverse guy and he wanted Microsoft to be pretty much everywhere lol. Not that I blame him, just applauding his efforts as Microsoft wouldn't have been where it is today in terms of profits without the Ballmer. Not entirely discounting Bill gates either; who too was a visionary and shaped Microsoft as well as the tech world as we know it today. Of course there are countless other individuals who have had profound effect on technology as a whole, quite a few I imagine are unknown to the world (would say the team behind at the computer at xerox as one example).
  • This would be awesome
  • No thanks.  Keep computers out of cars please. 
  • Go back to 1981 then. 
  • Until car manufacturers can build a car without it having issues 20k miles in, I stand by my statement, and also declare yours as idiotic.
  • So basically you're going to stomp your feet and start name calling without any real base for it. Computers have been pretty wide spread in cars since the 80's, hence the go back to 1981. If I really wanted to get technical, I think it was VW that was the first to put a computer in their car in the mid 70's. So my comment was both relevant and factual, where as yours is just insulting and useless. As far as your 20K comment, my current vehicle with 27K on it, and my last vehicle which I traded in with 147K on it, easily made it past your 20K requirement. So I guess on both counts you have to not stand by your statement.
  • It was you who started with the insults.  And by computers, I mean like the ones mentioned in the article, hence why I'm commenting on this article.  Amazing that I have to explain these things to you.  
  • Really, because I have yet to insult you. Also, I hate to break it to you, but your ECM has far more control over whether or not your vehicle ends up in the shop than your infotainment system does. Not to mention, the failure rate of infotainment systems in the first 20k is far less than other main components of the vehicle that have zero to do with anything computer related.
  • All you have to do is go back to a recent statement made by general motors to the effect that their goal is to make vehicles with more technology and a shorter service life. So yes, computers in cars are to blame for why vehicles have to be replaced sooner. It may not be because they break, but with so much technology, they go obsolete almost instantly. And that is why I'm not a fan of all this tech in my car, and why I drive a 2001 Honda that I can fix myself instead of a newer car.
  • Ooh Minority report style displays? Count me in as a buyer!  
  • How about working Cortana out to all of us who STILL haven't got access to the service...
  • Microsoft, late to the party again. They had a head start with the Ford sync platform, and just like mobile a few years ago, it was neglected and left to wither on the vine, while Apple and Google swoop in, eat their lunch, and are planning for autonomous cars fully powered by Google or auto hardware by Apple. Google and Apple are smart enough to know that cars *are* the next big market for mobile, where you literally have a captive audience inside the cabin.   What's more mobile than a car?
  • Microsoft, Don't forget Groove Music in that device!
  • Great. Now if they make it so that I can change to this on any Android Auto, I'll be set. Then, maybe car makers can settle on standard ARM CPU and touch and doing an aftermarket audio system will be just a software upgrade with a USB stick.
  • Microsoft's vision is too limited in auto. Google  is testing fully autonomous cars powered by their software, while Apple is looking into building entire cars, like Tesla,  while Redmond fiddles with whatever.
  • And that will be the demise of both Apple and Google...both companies' stocks have been hit because they are hitting a brick wall...software companies making cars is bizarre and a very high-risk, high-leverage business...just ask FCA (Chrysler)...
  • Apple is a hardware company, and with their $200 Billion cash pile, they could swallow Tesla easily without breaking a sweat.
  • Yeah but can they make a good car? and when they make that good car can we purchase it? I see plenty of Tesla on the road, nice cars for certain. I don't know a single person that owns one, and that's what I'm talking about. If Apple's car competes with Tesla and forces the S to be priced under $50K, I'm down for an Apple car. but I'm not paying $1500 a month car payment plus insurance. At that price, my 300C is cheap to drive by comparison.
  • Surely continuum should make such systems redundant?
    A Miracast capable screen connecting to a phone that automatically applies a carmode overlay...that's my kind of in car system!
  • So much this... This is a huge part of my reasoning as to why W10M (in phones specifically) should have a full landscape mode.
  • Add to pioneer head units please
  • I would LOVE this in my car! I use cortana all the time, and don't trust any other AI assistant at all, I would throw my wallet at this so hard if I could buy a head unit that had Cortana built in and the UI was similar to my Phone, tablet, laptop, xbox!
  • Guys. Stop it with the "rocking" in articles. It's embarrassing.
  • Since the announcement of HoloLens, I've thought Windows Holographic could eventually be great for use in a HUD in a vehicle. Not only for displaying info like this, but even for crazier things, like GPS info, markers for gas stations and restaurants, projections where other vehicles are thing to cause you problems... Who knows what else. Imagine the possibilities of AR in your car.
  • Infrared HUD on the windscreen, i.e. Night vision would be awesome!
  • Windows platform for the car will definitely go with regional restrictions so why bother.
  • This will be interesting if they can connect to ios, android and wp.
  • They may need to lawyer up and force their way in just like all the other web services did with Windows.
  • They need to hurry up!!!
  • Yea it would be cool to see it after Apple and Google already own that market, same catching up drama again
  • I wanted to hold out for this. But, I'm going to get a Pioneer NEX and deal with Android/iOs Auto until this gets fleshed out.
  • This i like!
  • Considering Microsoft's strategic approach to productivity, I think a Windows 10 IoT car would work better with the contending ecosystems than Android Auto and of course Apple Car Play... I just hope we won't be forced to change our car or smartphone to get both working well together!
  • Well MS had better hurry the hell up with this and get some exclusive deals with auto manufacturers before Apple and Google snap up all the auto manufacturers.
  • Just that screen shot used is a lot more clean and neat than the crap on my ATS, or pretty much anything else out there.  Enough Chrome, just make these car UI fast and lean with minor customizability if it applies! God, and Cortana would be a "God send" given how crappy voice recognition is on Cadillac CUE.  My old Honda Accord didn't fair much better, either.  I mean seriously, how do you get "call Angelica Eisner" from "play album Jealous?" Cortana's voice recognition is soo much better and she's snappy.
  • I didn't watch the original BUILD 2014 coverage of Windows in car... but, I'm glad you linked to it... I love what I see.  Those big tiles, and sleek interface are perfect for driving.