AI, Bots and Canvases Part III: Gates and Ballmer paved the path for Nadella's AI and bots

For those who have just "tuned into" the tech scene, the bot and AI future we're now entering may appear to be a new revolution fueled by the "me too" efforts of established companies and start-ups alike. This assumption would be an error. The industry has been moving toward this goal for decades. In fact, the technology has only recently caught up with the tech dreams of those who came before us.

Investments in machine learning, natural language processing and deep neural networks have finally begun yielding the fruit where artificial intelligent sidekicks that know us and can act proactively on our behalf seems well within reach. Though many companies have invested in, advocated for and applied these technologies in their products through the years, Microsoft has been one of the earliest and most deeply invested leaders in the industry.

A vision forward

Cortana is now extensible to what Microsoft refers to as "experts", or bots that developers build to engage users as proactive, intelligent apps. As part of Microsoft's Conversations as a Canvas strategy, Cortana is, in essence, a cross-platform "platform" that can benefit from the large Android and iOS install base.

Cortana, Microsoft' evolving cloud-based, device agnostic, intelligent agent has long been in the making. And even now we are only glimpsing the coalescing of the fundamental variables that Redmond is working to materialize into a ubiquitous intelligence that unifies the entirety of a user's digital experiences. These efforts began under the tenure of Microsoft's first and second CEO's Bill Gates and Steve Ballmer respectively.

As Moses led the children of Israel out of Egypt but not into the Promised Land, Bill Gates and Steve Ballmer heralded the investments that have brought Microsoft to the border of a new age in personal computing. As Joshua inherited the charge of his predecessor and led Israel into Canaan, Redmond's current chief executive, Satya Nadella, is leading Microsoft into the age of AI and bots.

Though Nadella is at the helm during this pivotal transition, we mustn't forget those who brought the company to this point.

Bill opens the gates

Microsoft's first CEO Bill Gates was a visionary. His dream to put a computer on every desk and in every home was ultimately a means to bring Microsoft's personal computing environment to the masses. And it was largely achieved.

The present age of personal computing that is not bound to the desk required an evolution of the strategy to bring Microsoft's brand of personal computing to the masses. A personal computing environment that has the benefit of a consistently connected internet, is not bound to a single device and has access to a vast repository of information required an intelligent UI to help users manage digital experiences in such a dynamic environment. This advancement is a tremendous evolution to the ill-fated Microsoft Bob which was introduced over 20 years ago.

In the video below Gates expounds on the utility of Microsoft Bob but also foreshadows a more advanced AI to come.

Years later Gates expressed a vision of an intelligent agent that would be personal, present and deeply integrated into a user's life. It would be wholly consistent with the modern and mobile state of personal computing:

As everyone gets essentially what we'd call the personal agent—it's been talked about for decades and now really is possible—we see where you're going, we see your calendar, we see your various communications, some of those communications we can actually look at the tags, look at the speech, try to be helpful to you in your activities…I think that we will be more connected, so that when somebody wants to find a gift of a certain type, or take a trip in a certain way, that there will be a closer match."

Though many Windows Phone fans, myself included, initially hoped that Cortana would never go cross-platform, her venturing to iOS and Android coupled with her deeper integration within Microsoft's other services are key to Microsoft's vision for the AI. Microsoft does not view Cortana as a Microsoft-ecosystem-focused AI, but more of a boundless platform for intelligent interaction with all digital experiences.

Ballmer kept the ball rolling

Steve Ballmer succeeded Bill Gates as CEO of Microsoft in 2000. Though Ballmer's leadership style differed from that of his predecessor, the goal of bringing an ever-present intelligent agent to the masses remained.

Steve Ballmer

Steve Ballmer (Image credit: Windows Central)

During his tenure investments in the foundational technologies required to make the dream a reality continued. Steve Ballmer articulated the following description of his and Microsoft's AI vision in a July 11th, 2013 memo (opens in new tab):

"Our machine learning infrastructure will understand people's needs and what is available in the world, and will provide information and assistance,"…"We will be great at anticipating needs in people's daily routines and providing insight and assistance when they need it. When it comes to life's most important tasks and events, we will pay extra attention. The research done, the data collected and analyzed, the meetings and discussions had, and the money spent are all amplified for people during life's big moments."We will provide the tools people need to capture their own data and organize and analyze it in conjunction with the massive amount of data available over the Web."Our shell will natively support all of our essential services, and will be great at responding seamlessly to what people ask for, and even anticipating what they need before they ask for it."

Ballmer expanded on the all-encompassing nature of a ubiquitous cloud intelligence by sharing Microsoft's vision of a family of first-party devices that this intelligence and the Windows platform would unify. We saw the fruit of this vision two years later on October, 6 2015 during Microsoft's Windows 10 Devices Event.

A few years ago in a speech I gave at CES, I observed that there was a shift underway. We were headed from a phone, a PC and a TV to simply three screens and a cloud — and over time, a common software-based intelligence would drive all of these devices, bringing them together into one experience for the consumer.""As devices proliferate, it has become clearer that consumers crave one experience across all of their technology…Going forward, our strategy will focus on creating a family of devices and services for individuals and businesses that empower people around the globe at home, at work and on the go, for the activities they value most.To take advantage of our critical competitive assets, we will center our work on…A business model based on partner and first-party devices with both consumer and enterprise services… A family of devices powered by a service-enabled shell… No technology company has as yet delivered a definitive family of devices useful all day for work and for play, connected with every bit of a person's information available through one cloud.Our devices must share a common user-interface approach tailored to each hardware form factor.

Reflecting on Ballmer's vision

Reflected in Ballmer's statements is Microsoft's commitment to the duo user by providing a portfolio of partner and first-party devices which are useful all day for work or play. These devices would be connected by an intelligent cloud and common UI. Ballmer's reference to the company's intelligent agent indicated that it would be user-focused and would know and support the user in both professional and personal settings.

Ballmer predicted this ubiquitous intelligence would unify a family of first-party devices.

This duo user focus is an important point when we consider that computing has moved to the cloud, and that Redmond's hardware is purposefully context sensitive easily transitioning between professional and personal productivity. Microsoft envisions a bot-connected Cortana that effortlessly flows with a user across services, and devices growing through the rich content of user's experiences and data gleaned from the system. The systems intelligence would mature and become more proficient at anticipating and serving a user's needs.

Naturally, the success of a company's endeavors doesn't rest solely in the plans of its leaders. Scores of talented individuals are required to bring those dreams to fruition. Time and space will not allow us to acknowledge all who have contributed and are contributing to bringing Microsoft's AI vision to fruition. However, we've "heard" the broad overarching views of previous leaders. Let's now listen to the voices of some of those who have helped and are helping that vision to materialize.

Larry Heck talks Microsoft's AI tech

Larry Heck is currently a Principal Scientist with Google Research. Before joining the Mountain View company, as a Microsoft employee, Heck was credited with starting the conversational-understanding (CU) personal-assistant effort at Microsoft in 2009.

Prior to his current post he worked on a technology vision for virtual personal digital assistants for the Bing research and development team. At the request of Zig Serafin, who was appointed by Ballmer to unify Microsoft's speech efforts across the company, Heck joined his team as the chief scientist. Through this position, Heck and the team began building the plan that later evolved into what (or who) we know as Cortana.

Heck had this to say in an April 2014 interview (opens in new tab) about the current state of digital assistants:

I believe the personal-assistant technology that's out there right now is comparable to the early days of the sense that we still need to grow the breadth of domains that digital personal assistants can cover. In the mid-'90s, before search, there was the Yahoo! directory. It organized information, it was popular, but as the web grew, the directory model became unwieldy. That's where search came in, and now you can search for anything that's on the web.""Current implementations target the most common functions, such as reminders and calendars, but as technology matures, the personal assistant has to extend to other domains so that users can get any information and conduct any transaction anytime and anywhere."…"Having a long-term vision means we have a long-term architecture. The goal is to support all types of human interaction-whether it's speech, text, or gestures-across domains of information and function and make it as easy as a natural conversation.

It is interesting to see how two years after these statements we see how Cortana can be interacted with via Microsoft's new inking platform in addition to the standard speech and text interactions that are available.

AI's basic fundamentals according to Heck

"In fact, we've pioneered efforts in each of those areas."

According to Heck, "The base technologies for a virtual personal assistant include speech recognition, semantic/natural language processing, dialogue modeling between human and machines, and spoken-language generation" and "each area has in it a number of research problems that Microsoft Research has addressed over the years. In fact, we've pioneered efforts in each of those areas", he says.

Heck continues with emphasizing the long history Microsoft has in these critical areas of natural language processing, machine learning, deep learning and deep neural networks:

The underlying vision for this work and where it can go was derived from Eric Horvitz's work on conversational interactions and understanding, which go as far back as the early '90s. Speech and natural language processing are research areas of long standing, and so is machine learning. Plus, Microsoft Research is a leader in deep-learning and deep-neural-network research.

Microsoft's years of investment in this increasingly competitive space has helped prepare the company for the challenges it will face in its quest to emerge as the leader in AI and bots.

Mike Calcagno and Larry Heck talk Microsoft's AI advantage

Mike Calcagno, Director of Engineering for Bing experiences joined Larry Heck in an interview where we, through the conversation get a clear picture of the strategic advantage Microsoft brings to AI and bots. As a result of its various tools and services such as Bing and Skype, Microsoft has a wide net through which the company can feed data back into the system to continually improve upon it. Heck explains:

I think a lot of advancements in speech recognition is dependent on the data and how much data you have that's fed back into the system and the quality of data you have fed back. And the amount data we're getting right now that we're pulling back from particularly Bing voice search is tremendous. And we're extending speech recognition capabilities also into Skype, the Skype translator, for speech to speech translation so that there's speech recognition that occurs when the person is talking in English and translation and then synthesis into say German. Well, all of that data on the translator side, the voice search side and Cortana side and all of the other speech applications feeds back into a single recognizer we use across the board for the whole company.

So I think one of the things that is an advantage for us is that we have so many different scenarios, and products that are out there that feed back into this speech recognition technology.

These varied points of data input provide Microsoft with a unique resource to help evolve its AI. Calcagno also stressed that Microsoft recognizes the usefulness of both Siri and Google Now but Microsoft's goal was not to make Cortana a "knock off" either of its rivals.

Long road ahead

Microsoft has a long history in machine learning, natural language processing, and deep neural networks. This history coupled with widely used cross-platform tools and services which consistently glean human language data from users and reintroduces it into the system positions Microsoft efforts at the front of the AI race. Redmond's Bot Framework and Conversation Canvases also positions Microsoft's ecosystem as the developer's platform or "devbox" for intelligent apps or bots.

Finally, Cortana, unlike her competitors, as a Conversation Canvas agent, is being positioned as an unbounded intelligent platform to complete computing functions independent of a user's device or operating system. She is slowly evolving into Microsoft's vision of an intelligent cloud-based UI that will know users, and as a meta-app interacting with bots, anticipate and facilitate a user's needs.

We are at the beginning of a long and competitive road.

Admittedly we are at the very early stages of this journey. As a parent joyfully (and fearfully) celebrates the transition of their child from infancy to the first day of school, we are celebrating the transitioning of Microsoft's AI and machine learning efforts into the mainstream. Still, as with a child in kindergarten, there remains a very long road ahead.

So as we watch Cortana's evolution, take the time now and then to revisit these visionary statements from those who came before. Yes, there is competition from the likes of Google, Viv and even Apple within its closed ecosystem. But taking a look back every so often will help us see just how this ambitious effort, which began many years ago, is materializing within an increasingly competitive space.

See more

So what are your thoughts? Is Microsoft on course with its AI vision which it set forth years ago? Does competition from rivals require a shift in plans? Is Microsoft's industry-wide platform play too broad? Does it dampen the uniqueness of Microsoft's ecosystem? Sound off in comments and on Twitter!

If you missed parts I or II or these related pieces check them out here!

Jason Ward

Jason L Ward is a columnist at Windows Central. He provides unique big picture analysis of the complex world of Microsoft. Jason takes the small clues and gives you an insightful big picture perspective through storytelling that you won't find *anywhere* else. Seriously, this dude thinks outside the box. Follow him on Twitter at @JLTechWord. He's doing the "write" thing!

  • Thanks for reading folks! As we can see Nadella is carrying the baton that was passed to him from his predecessors. As we see the world of an intelligent agent that anticipates and knows our needs was well articulated by both Ballmer and Gates. I know that we are a long way from the fulfillment of the vision, but what helps keep the big picture in view is looking back to where things began, the visions that were stated in the past, the explanation of the technology as with Heck and Calcagno, that is bringing it all to pass, and looking at the present, how far we have come. I know that there will likely be a lot of cynicism, about "not being" where the vision projects Microsoft and the industry may be in time. Nut that's not the point.:-) Of course we're not there. We're simply looking, in this series, at the vision, and in this piece, some of the history that got Microsoft (and the industry) here, within the context of where the firm hopes to go. Well, you know the drill....LETS TALK!!! :-)
  • Key word at the end TALK. Haha...your editorials are great!.... but always seem to bring out the trolls, torches and pitchforks. Great piece once again, sir!
  • Thanks for the article, well-written indeed.
  • I feel like I'm being indoctrinated.
  • Ahhhh... My favorite editorialist.
  • As bad a ceo that ballmer was in my opinion, he did kinda lay the ground work for the whole universal platform, which i, at the moment, am really impressed with, as a consumer, and as a small time ( like really small) business owner. I really am looking forward to the whole universal platform with Cortana and the bots.
  • LOL...LOL
  • That logo is still better and timeless.
    Unlike the square window
  • One Windows Platform ignores the biggest screen of them all.... the TV
  • It doesnt, you have window IOT.
  • That's not TV.
    Atleast not the Media Center way
  • Well you have xbox for that.
  • I don't need xbox...I have xbox games on phone, pc, tablet whatever.
  • Well than your regular TVs are not powerful enough(computing power)to give you a full media centre experience and that why I said window IOT
  • I have Windows 10 and Cortana on my TV. Just plug in an Xbox One.  
  • Windows 10 is useless connected to the main TV.
    I have 8.1 for that.
    That's why I say UWP should've taken Media Center into account.
    As it is, its not universal enough
  • Bots will more n more be deployed in enterprise but in consumer space bots will not replace apps. simply because apps generate revenue via ads, same is not feasible via bots
  • Not all companies push ads in Apps. Uber, for example, does not. This is also a prime example of an app which could be streamlined by a bot. Same could be said of travel apps, like United, which also do not serve up ads inside their app. If I could use a bot to check-in, and get the QR code for a flight, I'd be more than willing to delete the United app.  What you are over looking with your assertion, is that not all apps make revenue in the form of ads. This means their regular sales must offset the cost of developing and maintaining every app on every platform, plus web services. However, they could cut costs, and offer a more unified expereince to all their users, by developing one or two bots, which integrate into the majority of supported platforms (some of the Bot platforms, like Skype, support bots designed for other platforms). Bots, like everything else, are not a "one size fits all" solution, but there are plenty of companies which benefit from the use of Bots. They wont replace every app under the sun, but a decent number could be repalced.
  • Yes you are right, I dont deny this, but then my point was to counter the statement based on Jason's previous articles which states apps are dead, smartphones are dead.
  • Hey techiez thanks for participating but I never said apps are dead. I wrote two concluding pieces of the untold app gap series showing the path to bots. But never said apps would completely fade off the seen. The link above will bring you back to that series. :-) Also the smartphone is dead series was more a analysis of the evolution of the smartphone (which was originally seen as a phone with the primary function of voice communication with ADDITIONAL features.) The statement that the "smartphone is dead" was a statement that the capabilities of the device has evolved beyond that early use. As I note in the series, the "phone" is essentially used as a mini tablet first and voice calls, ( the phone function) actually ranks six on one study. :-) Smartphones are dead: Thanks for participating!!!
  • All of these years and still limited to a select few countries
  • -site posted as comment, rather than as a reply to another comment-
  • Yes he didnt do alot as CEO. I wish Gates or Balmer return one day.
  • Satya has done a decent bit as CEO. Bare in mind, though, that it takes a few years before a new CEO's plans really start to be visible. A new CEO isn't going to release entirely new products or services, which had nothing to do with the previous CEO, in their first 2 - 3 years. It's unrealistic to expect Satya to start the same number of new project in 2-3 years, to match what Balmer did in ~14 years, or what Gates did in ~20 years. Honestly, I wouldn't want Gates or Balmer to return as CEO, or have "control", again. Both Balmer & Gates had their ups & downs. Bare in mind that they were not perfect. They made mistakes, too. Part of the problem during Ballmer's era, was the fact that Gates was still partially in charge, and both wanted to go in a different direction. From my perspective, when a company starts to struggle, like Microsoft was, it's good to shake things up, and shift leadership around. New leaders will tackle problems in a different way. A good example of this is the shift in leadership for Xbox & Windows. Phill & Terry are doing a pretty terrific job, from my perspective. They are both really pushing the boundaries and utilizing the feedback they recieve from end-consumers (sure, not every piece of feedback is used, but it's a huge change from the previous regime).  That being said, all 3 CEO's have challenged the status quo, and pushed Microsoft in new directions. It will be interesting to see just how far Microsoft will go in the next 10 - 20 years.
  • I honestly think we are at a resurgence of MS dominance, I think the app warehouse model is in its wanning years as bots and ai give rise to the next wave a of mobile experiences. Apps will remain, but will become less and less important, at least service base apps. I think MS Dominance in Enterprise will lend it an upper hand in consumer and mobile. I think WM10 way forward is though make it the premiere mobile platform for these AI/Bot interactions and showcase to shine. Windows mobile will be the platform to allow these experience to be seamless as iOS and Android wont have the os level integrations as tight. I think we will be looking at a different landscape in mobile in 2-3 years. W10M starts to pay off next year I believe, that's when it start to make sense if it can grab hold of Enterprise than it has chance of relevancy. I'm not saying it will beat iOS and android but it will carve out a respectable and noteworthy userbase if Enterprise embrace it.
  • Windows Mobile will not be the showcase for Microsoft's bots. They will be showing them on Android or iOS, otherwise no one will see, let alone use them. Microsoft isn't the only one moving towards this possible future, they cannot make a mistake like making bots exclusive to Windows. The experience needs to be fantastic everywhere. Posted via the Windows Central App for Android
  • true story, one day cortana will be as advanced as Clippy was
  • Blah blah blah yeah whatever. Cortana is crap on W10! Especially mobile. At least on 8.1 all the card notifications worked and we had a proper dark theme. Now it's just plain crap. I get reminders but nothing else works. Severe weather and sports notifications were great when they worked. #MSFAIL
  • I live in a small Southern African country. No Cortana, no Microsoft store, no Bing rewards, no Groove Music pass, no local apps on WM10.
    You're here talking about a mystical future when Microsoft can't even meet half It's commitments in the present?
    I'm done son. Posted from my Note5
  • I feel your pain, here in Aus Cortana is a shadow of what she was on 8.1 and MS supports little else locally.
    The big difference is, Google and apple have a global plan, MS only has a plan for the USA.
    Ultimately this could be their downfall.
  • It's truly a joke that literally 3/4 of the world doesn't have access to a lot of the features MS advertises. Just curious, if you don't mind, which country precisely? Posted via the Windows Central App for Symbian
  • I honestly feel that Microsoft's goal is amazing and that they should continue to pursue it at all costs. However I feel that their current approach isn't working. They should focus on JUST Cortana. Putting it in apps, expanding the UWP, and enabling users and developers to do more with their machine through Cortana. Creating a separate system for Skype and another one for Bing and Xbox goes against their idea of a one-cloud system.
  • I honestly feel that Microsoft's goal is amazing and that they should continue to pursue it at all costs. However I feel that their current approach isn't working. They should focus on JUST Cortana. Putting it in apps, expanding the UWP, and enabling users and developers to do more with their machine through Cortana. Creating a separate system for Skype and another one for Bing and Xbox goes against their idea of a one-cloud system.
  • And here is why Cortana is on Android and iOs. In machine learning more data = better models. 
      a lot of advancements in speech recognition is dependent on the data and how much data you have that's fed back into the system and the quality of data you have fed back
  • I'm intrigued why Larry Heck joined Google.