Microsoft Research shows potential for combined thumb and pen interactions on Windows tablets

By published

One of the major highlights of Windows 10 on tablets, particularly with Surface, is its pen and inking capabilities. A new video demonstration from Microsoft's Research division shows how that pen and touch input could be enhanced in the future with new interactions that open up all kinds of productivity possibilities in everything from spreadsheets to your browser.

One of the best examples of simultaneous pen and thumb input shown in the video is its first: Excel. Using a sort of radial menu under the thumb in combination with the pen allows for quick copying and pasting of cells, formatting, and navigation. Though it would definitely take some adjustment, proficient use of the simultaneous input techniques could definitely speed up common tasks in Excel.

The rest of the demonstration shows off some other interesting ideas ranging from manipulating layer controls while moving objects in a drawing app, to quickly opening hyperlinks in separate browser tabs with a thumb press and pen tap.

These are all obviously experiments at this point, but they provide a peek into some of the cooler interactions we could see applied in the future. What types of pen and touch input interaction would you like to see for tablets?

Thanks, @farshadatis, for the tip!

Dan Thorp-Lancaster
Dan Thorp-Lancaster

Dan Thorp-Lancaster is the Editor in Chief for Windows Central. He began working with Windows Central as a news writer in 2014 and is obsessed with tech of all sorts. You can follow Dan on Twitter @DthorpL and Instagram @heyitsdtl. Got a hot tip? Send it to daniel.thorp-lancaster@futurenet.com.
62 Comments
  • This is incredible, I need this in my life.
  • Tablet Pro on the Windows Store is the closest materialization we have so far...
  • I'd be surprised if they implement this.
  • Pleasantly I hope
  • This should be stated before every MS Research video ...
  • Pretty neat stuff. Although, I always find it odd whenever Microsoft releases videos like this and the PCs in use have Chrome pinned to the task bar. Why give their main competitor free advertising like that?
  • Because they probably don't care that much.
  • Because they miss the days when google just wrote stuff to run on Windows. Lol
  • The people that don't know what chrome is wont know just based on the icon so it's no help to them. And the people that already do use or know about chrome wont care. It's not an issue in a research demonstration.
  • The correct answer is because it's easiest to develop with Chrome. I know because I'm a developer.
  • I love the Excel demonstration. I use Excel a ton on my S3 and those would be helpful.
  • This looks like an ergonomics disaster in the making for southpaws like me.
  • I would assume that there might be a feature to flip it for us lefties.
  • That should be that case, but there's a big chance that often they will forget us, design an intricate GUI based on use of the left thumb and then last minute remember that there should be a left-handed version too and cook up some half baked left-handed version. When it comes to controlling touchscreens with my fingers I am pretty ambidextrous, but I really can't use a pen with my right hand, so the combination pen + thumb could turn out pretty bad.
  • Microsoft worked on making the pen work with both hands since the beginning. The very first Tablet PC already had a left/right-handed configuration so that menus and other UI appear on the proper side of the pen after pointing.
    I don't think they would overlook that when bringing new pen+touch interactions.
    And remember Bill Gates is left-handed, he had decades to hammer into every team that it matters.
  • Hopefully they can find a way to fix that for you... your left-handism I mean. :)
  • "southpaws" ? What is that?...💁
  • "Southpaw" is slang for a left-handed person. Possibly the usage is predominantly in the United States.
  • Southpaw is not slang. It is a standard English term especially in boxing. We do hope though that MS keeps southpaws in mind too when and if they implement this.
  • Southpaw is commonly known. In fighting, mostly comes from boxing. With a stance you angle your body with one side put slight further then the other towards your opponent. Most people are right handed and so their right arm is the stronger arm. So, they have their left arm be more towards your opponent and your left arm is your jabbing arm while your right arm is then used for power. Southpaw is then when someone leads with their right side and their right arm is then the jabbing arm.
  • My thought exactly.
    Microsoft should allow placing the thumb widget on the right side as well.
  • That would be very helpful in Excel. It is an absolute pain trying to select menu items with the pen.
  • Yes please!
  • I use an application called Tablet Pro to do something similar (but way way way less cool than this). I use it to add simple on-screen buttons for illustrations when working with Affinity programs (and if I was to own adobe products same thing). This looks super cool, with a little polish could be awesome.
  • WOW!
  • Amazing, add this to Photoshop and Illustrator like a virtual Dial and productivity for designers would explode.
  • Check out Tablet Pro on the Windows Store...
  • But seriously, if Microsoft continues to move down paths such as this; they really will have a platform that distinguishes itself based on it's own merits. I'm very excited by this concept.
  • I like that thinking. They've really worked hard over the years to bring new things to life . The problems have been unclear messaging/marketing and lack of a cohesive integrated strategy across product lines making the whole greater than the sum of the parts.
  • Yeah which is rather unfortunate. They seriously have brilliant people in the company, especially in MS Research and other departments, but it all hinders with their internal communication and lack of cohesive mentality that some of great ideas may not or will be very slowly gets implemented. The new MS though is better and things like this may be faster down to the pipeline. As always likely internal corporate politics especially of how huge the structure of MS will be a problem. But yeah, this research demo is seriously must be implemented sooner than later on Windows 10. Improvements to pen + touch interaction is crucial to make pen input become common use for tablets, and became a standard. IMO tablets without pen isn't really much useful and doesn't stand-out from phablets.
  • I want this in Redstone 3!
  • Bring 'em here!
  • Apple Pencil has this ability already. Simplest demonstration was their Notes app. You can move the ruler and pen at the same time. This was something I was waiting for from Microsoft for forever. Been a TabletPC user for a long time, then Apple shows off the iPad Pro, I jumped shipped because their first iteration did everything that Microsoft has been envisioning for more than a decade, and Apple hits it right away, tilt, pressure, finger and pen input. >.> Curse you Microsoft.
  • This is how surface dial was created. 
  • Except this doesn't require any separate device to be worrying about...much better than the Surface dial on devices smaller than the Surface Studio.
  • Looks really intuitive. Hopefully it will be implemented soon.
  • This is amazing and needs to be implemented ASAP to empower the Windows 10 platform.
  • this looks amazing! i love the excel example! the potential is endless and it looks so intuitive! cant wait to seeing more
  • Like the idea a lot, but this seems very complicated to master! I also wonder how they'll handle palm rejection.
  • Awesome. A portable surface dial in a sense, but has huge benefit for mobile computing. Hope they get this out for RS3.
  • Amazing!
  • At least one of the researchers was on the InkSeine team back in 2008! InkSeine had sooo much potential, I was praying they'd integrate the tech into OneNote. Anyway, here's a link to the old InkSeine page and video.
    https://www.windowscentral.com/e?link=https2F%2Fclick.linksynergy.co...
  • I remember using the radial menu with pen on OneNote. Anything like that would be amazing to have again
  • Neat!!!
  • Now release it after all the competition releases it !
  • Wow! My Surface Pro 2 is gonna be getting a whole lot more useful with these amazing shortcuts integrated with Windows 10. You simply can't get this great use case on merely a Laptop.
  • Do we have a patent for this or something. Apple WonderTouch might not be too far, otherwise.
  • Even if they patent it (which they definitely should), wouldn't stop that cheap rotten fruit from shamelessly infringing on it and crying "the license fee is too high"...
  • https://www.youtube.com/watch?v=6lWe9PvabAo
    this one is awesome too
  • Stunning
  • Would be awesome that when the pen is active,  and the tablet recognizes it,  your thumb turns into a dial type application.
  • It will not work! On Surface Laptop :p
  • Why?
  • Wish to see this feature implement.
  • Someone please get Satiya to authorize this for production.
  • I've been saying ever since I saw the patent for a flip out phone from MS that the Surface Phone could have a quick access finger touch front screen. And the bigger inside screen for use with a pen the size a Nintendo 3ds has. To me its a no brainer. As the DS is so so accurate on the small screen it has.
  • This is a solid feature. I would find this extremely useful.
  • In other words the Surface Dial just bit the dust - yet another product I have purchased from Microsoft that wasn't worth the casing it came in (maybe 8 apps total support it). I guess I will put the Dial next to my 950 which has gone into continuous reset mode (I wish there was an app for that), which sits next to my Microsoft Band 2 with the cracked band (happened twice, first under warranty, then again a few weeks after the warranty expired). The Band 2 sits next to my Kinect sensor for the Xbox one which was promised to be more useful than my Kinect Sensor from the Xbox 360, neither of which are supported by anyone today, including Microsoft (canceling the workout site was simply inexcusable). I also have a Lumia 920 and a 1520 in that box, though I must say the 920 was a nice piece of hardware (thanks Nokia). I am hoping my SurfaceBook Pro with the Performance base and my Surface Studio that I am using to write this don't end up in the same box any time soon, but given recent history, the outlook is not promising. I hate being a Microfan because I know I will be the first dope in line to buy that phantom surface non-phone phone when it arrives - Arghhh!
  • I feel your pain!!! No I don't have one, But the dial is very new... And when it gets better discovered, it will stay. Wish I could say the same thing about the Band....
  • Looks great; sooner rather than later, please!
  • This is great innovating MS!!!
  • Anyone remember their patent for a scroll control interface on the back of devices?