RSSTop 10 Windows 7 Features #5: Multitouch
By Scott M. Fulton, III | Published May 15, 2009, 11:27 AM
For close to two decades now, the design of applications has changed surprisingly very little. At their core, apps wait for users to generate input, and they respond -- a server/client model of processing on a very local scale. So in a very real way, what applications do has been a function of how they respond -- the whole graphical environment thingie you've read about has really been a sophisticated way to break down signals the user gives into tokens the application can readily process.
The big roadblock that has suspended the evolution of applications from where they are now, to systems that can respond to such things as voice and language -- sophisticated processes that analyze input before responding to it -- is the token-oriented nature of their current fundamental design. At the core of most typical Windows applications, you'll find a kind of switchboard that's constantly looking for the kinds of simple input signals that it already recognizes -- clicking on this button, pulling down this menu command, clicking on the Exit box -- and forwarding the token for that signal to the appropriate routine or method. Grafting natural-language input onto these typical Windows apps would require a very sophisticated parser whose products would be nothing more than substitutes for the mouse, and probably not very sufficient substitutes at that.
If we're ever to move toward an analytical model of user input, we have to introduce some sophisticated processes in-between -- we have to start endowing apps with the capability to ask the question, "What does the user mean?" And while this edition of Top 10 isn't really about natural language processing at all, it's about Microsoft's first genuine steps in the direction of evolving applications in the direction they need to go, to be able to absorb natural language as the next big step.
That first step, perhaps unexpectedly, is multitouch. It's a somewhat simpler version of solving the bigger problem of ascertaining meaning through input, because now that Windows apps will begin being able to process input coming from two or more points on the screen at once, the relationships between those multiple points will need to be analyzed. For example, when a program receives a series of signals that appear to show multiple points, arranged in a generally vertical pattern, moving slowly and then very quickly to the right...could that mean the user wants to "wipe the slate clean?" "Start over?" "Clear screen?" What's the probability of this?
Analyzing input changes everything, and if enough developers invest their true talents in addressing this necessary element of evolution, this could change everything for Windows apps -- everything.
Where we are now on the evolutionary scale is not too far ahead of where we started roundabout 1982. The creation of Common User Access (the use of graphical menus and dialog boxes that follow a standard format) led to the development of a kind of "switchboard" model for processing input. And if you follow the model of Windows programming espoused ever since the days of Charles Petzold's first edition of Programming Windows, that switchboard is the part you build first -- if you're a developer, everything you make your program do follows from what you put in its menu.
Veteran Windows programmers going back to straight C know this switchboard as the WndProc() procedure; and although languages that have crept into IDEs since the Windows/386 days do use different models and conventions, for the most part, they just provide developers with shortcuts to building this basic procedure. Literally, this procedure looks for the unique ID numbers of recognized input signals, called "window messages" or "mouse events," denoted in the nomenclature by the prefix WM_. A long switch clause goes down a list, checking the most recently received event ID against each possible response, one at a time, and connecting with the procedure associated with that response once there's a match. It's like looking for a plug on a telephone switchboard left-to-right, top-to-bottom, every time there's an incoming call.
Throughout the history of Windows, the evolution of application architecture has typically taken place in either of two ways: Either Microsoft improves some underlying facet of the operating system, which leads to an improvement (or at the very least, an obvious change) in how the user perceives her work immediately; or Microsoft implements a change which developers have to seize upon later in order for users to see the benefits down the road. As you'll certainly recall, the change to the security of the system kernel in Windows Vista was immediate, and its benefits and detriments were felt directly. But when Microsoft began introducing Windows Presentation Foundation (WPF) during the twilight of Windows XP's lifecycle, it took time for new developers to transition away from Foundation Classes (MFC), and many still choose not to.
Multitouch is one of those changes that falls into the latter category. Since December when the company published this white paper, Microsoft has been calling its latest layer of user input Windows Touch. Windows 7 will have it, and presumably at this point, Vista can be retrofitted with it.
Windows Touch is an expansion of WPF to incorporate the ability to do something the company's engineers call coalescing -- ascertaining whether multiple inputs have a collective meaning. Using two fingers to stretch or shrink a photograph is one example, or at least, it can be: In an application where two fingers may be used for a variety of purposes -- twirling, changing perspective, maybe even minimizing the workspace -- an act of coalescing would involve the application's ability to register the multiple inputs, and then ascertain what the user's intent must be based on the geometry that WPF has returned.
The concept of coalescing was introduced to us last October by Reed Townsend, the company's lead multitouch engineer. As he told attendees at the PDC 2008 conference that month, the revised Win32 Application Programming Interface (which will probably still be called that long after the transition to 64-bit is complete) will contain a new "mouse event" called WM_GESTURE, and it will handle part of the job of coalescing motions, in order to return input messages to an application that go just beyond what the mouse pointer can do for itself. Rotation, zooming, and panning, for instance, may require a bit more decision making on the part of Windows Touch -- a kind of on-the-spot forensics that filters out panning motions from scrolling motions, for instance.
Since the first editions of the Tablet SDK were produced during XP's lifecycle, Microsoft has been building up a vocabulary of coalesced gestures that developers may be able to utilize through the assembly of gesture messages. Perhaps a down stroke followed by a right stroke may have some unique meaning to an app; and perhaps the length of that right stroke may impart a deeper meaning.
Simply endowing programs with the capability to judge what the user means based on the input she appears to be giving, changes the architecture of those programs, particularly in Windows. Born from the company's experiments with Surface, a simple manipulable globe application has provoked Microsoft's engineers to think very differently about how input is processed.
The idea, as we saw last October, is simple enough: Place a globe on the screen which the user can twist and turn, but then zoom in to see detail. The project became more detailed once the engineers began to appreciate the complexity of the problems they were tackling. Stretching and shrinking a photograph is a simple enough concept, particularly because a photo is a) two-dimensional, and b) rectangular. But a three-dimensional sphere adds new dimensions, as users are likely to want to use the device spherically. "Down" and "up" become curved, sweeping motions; and zooming in on an area becomes a very complex exercise in transformation. Add two-dimensional floating menus to the equation, and suddenly the application has to analyze whether a sweeping motion means, "Get that menu out of my way," "Pin that menu to that building," or "Make that menu smaller for me." And when a big 3D object such as the Seattle Space Needle comes into view, when the user touches its general vicinity, does he mean to be touching the building itself? Or the globe on which the building is positioned?
In order to make input simpler -- to have a moving globe that folks will already know how to move and zoom into and look closely at and do things automatically with -- the application has to incorporate orders of magnitude more complexity than they've ever worked with before. The moment the program "guesses wrong," and the globe behaves in a way users didn't expect, they'll feel disconnected with it, as though an object in their real world were to wink out of existence or morph into some foreign substance.
Next: The immediate upside of multitouch...
For large screens, IMO, a stylus is better than a finger because it produces no fingerprints and doesn't dirty the screen as much.
Score: 0
|One problem with stylus/fingers are that your wrist/hand generally likes to rest on a surface when you use your fingers to hold a stylus or press on a screen. Without doing this you'll experience fatigue in your hand/arm...
Score: 0
|Some touch surfaces only react to a stylus, though. I had a HP tablet pc, and you could rest your wrist on it and write with the stylus like you were using a pen on a paper pad. All they need to do is encorporate this and they're rocking.
Score: 0
|Been running the betas here and there up to 7077. Windows 7 is nothing. It removes features more than it adds. It Vista Service Pack 2.1. Microsoft had better het its head out of its butt and ribbon-obsession out of its GUIs or Windows 7 will flop, particularly in the enterprise with IT departments, again.
What trash. They caught 90% of the market , then they go and change things.
Score: -3
|...wow.
Score: 0
|To all the haters, I predict there will soon (2015) be a 6"-8" lower-res (but still high enough res) wireless touchscreen that you'll be able to use like today's touchpad, only with two/more fingers simultaneously. This will keep your future very large 30"+ monitor clean, your arms unexercised ;) (it's gonna be very tiring moving all over that space) and when you're done using the touchpad-touchscreen you will quickly clean its small surface (it'll probably have an automatic windshielf-type miniature physical wiper that senses grease) and hang it on the wall or on a stand on your desk in the office ... and it'll double as a digital album/reminder/remote control.
It's very obvious this has to happen...
Score: 0
|Sound like some one needs to patent a built in cleaning stripe for your fingers at the bottom of monitors next.......oh wait. ;)
Score: 0
|I could be wrong and I am sure someone will point it out :) but didn't MS offer a virual Keyboard as long ago as the late 90's early 2000? If this is the case then a form of touchscreen has been around for awhile. From a business standpoint, MS has a computer that is mostly in airports that, I believe operates on touchscreen to find hotels, etc....no big deal.
When you slam me down be gentle, I have a bad back:)
Score: 0
|http://www.hp.com/united...l?jumpid=hpr_R1002_USEN
Multitouch on Vista.
Score: 0
|About the whole touch thing:
Keep your filthy paws off my pristine monitor screen, my nice clean phone display and my crystalline mp3 player display.
Love the concept: seeing my screen through a haze of grease and dirt. Yup yup yup...
Score: 0
|Try soap.
Score: 0
|Yah - on my 24" Dell UltraSharp. Sure. I'd sooner break the digits of the miscreant who touched it. And I'd enjoy the hell out of doing it too. :) :) ;)
Score: 0
|Actually, I was referring to washing one's hands before using; however, it wouldn't be as humorous to say. And you have to be careful how you wash your screen anyways or you might lose parts of it when it gets eaten away.
Score: 0
|Good. I'm glad to know there's someone else out there who wishes death on all screentouchers.
It's like they can't point to something on a screen without having to physically touch it as they're too inept.
I keep screen wipes next to my monitor at work and purposely make a point of wiping the screen down after they've finished touching post gobbling a doughnut. It seems to stave them off for a while.
Score: 0
|I'd really love to know what you guys are doing... The LG Voyager I got several months ago has a touchscreen which is doing just fine (though it's a tad hard to see in the sunlight).
Have you tried washing your hands occasionally? ;-)
Score: 0
|Never gonna work. Any idea how much oil the human body secretes? Formidable.
Score: 0
|I'm picky. Anyone who comes into my office and does the annoying "let me touch your monitor to make a point on what's onscreen" usually leaves minus a limb.
You have no idea how many lawsuits... :) ;) ;)
Next there will be something completely retarded like touch screen TV. Yeah - 50" screen with smudges. Let me get my machette...
Score: 0
|well, if multi touch is one of the added features in win7, whereas those new digital screens allows users to touch, strech and move with the fingers are involved;
it is another feature not needed by mainstram americans or business's.
yes, perhaps all the televised news use them because they are fancy attactions, like on election day tallying of the states won or lost,
but in reality one wonders how durable those touch screens are.
i mean have anyone ever seen a touch screen order tablet like those used by the fast food chains? they are hard to keep clean, finger nails must be trimmed.
if they are larger, like the size of a wall one has to have reasonable mobility and use of their arms to move around the touch screen display.
for most of us a keyboard and a mouse within hands reach is ok.
Score: 0
|I've seen touch screens used on trading floors for a long time, and there are relatively few issues with them, whether they're tablet PCs or monitors attached to PCs. And this was before Windows natively supported it.
Microsoft's move to incorporate it into the OS will only make it perform better.
Score: 1
|You like to be negative, don't you?
Like you say, touch screens have been used for long in many places for quite long times (I was repairing those back in 2005, and of course they weren't new then).
What you seem to ignore is that the technology has evolved a lot, and that there are different types of touch screens, just like they did with PDA's (Remember when you had to use the supplied pencil, because using the finger or another thing didn't do it?).
By the same thinking, you could say that touch phones are absurd, and you need a keyboard.
Scott was pretty clear on his article, and you missed the whole point. Multitouch is not a goal, it's a path. A way to get to something bigger.
I am a coder. I definitely don't think voice recognition or virtual keyboards onscreen could help at all to write code. Yet, i know the advantages that touch screens already provide, and now those possibilities will go beyond replacing the mouse pointer.
Score: 2
|i'm not negative, just a pragmatist.
tiny touch screens and within "hands reach", so your counter argument lacks justification.
on the extreme side using 108 inch touch screen as an example of why i think touch screens are not proficient:
we can be assured that a person using this so-called convenience of touch screen on a 108 inch screen, will likely have to be athletic. touch screens are a novelty, in my opinion.
a little device like a mouse or a laser pointer or our voice to do the work for the human, is the evolution of technology that should parallel human evolution.
speaking or yelling our thoughts "or" pointing with a device that mimics our fingers, is more efficient than moving are shoulders, arms, hands and fingers, bouncing from one end of the screen to the other of a large touch screen is not proficient. it is technology that is novel.
perhaps, "touch lcd screen displays" should be replaced with holographs where the "screens" move within hands reach is the next step.
Score: 0
|Operating devices with our mind (such as via biofeedback) is closer to parallel human evolution.
Score: 0
|It may not necessarily be for mainstream or business, but there are devices like netbooks and console table screens, kiosks, and similar which could certainly use it. Having the capability built-in greatly adds to it's versatility and convenience.
There are also other uses like scientific (rotate dna or chemical models, military, police, weather, hospital, and or special needs purposes). Or you can have special monitors setup to replace traditional mechanical controls. So instead of a room full of 1000 different controls. You could have 3 monitors lined up together to create a virtual control panel.
http://www.youtube.com/watch?v=XNmQXxsUESY
As for smudges, I am sure there will be special "skins" as was previously used on palm devices, phone screens, just a bigger size (i.e., Fellowes WriteRight). It would be slightly textured, to allow movement without smudging.
And don't forget, we are on the verge of "flexible" screens:
http://www.youtube.com/w...F-k&feature=related
What may seem superfluous today, could very well be standard fare tomorrow.
Keep an open mind :)
Score: 2
|You can't see this being useful, you can't see PowerShell being useful, you can't see Homegroup being useful...
All of these are failures of *your* imagination and creativity.
As for TouchScreens...
Tablets have been around for over a decade i their current form. They have been gaining in popularity, albeit slowly. Media Center systems, however, have become *huge*. Both of these would gain a great deal of usability improvement from touchscreens.
Are we going to use them in business? I can't say I really see a use for it, but it could simply be a failure of *my* imagination.
*laughing*
The "car" was ridiculed as a "toy" for the super-rich. I'd say those who thought that simply lacked the proper creativity. As for controlling 108" screens...you're being absurd again. They won't develop 108" "touch" screens for home use because it would be *much* easier to control those with an 11" blue-tooth enabled touchscreen from you lap on the couch.
Score: 0
|@mcox
using our mind may not be a good idea.
minds would have to be in a state of logic.
the best rule for computers to evolve into is:
" do as i say, not as i think"
otherwise, their screens will likely be always cycling with searches on how to kill their spouse or the boss or exotic love making, soft porn and nasty porn.
computers would be constantly crashing due to the illogical minds of the feebled.
perhaps, there are some humans that are evolving. but most assuredly there are millions more that are riding the wave of evolution carrying with them the sub human characteristics and a bible to mask their immoralities.
Score: -1
|@pct
what would you know about "evolution"?
we all have reason to believe that you are still evolving from monkey-ism.
Score: 0
|*laughing*
Sorry, man. For a second there I though you might be capable of reason... Thanks for the reminder.
DBBen: This is useless.
PC_Tool: You're wrong. Here's some examples of how it's already being used and sold and an example of the same mistake made by others regarding a different product.
DBBen: You're a Monkey!
Pretty much exactly what I've come to expect from you. When faced with reasoning contrary to your troll BS, you devolve into a 3rd grader. How cute...
Score: 1
|Hasn't the thing with the top of the screen maximization/restoration been around with Compiz Fusion in Linux, for quite some time now?
Score: 0
|@pct
your perception is distorted, as usual.
while you think you are conveying "laughter",
we all hear "grunting".
you simply can't smell how bad you stink to everyone else.
Score: 0
|Ah...more childish insults (and zero apparent ability to respond with anything else)...
Lord knows you're not good for anything else...you sure as hell can't seem to respond rationally to any contradiction of your absurd generalizations.
But keep up with your never-failing fallback to 3rd-grade "humor". It's sure to at least entertain the other drooling idiots.
Score: 0
|