Multi-touch support in Linux/Xorg/GTK+

So, for the first time ever (to my knowledge), the full multi-touch stack working on Linux:

Wee, multi-touch

This video features:

In my opinion, this is a quite important milestone, which reflects much work done lately in this camp, and from now on things can only get better!

And of course,

28 Responses to “Multi-touch support in Linux/Xorg/GTK+”

  1. Ernst says:

    Isn’t a lot of this patented by Apple? :-(

  2. Simon says:

    Ooh, very nice. Just need some good multi-touch hardware now… hmm, wonder how long it’ll take before someone has Linux running on Apple’s new toy :)

  3. Dmitrij says:

    What’s valen the turtle =P

  4. Rob Sharp says:

    I love it! It makes me want to hack a multitouch panel into my netbook…

    What hardware are you running that on, anyway?

  5. ethana2 says:

    Humans have been using multiple fingers to do things for..

    quite a long time.

    If Apple does indeed hold patents on such basic concepts as multi-finger interaction, then I count myself lucky that the gesture I have for them only really requires one.

  6. ethana2 says:

    If Ubuntu or Android reaches Just Works status on the iPad, I might actually consider getting one.

  7. Tobias says:

    @Ernst “Isn’t a lot of this patented by Apple?”

    That is Apple Marketing, don’t believe everything they tell. If you are interested in the facts, what is patented and what not have a look at this articles:
    http://www.engadget.com/2009/01/28/apple-vs-palm-the-in-depth-analysis/
    http://www.osnews.com/story/20876/Apple_vs_Palm_the_In-Depth_Analysis
    http://www.osnews.com/story/20861/Apple_Granted_Patent_on_Multitouch_Gestures_More

    @Carlos
    Nice work. I can’t wait to see all these thing upstream.

  8. ao2 says:

    Hi, for those who want to experiment with multi-touch and lack specific hardware, you could try with multiple mice, see also here:

    http://ao2.it/en/blog/2010/01/19/poor-mans-multi-touch-using-multiple-mice-xorg

    Not the same thing of a real multi-touch screen, but this is a quick way to test if software has at least basic support for multi-pointer/multi-touch interaction.

    Regards,
    Antonio

  9. carlosg says:

    @Dmitrij: See the photo I’m manipulating in EOG :)

    @Rob Sharp: This is an HP Touchsmart Tx2 with integrated N-Trig touchscreen, a reasonably cheap computer with MT technology, although actually I’m not that impressed by the hardware, hopefully things will be improving with further firmware versions

  10. Jan Jokela says:

    Hi Carlos,

    Is the lag in the demo related to software or hardware (bad quality touch screen)?

    Oh, and regarding patents, I would encourage people to ignore the software ones, since for a start they aren’t valid in the EU and we can argue for prior art where they are applicable. Regarding the hardware patents related to touchscreen technology, those are responsibility of the hardware manufacturers. And in fact, Apple does have a sort-of-monopoly on captative touch panels in the sense that their panels are light-years ahead regarding touch accuracy and precision.

  11. carlosg says:

    @Jan, yes, it’s related to the hardware, it actually produces lots of nose for each screen contact, making pointers jump around below the fingertip, so the sensitivity was reduced in the Xorg input driver, I hope these issues are fixed in further N-Trig firmware versions.

    And quite I agree, there’s much prior art in MT to just avoid doing anything related :)

  12. Felix says:

    Hi really great. Keep up your good work.

  13. Dylan McCall says:

    Now we just need kinetic scrolling built in to GTK+’s scroll containers :)

    This is awesome stuff! Can’t wait for xinput 2 to land on my stable system so I can start hacking on it. (There are lots of little goodies I think should be pushed into Gnome asap).

    With regards to patent fears, my personal approach would be to not use gestures; do it procedurally with a verlet cage physics simulation, where each touch event is just applying a force to the nearest point. (Should automatically produce beautifully smooth scaling / rotating stuff). That’s probably just because I over-think things.

    How does the multitouch screen driver talk to XInput? Does it create a new pointer for each and every touch event, or is there something else going on? Can clients get more information, such as a picture of the finger / whatever it is that is affecting each pointer?

    (And what kind of touch screen is it? Diffused Illumination, where it sees a detailed picture of whatever is on there; or FTIR, where it specifically sees fingers; or something else?).

    Here’s a related goody for you: http://2dboy.com/2010/01/29/fingers-full-of-goo/

    Awesome work!

  14. Aaron says:

    Is there a way using a webcam to get multitouch to work in linux ona a computer that does not support multitouch?? I know there is for windows 7.

  15. iFrade says:

    WOW, awesome! Well done!

  16. Cody Russell says:

    Garnacho, you kick ass! Keep up the awesome work!

  17. carlosg says:

    @Dylan, yeah, there are indeed plenty of things to do to make apps feel just good with MT :). Yes, ATM the X11 driver just creates a fixed number of subdevices, which later are assigned to a finger, these devices also appear in xinput –list, and can be assigned to a master pointer. Clients get to see each finger as a separate device, there are plans in the future to add blob events to X11, so clients would get to know the shape of the touch points. This device provides touchpoint rectangle length/width and orientation for sure. I think it uses capacitive technology, there’s not much info actually about N-Trig’s DuoSense.

    @Aaron, there is I think, but might not be what you expect. There are tutorials to build a multitouch pad with a webcam, and using touchlib, I think there’s also a X11 driver in development, see http://www.youtube.com/watch?v=pQpr3W-YmcQ&feature=fvw

    @Jono, @iFrade, @Code, thanks :)

  18. shane says:

    I left a comment on the digg page of this article. But since I see more comments here, I’ll just post the link here too.

    Actually, I think the first working multi touch stack on any desktop was on Linux. It was/is called MPX. It worked well before Microsoft’s “big ass table”… I forget its real name. Apple has only now come out with multi touch on a large(r) screen with it’s very absorbing iPad experience.

    Check out this video on YouTube posted back in July 2007. Full multi touch working with full use of touch, keyboard and mouse simultaneously and even working for multiple users on the same interface. What I find very cool is that when the two people are painting in the same window, changing the color for one user does not affect the other user. Pretty awesome!

    http://www.youtube.com/watch?v=Co3FONI6kHU

    Remember, you saw it first on Linux ;)

  19. acs says:

    Really cool carlos!!!

    Interfaces with multi touch support will change the way we interact with all kinds of computers.

  20. carlosg says:

    @shane, Well, MPX is a really important piece here, and there’s indeed much to thank to Peter Hutterer, but it’s not the full stack, back in 2007 there was no multitouch interface in the Linux kernel, and most applications don’t use the X APIs directly anyway.

    What I mean with “full stack” is getting MT events right from the kernel, into Xorg (7.5 features MPX/Xinput2), down to a mainstream graphical toolkit (GTK+ in this case) and to a multitouch-enabled application.

  21. mrjack says:

    Nice video !
    The responce of your actions seems to be excellent !

    Looking forward to see multitouch getting mature in Linux.

  22. stephen says:

    As somebody who purchased one of the first tx2 tablets, I’m really glad to see the pieces all coming together. I’ve been trying to test this on my machine, but so far I can’t get any touches (with stylus or fingers) to register. I know this isn’t the best avenue for support, but since this touches so many softwares I don’t know where to start.

    What I’ve done: (this is under arch linux)
    - Installed the new driver/firmware under windows 7.
    - built and installed the evdev-multitouch driver linked above (with the linked patch)
    - built and installed the xi2 branch of gtk2+ (including updating atk)
    - patched and then built the hid-ntrig.ko driver (using make M=drivers/hid) and then copied this new module over the old one
    - removed any linuxwacom stuff I had around
    - moved my xorg.conf to xorg.conf.bak
    - ran depmod and rebooted

    Interestingly enough, my Xorg.0.log shows the device being recognized as an N-trig DuoSense, complete with “(type: MULTITOUCHSCREEN)” but still no response from any applications (including a patched eye of gnome)

    Is there anything else you might have done to make events get recognized and used?

  23. Digikid says:

    Posted this on the site that I belong to. This is a huge step in the right direction for ppl to migrate to Linux instead of Windows.

    Well Done Carlos!!!!

  24. ayuthia says:

    Are you using the 2.59 firmware (the four finger touch version) from N-trig? I was trying out your source with it and found that the cursor jumps around a lot with that version. Did you experience this also?

  25. carlosg says:

    @stephen, you might also need Xorg 7.5 (I think only F12 ships it at this moment, Lucid also ships it I think), having multitouchd from http://www.lii-enac.fr/en/projects/shareit/xorg.html eases testing by creating pointers for each touchpoint

    @ayuthia, yes, I have that firmware installed and also experience this, there’s a hack to minimize it a bit in my evdev-multitouch patch, but it seems to me a firmware issue, someday I’ll have to try out the latest firmware, but I only read bad experiences about it on the internet.

  26. [...] Multi-touch support in GNOME by Carlos Garnacho [...]