TuioKinect Tracking

On January 1, 2011, in Kinect Dev, by Vlad Cazan

After connecting my Kinect to my computer I really wanted to play around with some hand tracking algorithms. I wrote a basic processing brightness tracking code that sort of worked but I knew that using some other libraries could really improve my results.

With some Googling I managed to find a library canned TUIO. “TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states.”

After a few hours of playing around with this software/library pack I was amazed on how powerful it really is. They have just released an open source app for OS X called TuioKinect. This program controls your Kinect and outputs multiple points. By setting the forward and backward threshold I was able use this to detect both my hands and output the X and Y coordinates.

Since this software was not just designed for Kinect, there are already 10s of different clients for different codes already written and optimized. I took a loot at the first example using Actionscript but could not manage to get it to work. There was some UDP socket that you are required to run to patch the connection from TUIO to Flash. I did manage to get the processing example working and the results were better then I expected. Here is a screenshot of the results. The lines on the right are the tracked movements from the hands on the left.

Setting up this does require a few seconds and is not optimal for end users. There must be a way to calibrate the front and back thresholds without anything input.

Clearly the next step that is required is a full body skeleton tracking algorithm which can use vectors for each joint just like the calibration in the xbox menu is.

If you would like to download TuioKinect visit:

I have also found a really good example of a fluid library for processing that is already integrated with TUIO. MSAFluid is a library for solving real-time fluid dynamics simulations based on Navier-Stokes equations and provides some amazing visual effects. Here is a picture of its output with the Kinect.

*Update* If anyone can not find the link to the MSAFluid example it is here:

To run this demo just run TuioKinect first, then run “MSAFluidTuioDemo” from processing and start moving your hands.

I think im going to be playing around with this app for a while! Like always, if you have any questions or comments please leave them bellow.

Happy hacking!

Tagged with:  

11 Responses to “TuioKinect Tracking”

  1. Doniani says:

    Hi Dear Vlad.

    I get some files of TUIO and more, but i don’t know how i can use with the MSAFluid. Its awesome this program, but really i need a setp by step to run TUIO with MSFLuid on Kinect.

    Can you help-me?

    Sorry for my english, i’m from Brazil.

    Thank’s a lot.

    • Vlad Cazan says:

      Hey Doniani,
      It is really easy to set up, what you have to do is download MSAFluid from:, and inside there will be an example called MSAFluidTuioDemo. You have to first start Tuio, then run the sample from processing and everything should work.

      Let me know if you need anything else

      Vlad Cazan
      cazanv at

      • James says:

        Vlad, thank you for this information. I’ve been playing around with Processing and MSAfluid for a few months, but I’ve been wanting to use my Kinect for Windows with it and didn’t know how. Your site is helpful.. And I’ve downloaded the tuiokinect files, but how do I “start” Tuio? I can’t figure out how to start it to use in Processing.

        Thanks for any help!

        • Vlad Cazan says:

          Hey just email me at cazanv @ and I can help you out. Thanks for the support!

          • Stan says:

            Hey Vlad,
            Thanks for your instructions.

            I run TuioKinect Mac app and when I run the MSAFluidTuioDemo, I receive “The function getTuioCursors() does not exist.” I use Processing 2.2.1, Kinect Xbox, Mac.
            Any hints?

            Big thanks,

  2. James says:

    Thank you for this helpful site, but the only thing I don’t understand is how to “start” TUIO. I can run everything else fine, I’ve been playing with MSAFluid for a few months, but I’d like to use my Kinect for Windows to track the hands of a user.


  3. Maomao says:

    I want to use kinect to track tuio( the pattern looks like qr code) in processing. Could you tell me how can I do it? I think it is related to tuiokinect but still confused.

  4. nada says:

    it appears none of this is supported anymore. new processing returns error saying OpenGL and TUIO library can not be found – even though OPENGL is part of processing now and TUIO library is installed. I guess more reason to C++/openframework

  5. ora exacta says:

    Am aflat aceasta pagina, dupa ce am cautat despre TuioKinect Tracking
    pe Google. Se pare ca informatia dvs e foarte valoroasa, mai ales ca am mai gasit aici si despre ora,
    ora exacta, lucruri interesante si folositoare. Mult succes in continuare!

  6. idolpx says:

    To get this to run in Processing 2.2.1 I had to make the following tweaks.

    In “TuioHandler.pde” change the following line from

    void updateTUIO() {
    Vector tuioCursorList = tuioClient.getTuioCursors();
    for (int i=0;i<tuioCursorList.size();i++) {
    TuioCursor tcur = (TuioCursor)tuioCursorList.elementAt(i);


    void updateTUIO() {
    ArrayList tuioCursorList = tuioClient.getTuioCursorList();
    for (int i=0;i<tuioCursorList.size();i++) {
    TuioCursor tcur = (TuioCursor)tuioCursorList.get(i);

Leave a Reply


Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

Free WordPress Theme