Real World Mapping with the Kinect

Many people have experimented with using the Kinect for more than just user interaction. One thing that I have been very interested in is extracting point clouds from the device.

People at the ROS (ros.org) project have gone to some trouble to determine empirical calibration parameters for the depth camera (disparity to real-world depth) here, and Nicolas Burrus has posted parameters for the RGB camera (relationship between the depth image and the RGB image) here.

Putting those together, one can take the depth image from the Kinect and turn it in to a metric point cloud with real distances. Then, those points can be projected back to the RGB camera centre to determine which RGB pixel corresponds to each depth point, and hence arrive a colour for each point in the cloud. This lets the surfaces captured in the image appear textured. With a bit of coding I came up with this:

A coloured metric point cloud taken inside my house. (That’s my hand in the foreground.)

One thing that I haven’t seen explored much is how the Kinect fares collecting point clouds outside. The claimed max range of the Kinect is 4m, but my device has been able to reach more than 5.5m inside.

Because the Kinect operates using infrared structured-light, infrared interference can reduce the range significantly or even result in no depth image at all. This is a problem when using the device outside as sunlight during the day plays havoc with the depth image returned by the Kinect. Of course, in the dark you will get a great depth image but no RGB image to colour the cloud!

There is a YouTube video posted by some robotics students showing how the sensor operates in sunlight:

Inspired by the video I decided to try it for myself – so I attached a Kinect to my car…

Using the software I had already written I could capture point clouds with metric distances relative to the Kinect. However since the Kinect itself is moving I wanted a different output. I wanted a real-world point cloud that spans many depth frames. Collecting all the information needed to reconstruct a 3D world that is spatially located meant I had to write a bit more software…

To spatially locate my car (and hence the Kinect itself) I used the GPS in my Google Nexus One along with BlueNMEA. This allowed me to get NMEA strings from the GPS in the phone via a TCP connection and log them. Using that information I could locate each depth frame and image frame and build a point cloud in a real-world coordinate system (so every point has the equivalent of a latitude, longitude, and altitude).

My software talks to the Kinect and Phone in real-time and logs all the data needed to export a point cloud. I wrote an exporter for the PLY format so I could easily view the data in the awesome open source MeshLab.

In the end I was able to capture some pretty cool looking things like this nice white picket fence:

Combining a section of depth frames you can get an idea of the power of the method. Here is a 26m section of road travelling at speed:

These points are all in real-world coordinates and could be put, say on Google Earth and appear in the right place. The point cloud is a bit messy because I did not have easy access to gyroscopes or accelerometers to track the motion of the car. Perhaps this is a good excuse to purchase a Nexus S! I did not bother to access the accelerometers in the Nexus One because it doesn’t have a gyro and so the outputs are of limited use for dead reckoning.

The project uses libfreenect and the .NET wrapper for same, along with OpenTK for the Matrix maths and Proj.NET for the spatial transforms. All amazing libraries and I’m in awe of the developers who spend their time maintaining them.

The code will live on GitHub here. It’s very hacky and mostly useful as a proof-of-concept. If you’re going to do anything with it, please wear a helmet.

Update #1: Reaction to Slashdotting

Update #2: Future Plans, Raw Data Downloads, and Better Point Clouds

Tagged , , , , , , , , , , , ,

60 thoughts on “Real World Mapping with the Kinect

  1. Hjalmar says:

    Hello, I have been looking for projects like this for a while and I have to say that the results you got are pretty good. I’m not a coder but I have two suggestions/requests. First I’d like to see this code make use of SLAM, for abvious reasons, (unless it has some serious flaws I don’t know about) and then I’d like to suggest that you put “A feature connecting several different kinects/computers to make one big collaborative point cloud” somewhere on the project roadmap.

    • Hi,

      Thanks for your comment!
      I did want to expand the project to multiple simultaneous Kinects at some point to expand the point cloud coverage.
      All I need is a few more Kinects!
      Software-wise, that shouldn’t be too difficult to do with the existing codebase for anyone who wants to take a shot.
      In terms of SLAM I am really interested in the use of the Kinect for robotics, but I’m not sure how much is worth writing myself given that the awesome coders at ROS have some really great stuff going.
      People have used ROS to control an iRobot Create (sort of like a Roomba) with a Kinect. It may be worth investing some time in that. I do want to purchase a Create so it might be a good excuse to get working on a automapping robot :)

      • Hjalmar says:

        If I remember correctly there is already a library for SLAM which you can use with point clouds so you don’t need any positional equippment, and it doesn’t have to be used with robotics if you attach it to your car, bike or backpack.

        I would use it to make a collaborative, huge 3D map which would update itself and get more accurate every time you or someone else provides new scanning data.

      • I will search and see what I find. Thanks! The original idea for this was kind of an openstreetmap for 3d so I think we are on the same page!

      • Manifold Shenberger says:

        Hjalmar,

        The SLAM libraries in ROS take a laser scan, odometry, and construct an occupancy grid from that. They’re currently not set up to perform SLAM using pointcloud data.

        I’m currently working on exactly this problem, and have produced some pretty good indoor maps using the Kinect which I can use to navigate, although they’re not nearly as good as the laser maps.

      • That’s awesome! I know I would like to see your results very much. I have never really done much with ROS but it seems like it is a good base to implement these things upon. There are so many talented hackers doing amazing things. It looks like I have some research to do!

      • Hjalmar says:

        Oh, I didn’t know that, sorry for the misconception. You seem pretty knowledgeable so maybe you know of a way to convert point clouds into occupancy grids? I’ll do some digging around to see if I can come up with something.

      • Hjalmar says:

        http://openslam.org/slam6d.html This piece of code seems to work with point clouds. If it does then it shouldn’t be too hard to combine two readings’ data into one map, at least if you don’t do it in real-time. If that is a requirement maybe you could run one instance of the program for each of the streams to create several maps and then e.g. send the maps to a server every once in a while and combine the maps there and then send it back to the scanners. It can be solved differently depending on how you want it to work and who’s supposed to use it.

      • Thanks for the link!
        I will check it out tonight.
        Looks very promising…

  2. RigorMortis says:

    I now live in hope that this idea will eventually be used to map RL locations for driving sims.

  3. [...] Martin Szarski strapped a Kinect sensor to his car, hooked it up to a laptop and Nexus One phone (for location data) and set about creating a “point cloud” of his car’s path. Using a phone with a gyroscope to better sense the car’s movement, Szarski said, the cloud would appear clearer. But for a 3D space captured by a $150 game controller, it’s pretty impressive, no? Imagine this cheap DIY tech in the hands of indie devs, who could create levels by exploring in real life. [...]

  4. [...] Martin Szarski strapped a Kinect sensor to his car, hooked it up to a laptop and Nexus One phone (for location data) and set about creating a “point cloud” of his car’s path. Using a phone with a gyroscope to better sense the car’s movement, Szarski said, the cloud would appear clearer. But for a 3D space captured by a $150 game controller, it’s pretty impressive, no? Imagine this cheap DIY tech in the hands of indie devs, who could create levels by exploring in real life. [...]

  5. [...] at home – Szarski’s made the code he used freely available, and you can find details on his blog. Now, if we can just find some way to hook it into [...]

  6. [...] Real World Mapping with the Kinect Many people have experimented with using the Kinect for more than just user interaction. One thing that I have been [...] [...]

  7. chris says:

    google street view 2.0

  8. Suddenly, I want to map my entire house this way. Maybe a university. Just need to spin the Kinect somehow, or perhaps hook up two at once on some kind of triangular cart. I wonder if you could combine night 3D IR scanning with daylight photos? More likely Google would just use 3D cameras from some kind of angle analysis.

  9. [...] Usando Kinect para crear Google street view en 3D [ENG]   blog.decoratorpattern.com/2011/01/23/real-world-mapping-w…  por mblanch hace 4 segundos [...]

  10. gadget says:

    gimmick and crap.

  11. Anonymous says:

    [...] proprio per questo, ha messo a disposizione di tutti il codice della sua applicazione sul suo blog: Real World Mapping with the Kinect Decorator Pattern (Martin Szarski's Blog) PuLpPeTrOz __________________ MILAN FOREVER Le Mie News Le Trovi Anche su: [...]

  12. steve says:

    If you want easy access to accelerometers and gyros check out http://www.chrobotics.com/index.php?main_page=page&id=1

    They make sensors with a micro-controller that does all of the hard sensor filtering for you. I think they also have one with a magnetic sensor as well for heading.

  13. mickmel says:

    Any chance of seeing a Google Earth KMZ of any of this data? I’d love to play with it in there.

  14. AvgJoe says:

    You really achieved some impressive results! You’ve also made slashdot.
    Congratulations.
    I work in construction technology and one of the newest trends is point mapping existing architecture and structure for inclusion in Building Information Models (BIM). The current machines do very similar functions as your creation but they are exorbitantly expensive and I am not aware of any that do not operate except from a fixed position.

    • TomDArch says:

      AvgJoe is thinking the same thing I am! I am an architect with a very small firm who works on small projects, often remodeling of or additions to existing buildings. This type of system could be very, very helpful in documenting existing buildings. Even if you had to drag a tripod from room to room and then align sets of individual scans, this would be very helpful, both to speed up measuring and improving accuracy. Of course, if a Roomba base drove around each floor of the building assembling a coherent model, that would be even better! (Although there are a bunch of reasons that you would want the primary scanning head 2m or higher off the ground… potentially with a second head near ground level… but I won’t be picky!)

      The amount of information you need to gather for good base drawings is astounding – room dimensions, of course, along with floor-to-ceiling heights, and door sizes, and window sizes and heights off the floor, and their locations, and ideally locations of plumbing fixtures, and electrical outlets, and ventilation registers, all the elements of staircases, and walls that aren’t square/level and on and on. Working from the resulting scan to develop a CAD/BIM version of the building would still be labor intensive, but there’s a ton of potential for many, many hours saved and much more accurate, complete drawings.

      • I’m definitley interested in using the iRobot Create as a base for an indoor mapping solution, however you might want to look at what is already out there since I know there are some impressive things being done with ROS and the exact same setup.

  15. steve wirts says:

    seems like this might work for 3d scanning of objects for use with a 3d printer..
    if it were integrated into a single device it would be the worlds first 3d copy machine!

    http://wiki.makerbot.com/3d-scanner-usage

  16. Alrecenk says:

    Interesting work. I did a lot of research into auto-calibration, point matching, and 3D modeling of a similar sort for several months last year, but it was all limited to arbitrary 2D images without any initial location data and I ran out of funding before I could polish or publish any of my algorithms. Using a depth sensor and adding GPS and accelerometers and the like adds so much potential to the quality of models that could be generated, and so much more data that could be used to specifically match points and refine the extrinsic parameters. I get excited just thinking about it. If you’re ever interested in bouncing ideas off of someone I would love to talk mad computer vision science. A little chunk of the work I did get done is online here: http://www.alrecenk.com/resources/ImageModeling/

  17. Resin Cacti says:

    I see this more for something like an online 3D social system, something like the metaverse of Neil Stephenson Snow Crash

  18. Resin Cacti says:

    I hope you can see into the future of the social network, you would make Fa<eb00k a thing of the past, Martin!

  19. Kinect Hacks says:

    [...] Real World Mapping with the Kinect « Decorator Pattern (Martin Szarski’s Blog) [...]

  20. [...] Real World Mapping with the Kinect « Decorator Pattern (Martin Szarski’s Blog) RSS 2.0 Comments Feed | Leave a Response | Trackback [...]

  21. Acurus says:

    This is soo cool! :)

    I think some of the problem with matching the clouds isn’t all down to lack of Gyro. The accuracy of a Phone-GPS isn’t that good. 10m in good conditions. What you could try is to create your own coordinate system, and placing the different point clouds relative to each other. Using common recognizable features in two or more of them.

    Thats how we do it when scanning inside buildings using “real” 3D-scanners.

    You might allso want to check out SAGA GIS (http://www.saga-gis.org/en/index.html) this is a open source Geographic information system that handles point clouds in a geographic setting.

  22. [...] The many meanings of life – New Scientist >> * Real World Mapping with the Kinect >> * Holocube scales up with life-sized, 70-inch, HC70 holographic projector (video) [...]

  23. Toby Howard says:

    Hi Martin, superb work, well done.

    I would like to show some of this in the Computer Graphics classes I am teaching. May I possibly get the point cloud files to show dynamically in Meshlab? I will give full attribution, of course!

    Many thanks,

    best wishes
    Toby

  24. [...] Real World Mapping with the Kinect (via Decorator Pattern (Martin Szarski’s Blog)) Leave a Comment Posted by underdark on 2011-01-25 Always wanted your very own 3d scanner? Try Kinect: Many people have experimented with using the Kinect for more than just user interaction. One thing that I have been very interested in is extracting point clouds from the device. People at the ROS (ros.org) project have gone to some trouble to determine empirical calibration parameters for the depth camera (disparity to real-world depth) here, and Nicolas Burrus has posted parameters for the RGB camera (relationship between the depth image and th … Read More [...]

  25. Josh Harle says:

    Excellent work Martin!

    I’ve been working with Structure from Motion for a bit now, and had a look at connecting up Kinect information using that. It is not realtime, but you can take the RGB images, use SfM to calculate camera positions, and then use that to translate your 3D pointclouds. I’d like to experiment a bit more with this, but don’t have enormous amounts of time. If you could make available a 20-600 odd image set of the RGB image output, I could give you relative camera positions.

    I have some code to pull camera positions into 3DS max, but unfortunately my translation matrix abilities are not amazing, and my kinect is not correctly calibrated, so I’d need your image set.

    • Fascinating! I will have to look into SFM more… I am also thinking about publishing a small dataset when I get the chance. It would be a good fit because I’m not going to have a lot of Kinect time over the next week or two. Be amazing to see what others come up with!

  26. [...] would be not give for maps navigable in the full 3D glory. Have a look at Martin Szarski’s blog for a more comprehensive insight on how he managed the 3D mapping [...]

    • Teresa says:

      Great eimprxeent showing the future and makes you think of the diversity of possibilities to augment people in a new reality. We have put it on the augmented reality overview. Thanks for sharing.

  27. [...] January 27, 2011 Martin Szarski Leave a comment Go to comments The response to my blog post was amazing and many people have suggested some really great ideas to improve the data collected. [...]

  28. [...] me about this and I just happen to just stumble across it. JC Fletcher of Joystiq has a piece on a ambitious guy who used his Kinect to map things around his neighborhood. Fletcher comes up with an [...]

  29. naesk says:

    Awesome!

    Looks like the potential first steps towards a openstreetview project to compliment openstreetmap ;)

  30. Flora Salim says:

    Hi Martin, this is impressive! A great working and low-cost 3D scanner. We’ve got one terrestrial laser scanner in the lab (architecture, design and engineering lab) which is terribly expensive and complex to set up and it scan at a fix point.

    Please check your admin email of this blog.
    I have sent you a message this morning to that email since I don’t know your personal email.
    Would like to see further development of this and wondering if you’d like to collaborate in our workshop that involves Kinect, tangible tables, 3D modeling and social network mashups.

  31. Tim Reid says:

    Hi Martin, great article. I’m interested in the potential of Kinect for some geoscience applications outdoors, e.g. getting the 3D structure of trees, and maybe monitoring snow and ice melt, as a potential low-cost alternative to laser scanning. Can you recommend the best drivers and if possible open source software to generate point clouds? Also would you know if the snow and ice would produce any problems in terms of their transmittance and reflectivity? And would we have to do it at night?
    Many thanks,
    Tim Reid, University of Edinburgh

    • Hi Tim,

      Thanks for your post!
      I have never tried using structured light on a highly reflective surface like ice – I use time-of-flight LIDAR sometimes and extremely high reflectivity can cause issues there. The wavelengths of the NIR LIDAR are pretty similar to the wavelengths used by the Kinect projector, but I don’t know about the CCD response in the Kinect compared to the diode in the laser scanners. Sorry!

      I would think that for close range scanning of trees it could definitely be an inexpensive alternative to terrestrial LIDAR. I have come to the conclusion that as a general purpose vehicle mounted sensor is isn’t super useful, but for close range it really cant be beat (and you get colour!).

      I used libfreenect in my projects which is the reverse-engineered community developed driver. If you are comfortable with C# the source for my code is MIT/X11 licensed and there’s a quick implementation of the calibration between the RGB sensor and the Depth sensor to look at. That’s here. If you want something as a quick proof of concept, there is an openFrameworks plugin called ofxKinect that does most of the hard work for you and just gives you a pointcloud. You would only have to write the code for export. There is a really active google group for libfreenect that has a wealth of information.

      If you want to talk more feel free to email me at admin@decoratorpattern.com. I would be happy to help develop your project further.

  32. [...] at home – Szarski’s made the code he used freely available, and you can find details on his blog. Now, if we can just find some way to hook it into [...]

  33. Sarah Norris says:

    I’m working on a senior design project that will incorporate a lot of what you have posted here. Do you have any source code that you have made available? Also, How did you compensate for the interference of sunlight with the IR sensors? When we were testing them we were not getting a very good depth map. Your work is impressive.

  34. Sarah Norris says:

    Also, Has anyone done any integration with Matlab and SLAM? My project is to build an autonomous robot that will be able to build a 3d map of a building’s floor plan and being able to use the functionality of Matlab would be great!

    • Hi Sarah,

      I wouldn’t expect too much from the code – it is mainly logging and sensor reading related. Furthermore it was written before the Kinect had so much awesome driver and wrapper support so it uses the ancient libfreenect stuff.
      However it is available here if you do still want to take a look: https://github.com/mszarski/KinectMapper

      Unfortunately I don’t have a lot of time to mess around with this anymore but there are some really nice SLAM projects around – I’ve linked to some awesome stuff before, but RGBDemo and some of the examples on the ROS Wiki could be really helpful for your project.
      If you’re indoors you won’t be able to log GPS like I did so you will have to rely either entirely on feature points from the RGB and depth cameras or some external inertial sensors.

      I use MATLAB a bit in a professional capacity but I’ve never used a Kinect with it. There doesn’t seem to be a lot of open source SLAM stuff around, it appears most Universities keep it to themselves for the most part.

      Good Luck with your project!

  35. DSLR-A900 says:

    csodálatos be, nagyon informatív. Én kíváncsi vagyok, miért ellentétes szakértői ebben az ágazatban nem veszik észre ezt. Továbbra is az írás. Biztos vagyok benne, amit egy hatalmas olvasói bázis már!

  36. amin says:

    Hello
    I need your help,…
    I want to convert depth map image into mesh and I need sample c++ code for that
    Do you have code or do you know where I can find sample of that
    Thanks before hand for your help
    regards
    /amin

  37. Luis linares says:

    Hello Martin. You have made such an awesome feat! However, I still don’t understand something: Were you able to get 3D point clouds from the Kinect, while being outdoors? While in the car, was the Kinect inside the car, or outside (perhaps, on top of the roof)? Did you have to add some sort of optical filter? Thank you!

    • Hi Luis, yes I captured outdoor pointclouds usibfvthe Kinect attached to the exterior of my car. In direct sunlight it functions very poorly, bit at dusk it is acceptible while being range limited compared to indoors.

  38. Azri says:

    Hey,
    Do you have the coding in C#, i need it for reference for my project.
    Your kind help would be really really appreciated.
    Thank You!
    -Azri

  39. [...] Real World Mapping with the Kinect « Decorator Pattern (Martin Szarski’s Blog) [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 179 other followers

%d bloggers like this: