Examples of OpenCV routines from the Processing library documentation. Of course, it’s up to you to build on these techniques and make art.

It’s a relatively easy thing for computers to “see” video, but “computer vision” goes a step further, applying a wide range of techniques by which computers can begin to understand and process the content of a video input. These techniques tend toward the primitive, but they can also produce aesthetically beautiful results. The best place to start with computer vision has long been the standard library, OpenCV. A free (as in beer and freedom) library developed by Intel and with ongoing use in a variety of applications, OpenCV is a terrific, C/C++-based tool not just for things like motion tracking, but video processing in general. OpenCV gets a lot of support in the C++-based OpenFrameWorks, but that doesn’t mean Java and Processing have to be left out of the fun. (In fact, I’d still recommend starting with Processing for OpenCV, because other tasks remain easier in Java.)

I expect to have us working a lot with OpenCV over the course of 2009. To get the ball rolling, I’m pleased to welcome guest writer Andy Best, a self-described “Interaction Designer/visualist/musician/general technology tinkerer.”

Andy Best

Before you get started, be sure to grab the library, following the instructions here (and check out the quite-nice, clear examples):
OpenCV library

Note that this will also work with standard Java projects, not just Processing. I’ll also be doing some cross-platform testing on Mac, Windows, and Linux with video processing using this library and GStreamer/GSVideo for Processing.

Here’s Andy:

Learning OpenCV with Processing

A relatively new library for Processing is the OpenCV library. It ports over some of the functions from the OpenCV library to work with Processing and is designed for realtime video processing.

In this tutorial I am going to cover some of the basic functionality of the OpenCV library for Processing in order to create an effect for live video, so you will need to install the library first. Instructions can be found on the site that I linked to above. I shall try to stick to OpenCV library functions and native Processing functions to make things a bit easier. This means that there is lots of room for optimisation in the final product.

The Processing implementation of OpenCV contains functions for image and video capture, image manipulation, blob detection and object detection.

A basic OpenCV sketch framework that captures frames from the camera and displays them onscreen is shown below:

import hypermedia.video.*;        //  Imports the OpenCV library
OpenCV opencv;                    //  Creates a new OpenCV Object

void setup()
{

  size( 320, 240 );

  opencv = new OpenCV( this );    //  Initialises the OpenCV object
  opencv.capture( 320, 240 );     //  Opens a video capture stream

}

void draw()
{

  opencv.read();                  //  Grabs a frame from the camera
  image( opencv.image(), 0, 0 );  //  Displays the image in the OpenCV buffer to the screen

}

The OpenCV library has an image buffer. In order for an operation to be performed on an image, it needs to be copied into the buffer. This can be done like below:

PImage anImage;
opencv.copy(anImage);

Once the image is in the buffer, OpenCV functions can be performed on it. Operations like capturing a camera frame will put the camera frame into the buffer.

A useful function is absDiff() (absolute difference). This calculates the difference between two images, the one in the regular buffer and one in a second buffer (which an image can be stored in using the ‘remember()’ function. absDiff can be used to perform a number of useful functions. One of these is background subtraction:

import hypermedia.video.*;        //  Imports the OpenCV library
OpenCV opencv;                    //  Creates a new OpenCV Object

void setup()
{

  size( 320, 240 );

  opencv = new OpenCV( this );    //  Initialises the OpenCV object
  opencv.capture( 320, 240 );     //  Opens a video capture stream

}

void draw()
{

  opencv.read();                  //  Grabs a frame from the camera
  opencv.absDiff();               //  Calculates the absolute difference
  image( opencv.image(), 0, 0 );  //  Display the difference image

}

void keyPressed()
{
  opencv.remember();              //  Remembers a frame when a key is pressed
}

When a key is pressed, the current frame is stored in memory and is used when calculating the absolute difference. This means that only changes between the stored frame and the current frame are shown.

Another use for absolute difference is for detecting movement. If the absolute difference is calculated from the current frame and the frame before, only the movement is shown:

import hypermedia.video.*;        //  Imports the OpenCV library
OpenCV opencv;                    //  Creates a new OpenCV Object

void setup()
{

  size( 320, 240 );

  opencv = new OpenCV( this );    //  Initialises the OpenCV object
  opencv.capture( 320, 240 );     //  Opens a video capture stream

}

void draw()
{

  opencv.read();                  //  Grabs a frame from the camera
  opencv.absDiff();               //  Calculates the absolute difference
  image( opencv.image(), 0, 0 );  //  Display the difference image
  opencv.remember();              //  Remembers the current frame
}

This movement image can be made more useful by converting it to greyscale and running it through a threshold filter. This will convert the pixels to black and white, which means that you can see if movement has occurred in a specific area of the image by checking to see if it contains white pixels. The standard threshold function takes 1 argument that is a value between 0 and 255. All pixels with a brightness equal to or above this value will be white and all others will be black.

opencv.convert( OpenCV.GRAY );
opencv.threshold(20);

We’re going to use this to create an effect that will overlay a coloured trail of the movement on top of a camera input.

Firstly, let’s create the trails.

import hypermedia.video.*;        //  Imports the OpenCV library
OpenCV opencv;                    //  Creates a new OpenCV Object
PImage trailsImg;                 //  Image to hold the trails
int hCycle;                       //  A variable to hold the hue of the image tint

void setup()
{

  size( 320, 240 );

  opencv = new OpenCV( this );    //  Initialises the OpenCV object
  opencv.capture( 320, 240 );     //  Opens a video capture stream
  trailsImg = new PImage( 320, 240 );  //  Initialises trailsImg
  hCycle = 0;                     //  Initialise hCycle
}

void draw()
{

  opencv.read();                  //  Grabs a frame from the camera
  opencv.absDiff();               //  Calculates the absolute difference
  opencv.convert( OpenCV.GRAY );  //  Converts the difference image to greyscale
  opencv.blur( OpenCV.BLUR, 3 );  //  I like to blur before taking the difference image to reduce camera noise
  opencv.threshold( 20 );


  trailsImg.blend( opencv.image(), 0, 0, 320, 240, 0, 0, 320, 240, SCREEN );  //  Blends the movement image with the trails image

  colorMode(HSB);                 //  Changes the colour mode to HSB so that we can change the hue
  tint(color(hCycle, 255, 255));  //  Sets the tint so that the hue is equal to hcycle and the saturation and brightness are at 100%
  image( trailsImg, 0, 0 );       //  Display the blended difference image
  noTint();                       //  Turns tint off
  colorMode(RGB);                 //  Changes the colour mode back to the default



  opencv.copy( trailsImg );       //  Copies trailsImg into OpenCV buffer so we can put some effects on it
  opencv.blur( OpenCV.BLUR, 4 );  //  Blurs the trails image
  opencv.brightness( -20 );       //  Sets the brightness of the trails image to -20 so it will fade out
  trailsImg = opencv.image();     //  Puts the modified image from the buffer back into trailsImg

  opencv.remember();              //  Remembers the current frame

  hCycle++;                       //  Increments the hCycle variable by 1 so that the hue changes each frame
  if (hCycle > 255) hCycle = 0;   //  If hCycle is greater than 255 (the maximum value for a hue) then make it equal to 0
}

The trails are created by blending the movement image with the contents of the trails image each frame. This image is then blurred and darkened so that the trails blur and fade out over time.

Before the trails image is drawn to the screen, the “tint” function is used, which is an inbuilt function of Processing. In order to cycle colours with ease, the colour mode is changed to HSB (hue, saturation and brightness) so that the hue can be cycled with the hCycle variable.

Finally, we can overlay this coloured trails image on the original webcam image by creating a holder for the original camera image and blending it with the trails image.

import hypermedia.video.*;        //  Imports the OpenCV library
OpenCV opencv;                    //  Creates a new OpenCV Object
PImage trailsImg;                 //  Image to hold the trails
int hCycle;                       //  A variable to hold the hue of the image tint

void setup()
{

  size( 320, 240 );

  opencv = new OpenCV( this );    //  Initialises the OpenCV object
  opencv.capture( 320, 240 );     //  Opens a video capture stream
  trailsImg = new PImage( 320, 240 );  //  Initialises trailsImg
  hCycle = 0;                     //  Initialise hCycle
}

void draw()
{

  opencv.read();                  //  Grabs a frame from the camera
  PImage camImage;                //  Creates an image and
  camImage = opencv.image();      //  stores the unprocessed camera frame in it

  opencv.absDiff();               //  Calculates the absolute difference
  opencv.convert( OpenCV.GRAY );  //  Converts the difference image to greyscale
  opencv.blur( OpenCV.BLUR, 3 );  //  I like to blur before taking the difference image to reduce camera noise
  opencv.threshold( 20 );

  trailsImg.blend( opencv.image(), 0, 0, 320, 240, 0, 0, 320, 240, SCREEN );  //  Blends the movement image with the trails image

  colorMode(HSB);                 //  Changes the colour mode to HSB so that we can change the hue
  tint(color(hCycle, 255, 255));  //  Sets the tint so that the hue is equal to hcycle and the saturation and brightness are at 100%
  image( trailsImg, 0, 0 );       //  Display the blended difference image
  noTint();                       //  Turns tint off
  colorMode(RGB);                 //  Changes the colour mode back to the default

  blend( camImage, 0, 0, 320, 240, 0, 0, 320, 240, SCREEN );  //  Blends the original image with the trails image

  opencv.copy( trailsImg );       //  Copies trailsImg into OpenCV buffer so we can put some effects on it
  opencv.blur( OpenCV.BLUR, 4 );  //  Blurs the trails image
  opencv.brightness( -20 );       //  Sets the brightness of the trails image to -20 so it will fade out
  trailsImg = opencv.image();     //  Puts the modified image from the buffer back into trailsImg

  opencv.remember();              //  Remembers the current frame

  hCycle++;                       //  Increments the hCycle variable by 1 so that the hue changes each frame
  if (hCycle > 255) hCycle = 0;   //  If hCycle is greater than 255 (the maximum value for a hue) then make it equal to 0
}

Since Processing doesn’t allow us to perform the tint function on an image directly, the trails image is rendered to the screen with a tint, then tint is turned off and the original image is blended with it.

That pretty much wraps it up for this tutorial, I hope that you found it useful. Be sure to let me know what you do with it! I’ve got another OpenCV tutorial on my site that shows how to use the difference image to perform collision detection with objects. You can check out my work and other processing tutorials here. Thanks to Peter Kirn for putting this out to the masses!

  • Adam

    Thanks for posting this. I was researching OpenCV and feeling intimidated last night. Thanks for the great head start!

  • davide

    hi there
    what if I get "the package hypermedia does not exist. you might be a missing library.
    ???
    I've followed your instructions and placed the library in the library folder.
    I'm running processing 1.0

  • http://createdigitalmusic.com Peter Kirn

    Davide, not sure if this is your problem or not, but be sure to read the new release notes for 1.0. Libraries do *not* go where they used to — they now get placed in the library folder in your *sketchbook* folder. More details in the readme.

    We're sort of doing out of order, skipping the basics here a bit. ;)

  • http://www.bridd.com bridd

    Thanks a lot for this tutorial – this instantly gives me ideas of how I can start incorporating OpenCV… Very cool !

  • Pingback: rascunho » Blog Archive » links for 2009-02-07

  • Erdem

    i know here is not the right place to ask this question but is there anyone who knows any gesture and also augmented library for processing??

  • http://www.andybest.net Andy Best

    @Erdem http://www.bryanchung.net/?p=227 and http://www.bryanchung.net/?p=287 are possibly what you're looking for. For more advanced ARToolkit stuff, I'd say that OpenFrameworks is probably the way that you wanna go

  • naus3a

    first of all: great post :)

    second: works like a piece of cake on linux and – speaking of augmented reality – i find that using openCV (hypermedia.video.* instead of the default processing video library) is the easiest way to use artoolkit under linux.

    third: have a laugh watching me play peekaboo with my sketch: http://www.youtube.com/watch?v=q-PIG9I9Sgk

  • Pingback: AHPROJECTS » Blog Archive » Resource Links

  • Pingback: Create Digital Motion » OpenCV Motion Tracking, Face Recognition with Processing: I’m Forever Popping Bubbles

  • Pingback: OpenCV Tutorial | Interaction Design Umeå

  • http://www.abstractmachine.net Douglas Edric Stanle

    Howdy!

    Great intro. Happy to see the library being used. Just a quick note to let you know that Stéphane and I are trying to find a week free in our currently busy schedules to update the OpenCV library and add all the new bells and whistles we've been discovering over the last few months. The version you're exploring is a little old (1 yr), hence the fact that we don't have Optical Flow or Augmented Reality included. But the project is definitely not dead, and we will be adding all that stuff, yes.

  • Pingback: R.Seiji » links for 2009-02-23

  • Ben

    Hi there,

    I cant seem to get this working, i'm getting an error message: "UnsatisfiedLinkError: hypermedia.video.OpenCV.capture(III)V"

    What does it mean?
    I've copied the OpenCV folder to the Processing Libraries. Have I missed anything?

    Thanks

  • Eric

    Hi there, I’ve been reading with lots of interest the tutorial about openCV, great work !am new in this kinda programmation and i’d like to work on computer vision, especially in gait recognition. However i dont even know how to read an image(extract pixels and store them in a matrix) with openCv (include vc++).
    So I kindly ask if u can help me with a code or a good tutorial. I need some help. thanks in advance
    ps: my goal is to rewrite the canonical view synthesis source code !!

  • jack

    Hi peter,
    Wondering what development environment the processing examples are pictured in?

    It looks nice, potentially nicer than the processing default.

  • Pingback: kodama.pixel » Blog Archive » Multimedia Authoring - Week 05

  • Pingback: z3255862’s blog » Multimedia Authoring - Week 05

  • Pingback: El’s blog » Blog Archive » Week 6 - concept down, project to go!

  • Pingback: Create Digital Motion » Virtual Magic: Augmented Reality Card Tricks with Marco, OpenFrameWorks

  • Pingback: Prodical’s Blog » Blog Archive

  • Pingback: z3077559’s blog » Blog Archive » code research

  • http://blog.rafolio.co.uk Rich

    this is a great tutorial, very well explained.

    What I am wondering is if there is a way of dropping an external image into the buffer and have it overlay onto the live video in the same fashion?

    thanks

  • Pingback: kodama.pixel » Blog Archive » Multimedia Authoring - Week 09

  • Pingback: z3255862’s blog » Multimedia Authoring - Week 09

  • Pingback: z3258930’s blog » Transforming Mirrors – Major, Major, Delayed Update

  • Pingback: Create Digital Motion » Set Up Ubuntu Linux for OpenCV

  • Kel

    Used this code for a head start on a special effect for in a larp with a soul trapped in a mirror using two web cams (one cam atop the screen that made the "mirror", the other for ghost special fx backstage). It turned out great! Thank you very much for the tutorial.

  • verma

    any of u can send the code to generate a video by using frames in opencv

  • tiny

    Thank you for code. But I have a problem when I copy code to Java I can't see frame captured from cam instead I see black background. Please help me.

  • Pingback: Create Digital Motion » Computer Vision Goodness: OpenCV 2.0, OpenCV on Android, Book

  • EATYone

    Many thanks for this fabulous tutorial!!!!

  • Pingback: KINETIX

  • http://sampl.us SamPL

    Great introduction–I used some of these techniques for a computer vision casual game here:&nbsp <a href="http://;http://www.sampl.us/blog/328/kinetix/” target=”_blank”>;http://www.sampl.us/blog/328/kinetix/

  • Pingback: OpenCV thoughts and Flash BitmapData fun | The Lawrie Cape Blog

  • http://none Kenneth

    Hi..could you help me…Im having trouble using opencv in java eclipse.

  • Pingback: minipimer.tv » RECURSOS TÉCNICOS: EXPERIMENTACIÓN CON VIDEO Y CACHARROS

  • Syria

    Hello

    I have project now it has face recognition I wanted to know if OpenCv can help me in face recognition and how that

    thanks

  • rpkrahe

    Thanks for the software examples. I get a blank window for the captured image. I'm using an HP laptop integrated Webcam under Vista. The Webcam drivers are up to date. The Webcam captures images fine with HP Mediasmart software. And OpenCV works fine on images I read in from a disk file. Any ideas how to get up off the ground using Processing and OpenCV to capture images from the HP Webcam? Are there any OpenCV functions to read other information from the camera to help troubleshoot? Thanks.

  • Arildo

    Thanks a lot for this tutorial! I'm studying Opencv for my project at university, this helped a lot!

  • Math

    For those having trouble with the hypermedia doesn't exist problem. I had the same issue and my problem was that I should have had the "OpenCV" folder (not OpenCV 2.0 processing – or something) in the library folder in processing.

  • Joy

    Hello Peter Kirn

    Thanks for your great work.It will be easy to understand who are want to work on opencv with java.I am also interested to opencv with java.I have already done some program using java and opencv.But I have fallen a big problem.I can not save picture from real time video.I want to save picture from video using java and opencv.

    I want help to solve this problem from every programmer.i am wanting any type of advice or code for solving this problem.

    Please help me.

  • jojio

    how to do capturing and separation of frames from video using webcam

  • Mana

    I'm using java, eclipse.

    I'm unable to eliminate the 'import hypermedia cannot be resolved' error.

    What exactly do i need to add to my project?

    I have the OpenCV 2.0 folder. Also what is CDT used for specifically? Is it needed?

  • Chris

    I have been trying on and off for a year to get

    OpenCV working with Processing sketches.

    I always get:

    The package "Hypermedia" does not exist etc etc warnings on compile.

    Many others have had similar problems – there's a thread that has been discussing the problem for over a year, with no solution apparent:
    http://processing.org/discourse/yabb2/YaBB.pl?num

    I have tried all the suggestions on this thread, and many others, on four different Windows XP machines, with every version of OpenCV ever released.

    I can't get the darn thing to work.

    Is anyone clever enough to have a solution?

  • http://itp.nyu.edu spike

    Just would like to point out that there is a minor typo at the bottom. It reads:

    if (hCycle > 255) hCycle = 0; // If hCycle is greater than 255 (the maximum value for a hue) then make it equal to 0

    }

    When it should read:

    if (hCycle > 255) hCycle = 0;

    I know all you smart people figured that out immediately, but from one programming noob to another: here you go.

  • Pingback: Installing OpenCV for Processing in Eclipse on a Mac « DECO } Han

  • E

    Thanks so much for this!

  • Estamiloca87

    side effects to cephalexin – cephalexin side effects stomach pain

  • MariaMoon

    Congratulations all of this rare event that happens once in a hundred years!

  • mark

    hi, I use Processing Program on Windows.Ican get error about Unsatisfied link Error hypermedia.video.openCV . How can I solve this error?Could you please help me.

  • Impendeded

    Hi.
    Somebody please tell me where can i download a lot of Classical Music mp3 music? Thx

  • http://www.facebook.com/austin.lazarus Austin Lazarus

    Can’t get the library to work in Processing.

    • thewindomerarlecharacter

      @facebook-669572465:disqus what was the error u get?

  • Cameron

    Awesome!

  • Juan

    Is it possible to use OpenCV to play a movie with audio?  If not, can you suggest a library to use with Processing to play videos with audio?  
    Processing’s Video Library is really bad.

  • Pingback: Processing tutorials | Farestrans

  • Pingback: Indefinit» Blog Archive » Testing OpenCV with Processing

  • Shiriz

    Hey that is really cool!!! I love it.. thanks for sharing =)

    But im just wondering if its possible to import an image to the background?
    Meaning the camera only detects the human motion, and not the background.Is this possible?

  • Pingback: Tutorials

  • Pingback: Tratamiento de vídeo en tiempo real con OpenCV (Processing) | Laboratori de programació creativa

  • pendik

    please help,, I try to capture using processing with opencv library follow the script above, but when running camera doesn’t view the image, just dark like this, how to solve this problem? I’m using windows 7 and camera prolink pcc3220. Thanks a lot master :)

  • guilherme

    please help !!! .. i try …did not , the erro is unsatisfiedlinkerro:hypermedia.video.opencv.capture

    • Donaldduck999

      Rename OpenCV.framework to opencv.framework

  • guilherme
  • http://www.intorobotics.com/ Calin Dragos George

    This is one of the best tutorial that inspire me to write an overview of tutorials about how to setup openCV, article that can be found here: http://www.intorobotics.com/tutorials-setup-opencv-for-android-ios-linux-and-windows/

  • gs123

    Still getting the error: unsatisfiedlinkerro:hypermedia.video.opencv.capture

    Even after renaming the OpenCV.framewok file.

    Please help!!

  • lki92

    I get a “Could not run the sketch” error. Please Help.

  • frustrated

    Not sure if anyone is still looking at these comments but I’m getting the error:
    A library relies on native code that’s not available.
    Or only works properly when the sketch is run as a 32-bit application.

    When I run the code with the 32 bit application I get the error:
    A library relies on native code that’s not available.
    Or only works properly when the sketch is run as a 64-bit application.

    I’m only trying to run the very 1st snippet of code shown above.

    Any help with this matter is much appreciated. Thanks.

  • http://www.clippingpathspecialist.com/ Atiqur Sumon

    How to solve the OpenCV? And give me the code. Whom I am zooming my camera, it’s does not work