Guillaume Charpiat's Projects in Images

The Shape Manifold


Images are zoomed when clicked.


Aim : to build statistics of a set of shapes and estimate suitable metrics, i.e. estimate which shape deformations are probable or not
Method : based on correspondences between close shapes, on information propagation via transport on a manifold
Difficulties : to handle very varied shapes, without training set density assumption (i.e. we do not suppose that we have seen all possible shapes during the training phase); also, shape matching is not reliable if the shapes are too different
Application : shape matching, shape priors, and, further, image segmentation with shape prior


Correspondence flow
To find point-to-point correpondences between two shapes, we search for the most reliable path between them in the graph of all already-observed shapes. The most reliable path is usually a set of small reliable steps (it is more reliable to match shapes that are relatively similar, than shapes that are very different).
correspondence flow between two shapes correspondence flow between two shapes
Here are two pairs of shapes, with, in each case, the optimal path (in the associated training set of shapes) between them. Then we are able to associate each point of each shape to its corresponding point on the other one. Then, a gesture applied observed on one shape (i.e., a deformation) can be transported to the other shape (since we know the correspondences), and we can let the other shape undergo the gesture as well.


Information propagation
Through point-to-point correspondences between shapes, we can propagate information (and since we know the reliability of the correspondences, we know how reliable the propagated information is). For example, if we assign colors along one shape from one frame in a video, we can propagate the colors to the other shapes from other frames. Points which are in correspondences will then share the same color. Thus we can visualize easily the quality of the correspondences found.
walking silhouette → walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette walking silhouette

This is better visualized with a video :
walking silhouette or walking silhouette
.gif .mpg



Dancing silhouette
We consider a video from Grimage platform, Inrialpes, for which we are given the segmentation of all frames. Here are a few sample frames (click on any of them to watch the video):
dancing silhouette dancing silhouette dancing silhouette dancing silhouette dancing silhouette dancing silhouette dancing silhouette dancing silhouette dancing silhouette

For any posture, we can estimate the most probable motions, in order to know what kind of deformation the posture is likely to undergo. We show here the first modes of deformation for different postures; note that they seem to be articulated and make sense for a human silhouette (while we have no articulated model of a human in these experiments, and while in the video, each posture undergoes only one movement, which is not representative of all movements that this posture can undergo):
1st mode 2nd mode 3rd mode 4th mode 5th mode 6th mode
Posture 1 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated
Posture 2 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated
Posture 3 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated
Posture 4 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated
Posture 5 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated
Posture 6 deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated deformation prior estimated

Each deformation is represented twice, once with positive amplitude, and once with negative amplitude (as in forward/backward for gestures). The red shape is the posture considered, the green field is the deformation applied, and the black shape is the result. Click to zoom and see details.



Associated publications:

Back to the Projects in Images page
Back to the main page

Valid HTML 4.0 Transitional