3D Motion Capture Using Wide Baseline Stereo Cameras

Abstract

Motion capture is the process of determining the kinematic and kinetic motions of objects or an object for analysis purposes. It can be performed by several means, such as mechanical, magnetic, inertial and optical. However, motion capture by computer vision based technique is gaining increasing interest. A significant part of it is human motion capture because of its number of potential applications and inherent complexities. The most significant complexities are capturing the pose and motion of a highly articulated human subject in outdoor environment, occlusions by self or other object and compensating the egomotion of camera for any jitter or vibration to name a few. Motion capture technology suffers mainly when the capturing is not performed in controlled surroundings which hinders motion capture in an outdoor environment requiring the traversal of large areas. Overcoming these challenges provide a great range of applications perspective as computer vision-based human motion capture is the best non-invasive solution for it. Applications are of broadly three types roughly, analysis, control, and surveillance. Although the most widely used optical motion capture system (MCS) to date has been Vicon, considering, it is evident that it will not be useful for many purposes of these application types as it only operates inside a laboratory with a robust structural setup. Therefore, as there is a need for more convenient MCSs in environments in which Vicon fails, vision-based MCSs have attracted the attention of many researchers, with their wide range of applications requiring different levels of accuracy. High-accuracy ones, such as in the clinical sector and gait analysis, usually use optical MCSs which are unaffordable for many users as they require complex arrangements of expensive equipment and, moreover, are sensitive to lighting conditions and shadows. Therefore, with the aim of overcoming these restrictions, lightweight stereo vision-based MCSs have been developed. Although there are many techniques available for them, very few have an ultra-wide baseline distance between cameras.

In the first part of this study, a stereo camera rig with an ultra-wide baseline distance and conventional cameras with fish-eye lenses is designed. This system is tested in different situations to validate its applicability as a stereo camera rig as well as its performance for motion capture and, eventually, articulated human motion capture in indoor and outdoor environments. Its cameras provide a wide field of view (FOV) which increases the coverage area and also enables the baseline distance to be increased to cover the common area required for both cameras' views to perform as a stereo camera. In this study, a marker-based approach, which provides an additional facility to track articulated human kinematics and is computationally very cheap as it does not require all the image features to be computed, is developed, with an adaptive thresholding method applied to extract each marker. As the cameras have fish-eye lenses, they cannot be well estimated using a pinhole camera model which makes it difficult to estimate the depth information. In this study, to restore the 3D coordinates, a unique calibration method is used to develop a relationship among the pixels’ dimensions and distances in an image’s and real world’s coordinates respectively. The quality of this 3D coordinate reconstruction is compared with that of an existing commercially available `gold standard' MCS, the Vicon system for indoor data, and is shown to have the same order of accuracy while incurring less cost and complexity.

In the subsequent parts of this study, occlusion detection is considered because, in the marker-based capturing of articulated human kinematics, the occlusion of a marker is one of the major challenges. The detection algorithm differentiates among types of occlusions and predicts any missing marker position where necessary. The designed approach is intended to capture motions where a traditional MCS fails to perform well, such as in an outdoor environment. Experiments are conducted in such situations and show similar results to those obtained indoors. Finally, as this design is intended to be mounted on a moving carrier, such as a drone or car, a method for compensating the camera's egomotion is proposed. Using it and the developed algorithms, data are taken from a cyclist in a situation in which both the camera and subject are moving outdoors under varying lighting conditions. While changing these conditions is another major challenge, the proposed system is not sensitive to different lighting levels and shadows.