2.2.1 Hardware and software systems.

2.2.1.1 Optical motion capture system. 15 Qualisys Oqus 300 and 310 cameras (Qualisys AB, Sweden, https://www.qualisys.com/) were used. The cameras were set to normal mode (full field of view) with a resolution of 1.3 MP and captured the 3D location of passive reflective markers of 0.7 cm diameter with a frame rate of 120 frames per second (fps). The Qualisys Track Manager (QTM) software was used for the acquisition of the optical motion capture data and for setting the synchronization triggering signal that was sent to the Grasshopper video cameras that were connected to the motion capture system.

2.2.1.2 Video cameras. Video data were collected using two hand-held smartphone cameras and two stationary computer vision cameras. For the hand-held cameras, the rear camera of the iPhone 7 (Apple Inc., USA, https://www.apple.com/) was used. The camera has a resolution of 1920 × 1080 pixels and contains the Sony IExmor RS, CMOS sensor. The video data was recorded with a frame rate of 30 fps. As computer vision cameras, we used RGB Grasshopper2 cameras (FLIR Systems Inc., USA, https://www.flir.com/) with a resolution of 800 × 600 pixels, 72 dpi, 24-bit depth and Sony ICX285 CCD sensors. The recording with these cameras was also done with a frame rate of 30 fps. The FlyCapture software provided by FLIR Inc. was used for setting up the cameras’ acquisition features and for processing the synchronization triggering signal coming from the motion capture system. We also integrated the MATLAB Image Acquisition Toolbox as it supports the Grasshopper computer vision cameras and provides blocks and functionalities such as hardware triggering, configuring acquisition parameters and recorded data format, and previewing the recorded data.

2.2.1.3 Inertial measurement unit sensors. The Noitom Neuron Edition V2 (Noitom LTD, China, https://www.noitom.com/) was used which comes as a bodysuit attached with 17 IMU sensors (Figs 2 and 3). Each sensor is composed of a 3-axis gyroscope, 3-axis accelerometer, and 3-axis magnetometer working at 120 Hz. In addition to the acceleration data, the IMU suit provides computed 3D displacements, velocity, quaternions, and rotational velocity for each joint (all represented in an initial global coordinate system). The IMU sensors’ dynamic range, accelerometer range, and gyroscope range are 360 deg, ±16 g, and ±2000 deg/s, respectively. The static error of the sensors is less than 1 deg for all roll, pitch, and yaw angles. The AXIS NEURON software provided by Noitom LTD was used for setting the acquisition features, calibration of the sensors, data capturing, validation of the recorded data, and for exporting the files to different formats.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Example pictures of one female and one male actor wearing the IMU suits used for the capture rounds S1, S2, I1, and I2.

https://doi.org/10.1371/journal.pone.0253157.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Placement of IMU sensors on the body.

https://doi.org/10.1371/journal.pone.0253157.g003

2.3.1 Data capture sequence “F”.

This sequence was captured using the 67 MoSh motion capture marker layout [15]. Subjects wore tight-fitting minimal clothing in order to minimize marker movement relative to the body. The markers were attached to the actors’ skin and clothes using double-sided tape. In addition to the motion capturing, video material was recorded using two stationary Grasshopper cameras and the rear cameras of two hand-held iPhones 7. For details on the synchronization of the motion capture system and the stationary cameras, see Synchronization Section. Participants performed the actions separated by a rest A-pose. The motivation for this capture round was to obtain accurate full skeletal (pose) information and frame-by-frame body shape parameters without any artefacts imposed by clothing. Therefore, this round is suitable for 2D or 3D pose estimation and tracking, and 3D shape reconstruction.

2.3.2 Data capture sequences “S1” and “S2”.

For these two sequences, subjects wore the IMU bodysuit and a reduced optical motion markerset layout of 12 motion capture markers that were attached to their body (4 markers placed on the head, 2 on each ankle and 2 on each wrist). In addition, the actions were recorded using synchronized computer vision cameras (see Synchronization section), and iPhone 7 rear cameras. In “S1” there was a rest A-pose between the actions, whereas in “S2” there was a natural transition between the performed actions. The reason for choosing a small motion capture markerset was that it provides sparse, but accurate data for some of the main end-effectors including the head, wrists, and ankles, and at the same time allows participants to wear natural clothing.

2.3.3 Data capture sequences “I1” and “I2”.

These two sequences were captured with participants wearing the IMU suit under their normal clothing. Additionally, video material was recorded using the hand-held iPhone 7 and stationary Grasshopper video cameras. Motions in “I1” are separated by a rest A-pose, whereas there is a natural transition between the actions in “I2”.

2.4.1 Motion capture data.

A cubic polynomial gap filling was automatically done in the QTM software for gaps of less than or equal to 5 frames. The trajectories were then labelled manually using the integrated trajectory identification tool. The resulting labelled trajectories were then exported to a C3D format.

2.4.2 Video data.

Each data capture sequence was recorded in one piece, without stopping the recording between the different actions. Therefore, the recorded sequences by the computer vision cameras were manually trimmed into individual single actions and the time-stamps (frame numbers) of start and end of each action were exported. Based on the time-stamps, the corresponding synchronized motion capture and IMU data were also trimmed into the same individual single actions.

2.4.3 IMU data.

The original IMU data stored in calculation file format (.calc) were re-organized and converted into MATLAB .mat files to get the data in a more readable structure.

2.5.1 Motion capture cameras.

The calibration of the motion capture cameras was done before each recording session following the measurement protocol in the Qualisys Track Manager software [16]. The software allows computing the orientation and position of each camera in order to track and perform calculations on the 2D data for conversion into 3D data. The average residual error of the calibration was kept below 0.8 mm and the calibration was repeated if this threshold was not met.

2.5.2 Video cameras.

To compute the intrinsic parameters of the Grasshopper computer vision cameras and lens distortion parameters, the MATLAB Single Camera Calibrator [17–19] was used, where focal length (), optical center (), skew coefficient (), and radial distortion () are estimated for each camera. The average re-projection error was kept to less than 0.2 pixel, and the calibration was repeated for higher error values. No calibration was performed for the iPhone cameras.

2.5.3 IMU device.

2.5.3.1 Model posture calibration: Before starting each session, a four-step calibration process was required to calibrate the actor’s posture. The four-step calibration process is performed by the actor posing in a steady pose, A pose, T pose, and S pose.

2.5.3.2 Neuron calibration: IMU sensors might accumulate some calculation errors over time. This usually causes posture computation problems such as drifting. Therefore, each individual IMU sensor should be calibrated after some time of usage. However, to make sure that recordings are accurate enough, we calibrated the sensors before collecting data from each subject following the Noitom Axis Neuron user manual [20].

2.5.4 Motion capture and video cross-calibration.

To cross-calibrate the motion capture system with the two Grasshopper computer vision cameras, the location of world points was aligned onto the camera coordinates. For that, the extrinsic parameters which represent the rotation R ∈ SO(3) and translation from the motion capture system’s coordinate system (world coordinates) to the camera coordinates were estimated using the semi-automated method proposed by Sigal et al. [9]. The trajectory of a single moving marker was recorded by the synchronized motion capture and video cameras for > 2000 frames. Given the recorded 3D positions of the marker in motion capture coordinates as world points and the 2D positions of the marker in the camera frame as image points, the problem of finding the best 2D projection can be formulated as a Perspective-n-Point (PnP) problem where the Perspective-Three-Point (P3P) algorithm [21] is used to minimize the re-projection error as follows: (1) where n is the frame number, N > 2000 is the total number of recorded frames, f is the projection function and K is the set of camera intrinsic and lens distortion parameters. The 2D position of the single marker was located using a Hough circle transform [22] and double-checked manually frame-by-frame.

To validate the computed extrinsic parameters, the parameters were evaluated on a separate single marker capture session. The average re-projection RMS error on this test run was around 0.8 cm. Synchronization and calibration were additionally validated by careful visual inspection of the accuracy of overlaid joint and mesh positions on the video data for a random selection of multiple rounds. The examples shown in Fig 4 are representative for the quality of the whole database.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Front and side view of aligned video frame, joint locations, and estimated body mesh (computed by MoSh++) for one female and male participant.

https://doi.org/10.1371/journal.pone.0253157.g004

2.6.1 Motion capture and video data.

For the data capture sequences “F”, “S1”, and “S2”, the motion capture system and the cameras had to be time-synchronized. In our setup, the video cameras were triggered by the synchronization signal from the QTM software of the motion capture system through ethernet. Due to the frame rate limits in the video cameras, the synchronization frequency was divided by 4 which reduced the video capture frame rate to 30 fps. The iPhone cameras were not synchronized with the motion capture cameras.

2.6.2 Motion capture and IMU data.

To use both IMU and motion capture data in the sequences “S1” and “S2” in a data fusion scenario, these modalities needed to be synchronized in frame. To this end, the cross-correlation between the z-axis location of ankles was used which was pre-computed in these two modalities. The two coordinate systems were not aligned, however, the differences between the orientation of the two z axes are negligible: the z axis of the IMU coordinate system is oriented towards gravity, while z axis in motion capture coordinate system is perpendicular to the floor. Because the motion capture system was synchronized with the video cameras, we additionally obtained synchronized IMU and video data.

Suppose and are the z component of tracked position of joint j at time-step n recovered by the motion capture and IMU systems, respectively (we are using the 3D positions provided by the IMU software instead of double-integrating over accelerations). The synchronization parameters, temporal scale α and temporal shift τ, are found by maximizing: (2) where the integral is the cross-correlation between and scaled version of . The optimal parameters, by which the highest peak in cross-correlation is achieved, were found using an exhaustive search for 0.9 ≤ α ≤ 1.1 (search step-size = 0.001) and −200 ≤ τ ≤ 200. The second term in the summation in Eq 2 was evaluated using spline interpolation. We found α = 1 for all samples meaning that there was no scaling. To ensure that the optimized parameters were robust, we normalized the resulted cross-correlation (Eq 2) to the maximum of 1 and only accepted those where the distance between first and second peak was higher than 0.3. Only in 3 out of all “S1” and “S2” rounds the parameters got rejected and the synchronization was repeated. Finally, we did a visual inspection of all accepted samples.

2.7.1 Visual3D software.

Visual3D is a biomechanics analysis software for 3D motion capture data [14]. In our Visual3D pipeline, the pelvic segment was created using CODA [23] and the hip joint positions were estimated using Bell and Brand’s hip joint center regression [24, 25]. The upper body parts were estimated using the Golem/Plug-in Gait Upper Extremity model [26]. The resulting skeleton at each frame is represented by 20 joints in two different formats: 1) in local joint transformations, that is the orientation and translation of each joint relative to the coordinate system of its parent joint in the kinematic tree, and 2) as global 3D joint locations.

2.7.2 MoSh++.

MoSh++ is an approach which estimates the body shape, pose, and soft tissue deformation directly from motion capture data [7]. Body shape and pose are represented using the rigged body model SMPL [4] where the pose is defined by joint angles and shape is specified by shape blend shapes. MoSh++ achieves lower errors compared to the original MoSh framework [15], which used the SCAPE body model [27]. It uses a generative inference approach whereby the SMPL body shape and pose parameters are optimized to minimize reconstruction errors. The skeletal joint locations are computed using a linear regression function of mesh vertices. The estimated SMPL body is extended by adding dynamic blend shapes using the dynamic shape space of DMPL to simulate soft tissue deformations. Each frame in the “MoSh-ed” representation includes 16 SMPL shape coefficients, 8 DMPL dynamic soft-tissue coefficients, and 66 SMPL pose coefficients as joint angles (21 joints + 1 root). MoSh-ed data of our motion capture recordings was computed in collaboration with the authors of AMASS [7].

The main difference between MoSh++ and Visual3D is that the models are optimized for different applications. MoSh++ is a better choice for character animation, and pose estimation and tracking, whereas Visual3D is preferred for gait analyses and biomechanics. MoSh++, on the one hand, can provide an estimate of joint transformations for all joints even if marker occlusion occurs. However, the estimated joint locations can be noisy when occlusions occur and the error may propagate to other joints. This is because MoSh++ uses distributed information by regressing from the inferred body mesh to the skeleton joints. For character animations, however, precise joint locations are often not important. For gait analysis and biomechanics applications, on the other hand, an accurate estimation of joint locations is crucial. Visual3D achieves this by doing the computations locally where each joint location is computed only from the surrounding markers. The only drawback of Visual3D representation compared to MoSh++ is that the joints cannot be computed at all if one of contributing markers is occluded. In the database, we indicated the time-stamps of the frames where such occlusions occurred.