Fig 1.
Analysis and modeling of interactions between schooling fish in a closed-loop virtual reality setup.
(A) Closed-loop virtual reality setup used to measure and analyze in real time the interactions between fish within a school and their effects on individual behavior. (B) The virtual reality setup makes it possible to track in real time the 3D movements of a fish moving freely in a hemispheric tank. (C) These data then feed a mathematical model controlling the behavior and movement of one or several realistic virtual fish of the same species, the 3D anamorphic image of which being projected onto the hemispherical screen constituting the tank wall.
Fig 2.
Operating modes of the 3D tracking.
Detail of the tracking process for the same frame set using D435 camera computed depth map (left) and classical stereo-vision computation from left and right infrared frames (right).
Fig 3.
Disparities between the depth map and the stereo vision tracking modes.
(A) Disparities computed with D435 camera (blue) and standard stereovision (orange) for 10 min of tracking data. (B) Raw depth for a 1000-frame sample in the same experiment.
Fig 4.
Graphical User Interface (GUI) of the VR system.
Trajectory simulator performing (A) circle and (B) rhodonea trajectories. The GUI allows to visualize in real time the trajectory of the real and VR fish (in the xy and xz planes), and to modify instantaneously the parameters of the model driving the VR fish.
Fig 5.
(A) Real Rummy-nose tetra, (B) 3D animated model, and (C) anamorphic rendering of the virtual fish projected onto the acrylic bowl by the rendering application according to the 3D position of the real fish.
Table 1.
Speed, swimming depth, and distance from the wall of the virtual fish in each experimental condition.
Fig 6.
Comparison of fish responses to a virtual conspecific versus a virtual sphere.
(A) Probability density function (PDF) of the distance between the real fish and the virtual fish (red, mean 9.3 cm 7.0 SD) and between the real fish and a virtual sphere (black, mean 15.3 cm
7.2 SD). Vertical lines indicate the mean value of each distribution. (B) PDF of the relative alignment of the real fish with the virtual fish (red) and with the virtual sphere (black). (C) The maximum correlation with the virtual fish is
, reached at
+ nT, and
with the virtual sphere, reached at
, with T = 6.28 s,
Fig 7.
Real and virtual fish trajectories.
Trajectories of the real fish (red) and the virtual fish (blue) over one minute when the virtual fish swims at a constant speed at depth z = 5 cm. (A–C) the virtual fish follows horizontal circular paths at a fixed distance from the wall, cm: (A)
cm/s, (B)
cm/s, and (C)
cm/s. (D) Circular paths of smaller radius:
cm/s,
cm. (E) Rose curves given by
, with
cm, n = 3, d = 5, and (F) with n = 3, d = 1.
Fig 8.
A 3-second sequence showing the real fish reacting to the anamorphic projection of the virtual conspecific.
The appearance and size of the virtual fish are adapted in real time so that the real fish (indicated by a white arrow pointing to its position) swimming with the edge of the bowl on its left has the illusion that the virtual fish is swimming nearby and to its right.
Fig 9.
Impact of virtual fish’s swimming speed on real fish’s behavior.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the swimming speed of the virtual fish: , 10, and 15 cm/s. The virtual fish (VF) swims at constant depth and distance to the wall in all cases,
cm and
cm respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
Fig 10.
Impact of virtual fish’s swimming depth on real fish’s behavior.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the swimming depth of the virtual fish: , 5, and 8 cm/s. The virtual fish (VF) swims at a constant speed
cm/s in all cases, and at a distance from the wall
, 5.4, and 5 cm for each depth, respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
Fig 11.
Impact of virtual fish’s distance to the wall on real fish’s behavior.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the distance to the wall of the virtual fish: , 5.4, and 10.4 cm. The virtual fish (VF) swims at constant speed and depth in all cases,
cm/s and
cm respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
Fig 12.
Behavior of real fish when following the virtual one along rhodonea trajectories.
(AD) Rhodonea trajectories of three petals. Red line: trajectory of the virtual fish. Density map: positions of the real fish. (A) Rose 1, wide petals (n = 3, d = 5), (D) Rose 2, narrow petals (n = 3, d = 1). (B) Probability density function (PDF) of the distance between fish along Rose 1 (magenta) and Rose 2 (brown) trajectories. (C) PDF of the speed of the real fish in each rose. Vertical blue dashed line: speed of the virtual fish. (E) PDF of the depth of the real fish in each rose. Vertical blue dashed line: depth of the virtual fish. (F) PDF of the angle with which the virtual fish is perceived by the real one, multiplied by the sign of the virtual fish direction of rotation (–1 for clockwise, + 1 for counterclockwise).