https://orcid.org/0000-0002-3062-4215
Correction
20 Feb 2026: Sanchez S, Escobedo R, Bastien R, Lenseigne B, Denis A, et al. (2026) Correction: An open-source closed-loop Virtual Reality system to investigate social interactions and collective behavior in fish. PLOS ONE 21(2): e0343509. https://doi.org/10.1371/journal.pone.0343509 View correction
Figures
Abstract
This study introduces a low-cost, open-source, immersive Virtual Reality (VR) system designed to investigate real-time social interactions and collective behaviors in fish. Understanding collective animal behaviors, such as schooling in fish, presents significant observational challenges due to rapid and complex interactions. To address these difficulties, we developed an innovative closed-loop VR environment allowing precise control and measurement of interactions between real and virtual fish. This setup incorporates high-speed 3D tracking, real-time visual feedback, and automated data processing, creating realistic and interactive scenarios. We present experimental results, obtained with rummy-nose tetras (Hemigrammus rhodostomus), showing that freely moving real fish consistently adjusted their speed, depth, and spatial positioning to follow virtual fish effectively. Fish matched moderate virtual speeds comfortably, struggled slightly at higher speeds, and actively maintained vertical positioning to sustain group cohesion. The implementation of this VR system provides critical insights into sensory and cognitive processes underlying collective movement and decision-making in fish. By revealing both flexibility and constraints in fish behavior, our findings underscore the potential of VR technology to study social interactions in controlled yet realistic settings. Ultimately, this VR system advances our understanding of collective animal behavior, providing a foundation for cross-species studies linking neural processes, behavioral interactions, and group dynamics.
Citation: Sanchez S, Escobedo R, Bastien R, Lenseigne B, Denis A, Moreau M, et al. (2026) An open-source closed-loop Virtual Reality system to investigate social interactions and collective behavior in fish. PLoS One 21(1): e0339909. https://doi.org/10.1371/journal.pone.0339909
Editor: Shicheng Li, KTH Royal Institute of Technology: Kungliga Tekniska Hogskolan, SWEDEN
Received: July 4, 2025; Accepted: December 12, 2025; Published: January 21, 2026
Copyright: © 2026 Sanchez et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data from all the experiments are available from the figshare repository (https://doi.org/10.6084/m9.figshare.29097605). The codes for the 3D tracking, the trajectory, and the rendering of virtual fish are available from the figshare repositories (https://doi.org/10.6084/m9.figshare.30188836.v1 (Acquisition and 3D tracking software)) (https://doi.org/10.6084/m9.figshare.30188842.v1 (Trajectory simulator)) (https://doi.org/10.6084/m9.figshare.30188845.v1 (3D anamorphosis rendering software)) (https://doi.org/10.6084/m9.figshare.30188839.v1 (Calibration routines)).
Funding: Agence Nationale de la Recherche (ANR-20-CE45-0006-1) Spanish State Research Agency, AEI-Agencia Estatal de Investigación grant PID2020-115088RB-I00.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
Understanding collective animal behavior has long been a fundamental goal in behavioral biology and neuroscience [1,2]. Fish schools, bird flocks, and insect swarms exhibit remarkable coordination that emerges from local interactions among individuals [3,4]. These interactions are crucial in shaping movement dynamics, predator avoidance, and decision-making processes in groups [5,6]. However, studying these interactions in natural conditions presents methodological challenges due to the complexity of tracking individual behaviors and measuring their real-time influence on the group. These constraints explain the need to carry out studies in controlled environments to decipher these interactions and cognitive processes involved at the individual scale [7–10]. Traditional observational methods and open-loop experimental setups often fail to capture the bidirectional influence between individuals and their environment. To address these limitations, virtual reality (VR) systems have emerged as powerful tools to investigate animal perception and behavior, allowing researchers to manipulate environmental variables and control social stimuli with high precision [11].
Recent advances in VR technology have enabled the study of visual-motor feedback in fish and other animals under open-loop and closed-loop conditions. For instance, VR studies on fruit flies have provided insights into how insects process visual motion and adjust their flight paths in response to virtual stimuli [12]. VR systems have also been employed to examine goal-driven navigation in zebrafish larvae by linking their tail movements to virtual environments [13]. Similarly, high-resolution VR platforms have been used to analyze neuronal activity in head-restrained adult zebrafish responding to dynamic visual stimuli [14]. Researchers have explored how animals respond to virtual predators, social cues, and environmental changes. For example, using a VR system for studying wild-caught coral reef fish, it has been demonstrated that these fish could distinguish between virtual predators, conspecifics, and non-threatening species [15]. In another study, [16] found that animals in VR could make complex navigational decisions based on visual cues, shedding light on how sensory information is integrated in the brain. Additionally, VR has been instrumental in studying threat avoidance in humans through the VRThreat toolkit [17]. This software simulates realistic danger scenarios, allowing researchers to examine human defensive behaviors in high-stakes situations. These studies have demonstrated that VR can be effectively integrated with neurophysiological techniques, allowing precise manipulation of sensory inputs and real-time tracking of behavioral responses [18].
Building on these foundations, recent VR-based research has provided new insights into how fish process social cues and make decisions in collective contexts. It has been shown that zebrafish do not simply average the directions of neighbors but can explicitly choose which conspecific to follow, producing bifurcations in their trajectories when confronted with multiple leaders [19]. Other studies demonstrated that larval zebrafish transform simple retinal occupancy statistics into motor responses, with a developmental transition from repulsion to attraction as cohesive shoals form [20]. Immersive VR has been used to reveal that juvenile zebrafish follow conspecifics according to a proportional-derivative pursuit law, robustly capturing pursuit dynamics across single- and multi-target scenarios [21]. Earlier work already highlighted the potential of immersive VR to study startle responses [22], and further showed how larvae chain discrete swim bouts into prey-capture sequences under closed-loop visual feedback [23]. Technical developments such as panoramic projection systems have expanded the capacity for wide-field, high-resolution stimulation [24]. More recently, closed-loop VR combined with two-photon calcium imaging revealed that some zebrafish generate neural ensembles encoding prediction errors, enabling them to escape more efficiently—evidence that vertebrates integrate both reward maximization and surprise minimization strategies when making escape decisions [25]. Together, these studies illustrate how VR provides a unique window into the sensory-motor and cognitive mechanisms that govern collective behavior, from elementary visuomotor transformations to higher-order decision-making.
In this article, we introduce an innovative and low-cost closed-loop VR system specifically designed to study real-time social interactions among fish. Our objective is to demonstrate how this technology allows precise control and measurement of behaviors, providing new insights into the mechanisms driving collective movement. This experimental platform integrates high-speed tracking, real-time visual feedback, and automated data processing to measure how freely swimming fish dynamically respond to computer-controlled virtual conspecifics. By simulating realistic social interactions within a VR environment, our system allows us to quantify the influence of local neighbor interactions on collective movement and decision-making. Unlike traditional open-loop VR systems that impose predetermined stimuli, this closed-loop design continuously updates the visual stimuli based on real-time behavioral feedback, ensuring a dynamic and interactive experimental paradigm [12].
One of the key advantages of this system is its ability to simulate controlled yet biologically relevant interactions that influence group coordination. Previous research has shown that synchronized movements involve selective social interactions between fish and that vision plays a crucial role in behavioral decisions [26–29]. By systematically varying the behavioral parameters of virtual conspecifics, we can investigate how sensory feedback modulates movement decisions, spatial alignment, behavioral contagion, and leadership emergence within groups.
We detail the technical components, validate our closed-loop VR system through controlled experiments, and present preliminary findings obtained with Rummy-nose tetras (Hemigrammus rhodostomus). This species has a strong tendency to school, with an intermittent swimming mode characterized by alternating bursts and coasting. While our experiments focus on interactions involving one real fish and one virtual fish, the system can easily include several virtual fish and adapt to other species by modifying the 3D shape and color of the virtual fish and their behavioral and swimming patterns. Ultimately, our VR system aims to clarify how freely moving fish perceive and integrate local social cues, enhancing our understanding of collective behaviors at both neural and behavioral levels.
2 Materials and methods
2.1 Experimental setup
The experimental setup is similar to the one used in FishVR [11]. The main differences are a larger bowl to accommodate adult rummy-nose tetras (3 to 4 cm in length) and a single depth camera as a capture device (Fig 1). The acrylic bowl is made by Immersive Display Group, London, UK. It is a subsection of a sphere filled with 15 L of water, resulting in a depth of 14.6 cm and a diameter of 52.2 cm at the water surface. Infrared light for tracking is provided by 8 IR lamps (wavelength of 850 nm) from underneath the bowl. The tracking camera is an Intel RealSense D435 depth camera [30]. It is positioned above the water surface, at 47.9 cm above the center of the water surface. IR filters are applied to the two IR sub-cameras of the camera. A single computer (Dell Precision 3640 Tower Workstation PC with a NVIDIA RTX 3070 graphics card) does all the real-time computations: tracking, trajectory simulations, virtual environment computation, and display. A DLP LED video projector (OPTOMA WUXGA) illuminates and displays the virtual environment on the bowl from beneath.
(A) Closed-loop virtual reality setup used to measure and analyze in real time the interactions between fish within a school and their effects on individual behavior. (B) The virtual reality setup makes it possible to track in real time the 3D movements of a fish moving freely in a hemispheric tank. (C) These data then feed a mathematical model controlling the behavior and movement of one or several realistic virtual fish of the same species, the 3D anamorphic image of which being projected onto the hemispherical screen constituting the tank wall.
Closed-loop interactions between real fish and virtual ones require 3 separate software elements:
- Acquisition and 3D tracking software that gets frames from the Intel RealSense D435 camera and tracks fish positions from them.
- Trajectory simulator that simulates virtual fish behaviors, with or without interactions with real fish.
- Rendering software that displays virtual fish according to the simulated positions and the real fish position (to perform anamorphosis rendering).
Network UDP requests transfer positions of fish (real and virtual ones) from one software to another.
2.2 Images acquisition and real-time 3D tracking of fish
2.2.1 Images acquisition.
In our experimental setup, we use an Intel RealSense D435 camera to acquire images of the real fish freely moving in the bowl. This 3D camera is autocalibrated and features two IR cameras, one RGB camera, and a laser projector. For each camera, it provides a video stream and a computed depth map. The acquisition software is based on RealSense SDK [31] to control the D435 camera and to acquire, at a rate of 30 to 90 frames per second, both IR frames (left and right) and one depth map aligned with the left IR frame.
2.2.2 3D tracking.
The 3D tracking software is written in Python and can operate in two different modes, depending on which camera provides the information it uses (Fig 2):
- Using directly the depth map. This offers the advantage of directly obtaining the real-world z coordinate of any tracked pixel (i,j) in the depth map image (the RealSense SDK automatically converts any pixel (i, j) into real-world (x, y, z) coordinates).
- Using the standard stereo vision from both the IR left and right frames. Each frame is processed with OpenCV [32] routines (mainly masking and background subtraction routines) to isolate real fish pixels.
Detail of the tracking process for the same frame set using D435 camera computed depth map (left) and classical stereo-vision computation from left and right infrared frames (right).
The advantage of using a depth camera is that the frame processing required to compute an accurate depth map is performed directly on its hardware, and is therefore theoretically much faster than external computation. Our first data sets showed surprising regular bands in the depth distribution of real fish positions, as if the fish only swam in discrete regular depths. Further investigations have shown that some disparities (disparity refers to the distance (in pixels) between two corresponding points in the left and right frames of a stereo pair) from D435 seem to be rounded to the nearest integer value, as shown in Fig 3A. Implementing our own stereovision computation from both IR frames solved the depth distribution problem (cf. Fig 3A). But, as shown in Fig 3B, raw depth data is much noisier with stereo vision than data from the depth camera. To mitigate this noise, computed depth using stereovision is smoothed with a Kalman filter (as implemented in Python).
(A) Disparities computed with D435 camera (blue) and standard stereovision (orange) for 10 min of tracking data. (B) Raw depth for a 1000-frame sample in the same experiment.
Tracking of individual fish is based on the raw data provided by the RealSense camera. Input is therefore a 16-bit depth map with a resolution 848 480 for immediate 3D tracking or two 10-bit 848
480 IR frames for stereo vision. Masking is performed on each data frame so that only targets within the bowl and within the desired depth ranges are considered. From the depth map, filtered data is delivered to the tracker as 3D points or, from the IR frames, as 2D points. Tracking is performed using a Random Matrix Tracker [33,34]. This specific kind of filter is a derivation of the well-known Kalman filter, most suited to the tracking of targets represented as a point cloud. It is akin to a traditional Kalman filter with the addition of an extension matrix that represents the spread of the point cloud as the tracking goes on. Each target/fish has its own tracker attached.
The target is represented by a state vector that contains the target’s dynamics up to a specific order (order two in this presentation: position and speed in 2D or 3D): , at time step k, where dot denotes derivation with respect to time. The spread is a Symmetric Positive Definite matrix: the covariance matrix
of the target. The measurement vectors
from multiple measurements reflect the position and spread of the target at the time step k. The tracking is performed in the reference frame of the camera image; therefore, the measurement matrix is :
. The measurement of the point cloud at time k is described as nk points with a covariance
:
. Since the point cloud extension
covers the spread of the measurements, we neglect more specific sensor noise. With these settings, the prediction equations become:
where is a constant related to the target’s agility (higher
means less agile target), T is the time prediction interval, F is the process model that assumes constant speed, and P, Q, and R are the state estimation, process noise, and measurement noise covariance matrices, respectively, as in a regular Kalman filter. The estimator next undergoes an update step based on measurements.
We start with the estimate of the measurement error variance considering the target spread and the sensor noise ( is an additional scaling factor),
then the innovation covariance and Filter gain,
and then the remaining steps, which closely follow the conventional Kalman update for the state vector and estimation covariance:
where is the barycenter of the measurement point cloud.
Next, the extension update includes the covariances induced by target dynamics and by measurement noise. The latter is computed explicitly as the covariance matrix of the measurement error with respect to the estimated state,
and similarly for the measurement covariance,
We now have the necessary building blocks to compute the extension contribution of target’s dynamics and measurement noise contribution:
Finally, once we have set + nk, the final state extension is written as a weighted sum of the different contributions:
The tracker is initialized with the knowledge of the number of fish to track. We assume that the first dataframe contains each fish visible, preferably separated from each other. The initialization procedure aims at providing a realistic initial estimate of the position and spread of each fish.
Therefore, the following procedure is repeated for each fish:
- Select a point from the data frame and initialize the tracker.
- Select a few close points below on a Cartesian distance Threshold.
- Repeat until convergence of the state estimate:
- Perform a measurement update run of the tracker based on the selected points.
- Define a search area with center
and covariance
.
- Add points to the fish based on the Mahalanobis distance:
with a threshold based on desired Chi-squared likelihood. - Remove the selected points from the data frame.
Once these steps are performed, the normal tracking loop takes place. Firstly, a prediction step of the tracker is performed for each target. Next, a new data frame is brought in. Data points are assigned to each target considering the predicted new state and extension
of each
fish. We use a nearest neighbor strategy based on the Mahalanobis distance between the point and each candidate target/fish. Furthermore, a threshold is applied to a maximal acceptable distance based on a reasonable size for a fish. Once measurements have been attributed to each target, the update steps as described above can take place, leading to a corrected state
and extension
of each fish. Fish clusters, occlusions, and other artifacts might, however, lead to scenarios where, for some time steps, no measurements are found for one or several targets. Tracking continues for these targets using the prediction equations only. If the situation persists for too long, a full reset of the filter is performed following the procedure used for initialization. Previously existing targets are attached to the closest new target in terms of Cartesian distance, and normal tracking resumes.
The tracker operates in real time at a frame rate of 30 to 90 frames per second. In our experiments, both process and noise covariances are set as identically scaled identity matrices. For each target, filter settings are as follows: reactivity is set to , the innovation scale factor is
, and
.
2.3 Trajectories simulator software
A C++ Qt [35] application simulates the trajectories of virtual fish. The application collects real fish 3D position and orientation from UDP requests sent by the 3D tracking software (at 30 to 90 requests per second) and, for each request, computes in real-time new positions and orientations of each virtual fish according to its simulated behavior. All orientations and positions are immediately sent via the UDP request to the rendering application. In the context of this paper, two behaviors are considered: passive (open-loop) circular trajectories and rhodonea trajectories (see Fig 4). However, our system also allows for the position and heading of the virtual fish to be controlled in closed-loop by a behavioral model (like the data-driven model of [10] for rummy-nose tetra) exploiting the current position and speed of the real fish. In particular, the parameters of such a behavioral model (comfort speed and depth, strength of the attraction and alignment interactions with the real fish...) can be modified in real-time in the Graphical User Interface (GUI) of our VR system (see Fig 4).
Trajectory simulator performing (A) circle and (B) rhodonea trajectories. The GUI allows to visualize in real time the trajectory of the real and VR fish (in the xy and xz planes), and to modify instantaneously the parameters of the model driving the VR fish.
Circular trajectories. Circular trajectories represent a simple behavior in which virtual fish swim at constant speed and constant depth in circles centered within the bowl. The application allows for dynamic changes to the speed, depth, and radius of the swimming circle. Virtual fish position is given by
where is the discrete time,
is the angular speed, R is the radius of the circle, v is the constant linear speed along the circle, and z0 is the fixed swimming depth. See Fig 4A.
Rhodonea trajectories. A rhodonea (also known as a mathematical rose) is a flat curve that has the shape of a flower with several petals (Fig 4B). We consider virtual fish performing a rhodonea curve centered within the bowl at a constant angular speed and constant depth z0. As before, the application allows adjustments of the maximum radius R of the petals and of the number of petals, determined by
: the number of petals is
if
is odd, and
if
is even. The discrete position of the virtual fish at time step tn is given by
2.4 VR-fish rendering
A C# Unity [36] application renders the virtual environment and virtual fish. From UDP requests sent by the trajectory simulator (at 30 to 90 requests per second), the rendering application gathers positions and orientations of both real fish and virtual ones. The real fish’s position and orientation allow for the computation of anamorphic perspective, while the virtual fish’s positions and orientations enable them to be rendered in accurate places in the bowl relative to the real fish. The rendering of virtual fish uses a realistic animated model of the Rummy-nose tetra (Fig 5). It is nevertheless possible to assess different fish species by substituting the 3D model with an appropriate one or, in the context of investigating non-social stimuli, by employing geometric shapes (e.g., spheres or cubes) in place of fish models.
(A) Real Rummy-nose tetra, (B) 3D animated model, and (C) anamorphic rendering of the virtual fish projected onto the acrylic bowl by the rendering application according to the 3D position of the real fish.
2.4.1 Calibration and projection mapping.
The basis of the Fish VR system is to project a rendering of a virtual environment on a surface, taking into account the perspective of individuals (anamorphosis) dynamically, using real-time tracking [11]. To render this projection of the fish’s point of view accurately, the system’s calibration is critical. This means that the position of the projection screen needs to be known exactly relative to the position of the fish. In the following, the 3D tracker’s coordinate system serves as the reference space for all measurements.
The process to calibrate the Fish VR requires one to get the following elements:
- the position and geometry of the projection screen, i.e., the bowl, in the reference space,
- the mapping between the projector’s pixels and their corresponding 3D locations on the bowl’s surface, and
- the position of the water surface in the reference space.
Using these calibration data, combined with real-time tracking of the fish, the system can dynamically calculate and render the appropriate virtual scene onto the bowl, pixel by pixel, simulating the view from the fish’s perspective.
To simplify calibration, the z-axis of the camera is aligned beforehand with the direction of gravity. This is not a necessary step, as the direction of the camera in real space can be recovered from the direction of the water surface. This simplification is allowed by the use of a single camera, compared to previous Fish VR setups [11], and will be assumed for the rest of the calibration.
3D scan of the bowl. The first step of calibration is to recover the position and geometry of the 3D bowl in space. The RealSense D435 is a stereoscopic depth camera that can offer a 3D mesh of the photographed scene. The accuracy of the stereoscopic camera can be increased when used with the embedded projecting laser. The scanning process consists of 5 steps:
- Ten 3D images of the bowl are acquired.
- The images are thresholded by distance to remove objects such as the table holding the bowl.
- The images are then averaged to reduce the noise.
- This 3D map is then directly fitted with a sphere.
- A 3D image of this fitted sphere is synthesized 3D image from the point of view of the camera.
The geometry and position of the bowl are then acquired, namely, the position of the center of the sphere relative to the camera, and the radius of the sphere
cm.
Projection mapping on the sphere. The second step is to establish a mapping that links each pixel coordinate in the projector’s output image to its corresponding 3D point on the surface of the calibrated sphere (the bowl). Each given projector pixel coordinate, , should be associated with the spatial position of its projection on the bowl,
.
Mathematical description of the mapping. To simplify the calculations in this section, we consider the referential where the bowl is at position , and the radius of the bowl is
such as:
The direction of a light ray coming out of the pixel of the projected image is given by
where ,
, and
and
is the direction of the projected image center. Here,
and
are the differences of angle given by 2 consecutive pixels in the horizontal and vertical directions. Pixels are assumed to be square, so
.
Image dimensions are usually even, so that the center of the image is not exactly on one pixel. Using a Full-HD projection (), we consider the range of horizontal pixels
to be [–959.5,959.5], while the range of vertical pixels
is [–539.5,539.5].
The 3D positions in space of each pixel
on the projection screen are defined as:
As these points are distributed on the bowl, we have for all ij. The points are related to the position of the projector by the direction of the associated light ray
,
where dij is the distance between the projector and the point. Then,
so dij is given by a second degree equation, whose physically relevant solution corresponds to the intersection point closer to the projector (assuming projection from outside the sphere):
Performing the mapping requires solving Eq (6) for all the points of the projected image.
There are 6 unknown parameters: the three components of the position of the projector , the translation of the sphere relative to the center of the image,
and
, and the projection scaling
.
Gathering dataset for the mapping. The positions of the projected pixels are measured with the camera using a structured light approach involving scanlines. The process is as follows:
- A mask is generated to identify where the projection hits the bowl. A circular mask on the image by the projection of a fully white image;
- The background subtractor is initialized with the projection of a black image. Multiple images are acquired and are then used to set up a background subtractor (using the MOG2 algorithm);
- Images are captured with a projection of one line of one pixel every 10 pixels, horizontally, then vertically;
- The background subtractor then the mask is applied to the images
- For each combination of a horizontal line (corresponding to iP) and a vertical line (corresponding to
), the two corresponding images are multiplied. The result highlights the single point of intersection;
;
- Their positions on the image are then identified by blob detection;
- Using the synthesized image of the bowl, the real position of each pixel,
, is recovered.
This dataset of correspondences between projected pixels and positions in real space
is used to fit the mapping.
Parameter Fitting. To facilitate the convergence of the non-linear optimization routine used to fit the 6 projector parameters, reasonable initial estimates are generated using the collected data. We proceed as follows:
- An initial guess for the projector’s 3D position is made, typically based on the experimental setup, assuming, e.g., that the projector points towards the sphere’s center from 1.5 m along the z-axis:
(7)
- The centroid of the observed projected points is calculated, yielding the average location in projector pixel coordinates,
, and its corresponding average 3D position,
.
- Potential small corrections
can be added to account for c not being the exact optical center:
(8)
- The angular size of a pixel,
(assuming square pixels,
=
), is estimated using the spatial distribution of points around the central point
. The average spatial distance ds between adjacent measured points near
is calculated, and
is approximated as the angle subtended by ds at the estimated distance from the projector.
These initial estimates provide a starting point for the iterative optimization algorithm, increasing the likelihood of finding an accurate solution for the projector parameters.
Identification of the water surface. With the 3D depth camera, the position of the water surface can be easily identified. Once the bowl is filled with 15 L of water, a cardboard plate is put on the water’s surface. Using the mask and a relevant threshold, the position of the cardboard plate can be easily retrieved on the 3D image. The associated point cloud is then fitted with a plane whose parameters are the distance from the water surface to the camera, and the direction of the normal of the plane (co-linear with gravity).
Mapping. From there, we can create a texture that maps each pixel of the projected image with the normal direction on the bowl. With this map, we need to render for each pixel the direction seen by the fish, taking into account its real spatial position. The process is as follows:
- An omnidirectional camera render is obtained from the position of the fish. 6 cameras pointing in each direction of a cube are combined on a cubemap;
- Using the texture map of the normal of the bowl, the direction seen by the individual is computed;
- Those directions are used to render from the cubemap;
- Finally, the rendering is then done on a texture, that is recorded by a full-HD orthographic camera.
There are additional constraints due to how the fish positions, handled by the CPU, are passed to the shader on the GPU. Data transfer from the CPU to the GPU is slow due to limited memory bandwidth. However, the shader does have access to the position of the object it is attached to. Hence, the rendered texture is carried by the object moving according to the position of the fish. To get this whole process working correctly, the camera rendering to the cube map needs to remain static while all the objects are compensating for the movement of the scene.
The dimensioned plan as well as the list of items necessary for the construction of the main frame of the virtual reality are provided in Supplementary information (see S1 Text, S1 Table and S1 Fig). All software components required for the VR-system—including the acquisition and 3D tracking software, the trajectory simulator that controls virtual fish behavior, and the rendering software that displays virtual fish based on both simulated and real positions—are also provided as open-source tools.
This approach is intended to enable the widest possible adoption of this experimental technique to study animal behavior and analyze social interactions, fostering reusability, adaptability, and collaborative improvement within the scientific community.
2.5 Experimental procedure and data collection
2.5.1 Ethics statement.
Experiments were approved by the Animal Experimentation Ethics Committee C2EA-01 of the Toulouse Biology Research Federation and were performed in an approved fish facility (A3155501) under permit APAFIS#27303-2020090219529069 v8 in agreement with the French legislation. All procedures were designed to minimize stress and handling. Fish were transferred from rearing tanks to the virtual reality setup with minimal manipulation. Each individual was used in only one one-hour experimental session per week. Swimming ability was monitored throughout; fish exhibiting impaired or absent swimming activity were excluded and replaced. No animals were sacrificed during this study.
2.5.2 Study species.
Rummy-nose tetras (Hemigrammus rhodostomus) were purchased from Amazonie Labège in Toulouse, France. Fish were kept in 16 L aquariums on a 12:12 hour, dark:light photoperiod, at 24.9∘C (0.8∘C) and were fed ad libitum with fish flakes. The average body length of the fish used in these experiments is 31 mm.
2.5.3 Experimental procedure.
The experiments reported in this article aim to validate the virtual reality device by demonstrating the effective nature of the interactions between the real fish and virtual fish and by quantifying the behavior of the real fish in response to three behavioral parameters of the virtual fish: speed, swimming depth, and swimming distance to the edge of the experimental bowl.
We first studied the impact on the fish behavior of a non-social control stimulus consisting of a black sphere. The sphere had a diameter of 3 cm, ensuring consistency with the dimensions of the original fish model, and was rendered as an opaque, dark-grey object. In this series of experiments, the same fish was first exposed for 30 minutes to a virtual fish moving along a uniform circular trajectory with constant depth (5 cm), speed (10 cm/s), and distance to the tank wall (10.4 cm). After 30 minutes, the realistic three-dimensional fish model was replaced with a spherical model following the same uniform circular trajectory. The experiment then continued for a further 30 minutes.
Seven conditions were then studied experimentally in which the virtual fish moves along a uniform circular trajectory with constant depth, speed, and distance to the edge of the tank (see Table 1). We also studied two additional conditions in which the virtual fish performs a rhodonea curve at a constant speed and constant depth.
We performed 7 one-hour repetitions with the non-social control stimulus, 10 one-hour repetitions for each condition in which the virtual fish performed circular trajectories, and 5 one-hour repetitions for each condition in which the virtual fish followed a rhodonea trajectory.
At the start of each trial, one fish was randomly removed from one of the breeding aquariums and placed in the hemispherical tank of the VR setup. The fish was then acclimatized to the new environment and lighting conditions during a period of 10 minutes before the trial started. The anamorphic image of the virtual fish is then displayed on the wall of the translucent dome in which the fish swims. The 3D tracking of the fish makes it possible to obtain its 3D position in the hemispherical tank, which feeds the model controlling the projected image in real time and which is also recorded at a frequency of 30 Hz. We then let the fish swim and interact with the virtual fish whose swimming speed, depth, and distance to the wall of the tank are kept at constant values, and only its anamorphic 3D projection changes in real time to take into account the relative position of the real fish compared to that of the virtual fish. Each fish tested under experimental conditions is used at most once a week and in a single trial to prevent any habituation phenomenon.
2.6 Data pre-processing
Before the data obtained during experiments can be analyzed, it must be preprocessed. This step ensures that only the periods of time during which fish are active are retained while eliminating errors from tracking. Thus, time intervals during which the fish remain motionless for at least 4 seconds are excluded from the analysis, as well as intervals when the fish’s speed exceeds 25 cm/s, or when the fish swims in a straight line at a constant speed for at least one second.
Next, the data are filtered to remove any instances where the fish’s position appears outside the bowl. To obtain trajectories with consecutive temporal data, gaps created by data removal are filled using linear interpolation. However, only gaps shorter than one second are filled this way. Beyond one second, this procedure can no longer be used to reconstruct the trajectory, which remains discontinuous at that point.
The trajectories obtained after this step thus consist of multiple segments within which the data remains consecutive. Each segment of a trajectory is then smoothed using a Gaussian kernel smoothing method, which helps reduce noise while preserving the essential features of the data,
where , so that the exponential is close to zero when i = n. The value of the parameter h = 0.5 s is chosen so that the smoothing is as low as possible in order not to average the data, while reducing noise as much as possible.
2.7 Data analyses
2.7.1 Statistical metric for probability distribution comparison.
To quantitatively compare probability distribution functions (PDFs) obtained under different experimental conditions, we used the Hellinger distance as a measure of similarity [37,38]. Given two normalized PDFs, F(x) and G(x), corresponding to the same observable, the square of the Hellinger distance is defined as
This metric ranges from 0 to 1, where if and only if the two distributions are identical, and values approaching 1 indicate distributions with almost non-overlapping support. The second form of the equation provides a straightforward interpretation: it measures the deviation from unity of the scalar product between the square-rooted distributions, viewed as unit vectors in Euclidean space. In practice, a small Hellinger distance (H < 0.2) indicates a high similarity between the distributions, while values greater than 0.2 indicate a meaningful dissimilarity. This approach provides a robust and interpretable quantification of differences between behavioral distributions and has been successfully applied to assess the agreement between empirical fish data and model simulations in collective behavior studies.
2.7.2 Correlation analyses.
In order to quantify how similar the dynamical behaviors of a real fish and a virtual object are, we calculate the cross-correlation function of their velocity vectors and
with the following expression:
Note that the velocity of reference is that of the real fish. As the virtual object often has a programmed periodic behavior (circular motion along circles of fixed radius and depth, or rhodonea of fixed parameters (n,d) and depth), this allows us to obtain a periodic correlation function and simplify the interpretation.
For circular trajectories, the period is simply given by , where
is the perimeter, R is the radius of the circle, and
is the speed of the virtual object. For rhodonea trajectories, however, the perimeter cannot be expressed so directly; further details are provided in the Supplementary Information (S1 Text).
3 Results
This section presents a detailed analysis of the fish’s response to the virtual fish along a variety of experimental conditions, with the aim of assessing the robustness and ecological validity of the setup.
3.1 Control condition with a non-social stimulus
We test a non-social control condition in which fish were exposed to a black sphere of similar size to conspecifics with a constant circular motion of radius R = 10 cm, speed cm/s, and depth z = 5 cm (see S1 Video). Fig 6 shows that, unlike with virtual fish, individuals did not maintain close spatial proximity to the sphere nor exhibit alignment matching, confirming that the following responses observed in our VR assays are specific to social cues rather than generic reactions to moving objects.
(A) Probability density function (PDF) of the distance between the real fish and the virtual fish (red, mean 9.3 cm 7.0 SD) and between the real fish and a virtual sphere (black, mean 15.3 cm
7.2 SD). Vertical lines indicate the mean value of each distribution. (B) PDF of the relative alignment of the real fish with the virtual fish (red) and with the virtual sphere (black). (C) The maximum correlation with the virtual fish is
, reached at
+ nT, and
with the virtual sphere, reached at
, with T = 6.28 s,
Fig 6C shows the cross-correlation of the velocity vector of the real fish, , with that of the virtual object,
+
, for both cases: the virtual fish and the virtual sphere, calculated using Eq 11. The reference speed is that of the real fish. Since the virtual object repeats the same state (position and velocity) with a period
, both correlation functions are periodic with this same period. When the virtual object represents a conspecific, the maximum correlation is nearly three times higher (
) than when it represents a non-social object (
), demonstrating unequivocally that the fish is far more responsive to the image of a conspecific.
The maxima are reached with a slightly negative delay, s + nT and
s + nT, with T = 6.28 s and
. This indicates that the real fish tends to follow the virtual conspecific with a short delay of about 0.13 s, whereas with the virtual sphere the delay is negligible but the correlation remains very low, confirming the reduced attraction to the non-social stimulus.
3.2 Behavioral response of real fish to the circular movement of a virtual fish
Fig 7 shows some of the different cases analyzed in detail in this section. Each panel represents the trajectories of both the real and the virtual fish, in red and blue, respectively, over one minute. Time is represented on the vertical axis and grows upwards, horizontal position is shown in xy-coordinates, and color indicates depth (which varies little across cases). In all cases, the trajectories clearly show that the real fish reacts to the virtual one as if it were a real conspecific, staying close and accompanying its motion throughout the bowl.
Trajectories of the real fish (red) and the virtual fish (blue) over one minute when the virtual fish swims at a constant speed at depth z = 5 cm. (A–C) the virtual fish follows horizontal circular paths at a fixed distance from the wall, cm: (A)
cm/s, (B)
cm/s, and (C)
cm/s. (D) Circular paths of smaller radius:
cm/s,
cm. (E) Rose curves given by
, with
cm, n = 3, d = 5, and (F) with n = 3, d = 1.
While the first three panels (Fig 7ABC) show that the real fish is able to appropriately follow the projected image on the nearest part of the bowl, the fourth panel (Fig 7D) reveals a crucial aspect of the setup: the real fish positions itself between the virtual fish and the edge of the bowl (see also Fig 8 and S2 Video). This means that the fish is responding to the anamorphic image projected on the opposite side of the screen, moreover, through the water, as if the virtual conspecific was perceived as being clearly inside the bowl —a remarkable phenomenon, given the significant distortion introduced by the water.
The appearance and size of the virtual fish are adapted in real time so that the real fish (indicated by a white arrow pointing to its position) swimming with the edge of the bowl on its left has the illusion that the virtual fish is swimming nearby and to its right.
This observation is significant for two reasons. First, it confirms that the optical setup generates a convincing illusion of the presence of a conspecific. Second, it shows that the fish interprets the projected image in a biologically meaningful way: it responds not merely to local visual cues, but to the coherent representation of both the typical size and behavior of a conspecific, maintaining typical distances and relative positioning as it would with a real partner.
We now take a closer look at each condition to see how the fish’s response depends on the different parameters. We impose a constant circular motion on the virtual fish and focus on three parameters characterizing its behavior: the speed , the depth
, and the horizontal distance to the edge of the bowl, denoted by
. The latter refers to the distance to the wall, commonly used in studies of fish behavior in quasi two-dimensional setups. As we will show, real fish tend to preserve their swimming depth, making it more appropriate to characterize the proximity to the bowl’s wall by using the horizontal distance rather than the cylindrical distance from the edge of the sphere.
We perform three sets of experiments, varying one parameter at a time along three values, while keeping the others constant. Note that the intermediate case is the same in the three cases.
3.2.1 Impact of the virtual fish’s swimming speed on the behavior of the real fish.
We first examine the effect of varying the swimming speed. Three swimming speeds of the virtual fish are considered: , 10, and 15 cm/s, while the depth and the distance to the wall are kept constant at
cm and
cm respectively. These cases correspond to the trajectories shown in Figs 7ABC.
In each case, the real fish adjusts well to the virtual fish, with the peak of the probability density function (PDF) of its speed coinciding with the value of the virtual speed for the two slower conditions (Fig 9B and S2 Table). The mean speeds of the real fish are 6.3, 9.4, and 11.7 cm/s, respectively. For the slowest case, the real fish tends to be faster than the VF, while for the highest speed, the PDF peak shifts slightly to the left, around 14 cm/s, and a secondary bump appears near 5 cm/s, indicating occasional much slower responses, which could be due to a loss of contact or to a difficulty to maintain such a high speed.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the swimming speed of the virtual fish: , 10, and 15 cm/s. The virtual fish (VF) swims at constant depth and distance to the wall in all cases,
cm and
cm respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
The distance of the real fish to the virtual one remains consistent across the three cases, with the peak of the distance PDF located at cm (Fig 9A and S3 Table), although the long tail in the three cases shows that fish can be quite distant. Similarly, the real fish maintains a nearly constant depth of approximately
cm, slightly above the virtual fish (Fig 9C and S3 Table), but occasionally falls at
cm.
Figs 9D and 9E show the relative position of the real fish in the reference frame of the virtual fish, placed at the origin and with its velocity vector pointing towards the positive values of the y-axis. Figs 9D corresponds to the xy-plane, while Figs 9E shows the xz-plane. These plots confirm that the real fish stays close to the virtual fish: at low speed, it is slightly in front; at medium speed, it is equally likely to be ahead or behind; and at high speed, it is often behind and slightly closer to the center of the bowl, i.e., farther from the wall. In all cases, the real fish tends to swim slightly above the virtual fish. At high speed, some trajectories extend as deep as 6 cm below the virtual fish.
S2 Fig shows the cross-correlation of the velocity vector of the real fish, , with that of the virtual fish,
+
, for the three tested speeds. For
cm/s and
cm/s, the correlations are high, comparable to those observed in Fig 6C when the fish interacts with the virtual conspecific. Even for
cm/s, the correlation remains well above the values obtained with a virtual sphere. The maximum correlations are
,
, and
, reached respectively at
s + nT1,
s + nT2, and
s + nT3, with
s,
s, and
s.
At cm/s, the real fish closely follows the virtual one, showing a strong correlation. The delay is negligible, which is consistent with the fact that this speed is close to the typical mean speed of rummy-nose tetra [10]. For
cm/s, the virtual fish moves too slowly, and the real fish often overshoots it, as indicated by the positive delay of nearly half a second. For
cm/s, the real fish tend to follow the virtual fish with a delay of more than a quarter of a second, still maintaining a high correlation that reflects consistent following behavior.
In light of these results, the trajectories shown in Fig 7ABC can be interpreted as follows: when the virtual fish swims at 5 cm/s, the real fish is faster and circles around it, moving back and forth as if waiting. When the virtual fish swims at 15 cm/s, the real fish generally follows but tends to shortcut, tracing smaller circles and occasionally skipping a full turn to wait for the next pass of the virtual fish. The intermediate speed of 10 cm/s appears to correspond to the comfort speed of the real fish when interacting with a virtual conspecific.
These results show that the real fish is strongly attracted by the virtual fish, and consistently seeks to stay nearby.
3.2.2 Impact of virtual fish’s swimming depth on the behavior of the real fish.
We now consider three depths of the virtual fish, , 5, and 8 cm, and constant speed and distance to the wall
cm/s and
cm respectively. In all three cases, the real fish maintains the same average distance to the virtual one, as shown in the three PDFs of the distance between fish, all sharply peaked at
cm (Fig 10A and S3 Table). Similarly, the real fish closely matches the speed of the virtual fish, with the three PDFs nearly symmetric and centered at
(Fig 10B and S3 Table). The same imitation pattern is observed in the PDFs of the depth (Fig 10C and S3 Table), with the real fish swimming practically at the same depth as the virtual one, slightly above it (as already observed in the speed variation experiment), except in the deepest case. There, the fish’s maneuverability and range of depth choices is limited by the reduced vertical space, only 3.5 cm, while the body height of H. rhodostomus is approximately 0.5 cm. Despite that, the variation towards greater depths remains similar across all three cases, with variability in depth selection rarely exceeding
cm.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the swimming depth of the virtual fish: , 5, and 8 cm/s. The virtual fish (VF) swims at a constant speed
cm/s in all cases, and at a distance from the wall
, 5.4, and 5 cm for each depth, respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
The deeper the virtual fish swims, the less space the real fish has to stay at the same depth. Thus, the greater the depth, the smaller the average distance at which the real fish stays from the virtual one: 7.97, 7.83, and 6.25 cm, for , 5, and 8 cm, respectively. The x-range around the virtual fish covered by the real fish, indicated by the radius of the density maps in Fig 10EF, reaches around 38 cm in the less deep cases, but only 15 cm when the virtual fish is at 8 cm depth (see the region of high density in blue and red in the color maps). Despite this, the peak of the PDF of the distance between fish remains unchanged across depths, indicating that the real fish maintains the same typical distance to the virtual fish. Finally, the asymmetry of the PDF of the depth
cm, with a longer tail toward deeper values, is explained by the limited space available below the virtual fish at that depth.
3.2.3 Impact of the virtual fish’s distance to the wall on the behavior of the real fish.
In the third experiment, we used three values of the distance between the virtual fish and the wall, , 5.4, and 10.4 cm, while keeping constant the virtual fish’s speed and depth at
cm/s and
cm respectively.
As in the previous cases, the real fish tends to adopt the same behavioral patterns as the virtual fish when the setup geometry allows it. The separation between fish is again the same in the three conditions, with the peak of the three PDFs of the distance between fish consistently located at 2.5 cm (Fig 11A and S4 Table). The velocity and depth of the real fish are again symmetric curves respectively centered in (Fig 11B and S4 Table) and
(Fig 11C and S4 Table), with, as before, a slight preference for being positioned above the virtual fish.
Probability density functions (PDF) of (A) the distance between the real fish and the virtual fish, (B) the swimming speed of the real fish, and (C) the swimming depth of the real fish, for 3 values of the distance to the wall of the virtual fish: , 5.4, and 10.4 cm. The virtual fish (VF) swims at constant speed and depth in all cases,
cm/s and
cm respectively. (DE) Relative position of the real fish in the reference frame of the virtual fish for the three swimming depths, in (D) the xy-plane and (E) the xz-plane. The virtual fish is located at (0,0). The y-axis points in the direction of the fish, and the x-axis is perpendicular to the fish and points outward from the bowl.
The high peak of the PDF of the distance between fish when the virtual fish is farther from the wall is due to the fact that in that case the maximum available distance for the real fish is reduced. The diameter of the small circle of the sphere at depth z = 5 cm is approximately 40 cm; since the virtual fish is at about 10 cm from the wall, the real fish can only move as far as about 30 cm. This is also visible in the radius of occupancy of the real fish around the virtual one (Fig 11E). When the virtual fish is the closest to the wall, there is practically no room for the real fish between the virtual one and the edge of the bowl. In contrast, when the virtual fish swims at 10.4 cm from the border, the real fish stays more frequently closer to the virtual one (the color map far from the virtual fish is clearer than when cm), and often swims in the region between the virtual fish and the wall, as shown in Fig 7D, demonstrating the effectiveness of the anamorphic projection on the opposite side of the bowl.
Similarly to what was observed in the condition closest to the surface, the fact that the virtual fish is very close to the limits of the swimming space (delimited by the edge of the bowl and the surface of the water) does not perturb the preference of the real fish when it is positioned at the side of the virtual fish where these limits have no effect.
3.3 Behavioral response of real fish to the motion of a virtual fish along rose-like trajectories
We now explore how the real fish responds to more complex trajectories of the virtual fish, specifically, two rhodonea trajectories given by the parametric equation , where
are the polar coordinates, with
cm, n = 3 and d = 5 (Fig 7E) and n = 3 and d = 1 (Fig 7F and S3 Video). We fix the speed and depth of the virtual fish to
cm/s and
cm respectively, which are the intermediate values used in the previous experiments in circles, and the shortest distance to the wall to
cm (so the maximum radius is
cm). The interest lies in how the fish reacts as the virtual fish transitions from moving along the edge of the bowl to traveling through its interior, where water distortion can have stronger effects on the image perceived by the fish, and with a varying curvature of the trajectory, particularly in the d = 1 case, where the petals are sharp and pointed, introducing abrupt changes on the projected image of the virtual fish. These trajectories are particularly demanding for the fish, as they often cross the central part of the bowl, where the anamorphic image projected on the screen is subject to considerable optical distortion due to the wider amount of water from the edge of the bowl.
Despite this challenge, the density map of Rose 1 (Fig 12A) clearly matches the trajectory of the virtual fish, with three broad and smooth lobes corresponding to the three rose petals. This indicates that the fish is able to accurately follow the virtual fish even under strong visual distortion. In contrast, the trajectory described in Rose 2 (Fig 12D) has narrower petals, requiring sharper turns from the real fish, and the density map is less defined, suggesting that the real fish has more difficulty in following the virtual one.
(AD) Rhodonea trajectories of three petals. Red line: trajectory of the virtual fish. Density map: positions of the real fish. (A) Rose 1, wide petals (n = 3, d = 5), (D) Rose 2, narrow petals (n = 3, d = 1). (B) Probability density function (PDF) of the distance between fish along Rose 1 (magenta) and Rose 2 (brown) trajectories. (C) PDF of the speed of the real fish in each rose. Vertical blue dashed line: speed of the virtual fish. (E) PDF of the depth of the real fish in each rose. Vertical blue dashed line: depth of the virtual fish. (F) PDF of the angle with which the virtual fish is perceived by the real one, multiplied by the sign of the virtual fish direction of rotation (–1 for clockwise, + 1 for counterclockwise).
In Rose 2, the green areas indicate that the fish spends more time near the tips of the petals, where it slows down to perform the turning maneuvers (Fig 12D). The central green region is simply explained by the geometry of the trajectory: the fish crosses the center three times more often than the outer tips, a pattern that also happens in the red regions of the density map of Rose 1.
In both roses, the real fish maintains similar distances to the virtual fish as those observed in circular trajectories, with a peak of the PDF at the same value, 2.5 cm, although of smaller size (Fig 12B). This means that the real fish swims slightly farther away than in the circular cases. Comparing the roses, there is a slight tendency to keep a greater distance in Rose 2, likely due to the sharper turns.
Also, in both roses, the fish is able to adjust to the speed of the virtual fish, with both PDFs of the speed centered on , but wider than in the circular case, showing that following these irregular paths is more demanding in terms of speed adjustment (Fig 12C). Moreover, the peak of the curve of Rose 1 is further to the right, indicating that the fish swims faster when following smoother turns.
Finally, there are no significant differences between the PDFs of the depth for the two roses; both curves are almost symmetric and centered very slightly to the left of the expected depth of 5 cm (Fig 12E).
Fig 12F shows the PDF of the angle , defined as the angle with which the virtual fish is perceived by the real one, multiplied by the sign of the direction of rotation of the virtual fish (–1 clockwise, + 1 counterclockwise). In both roses, once the direction of rotation is given, the virtual fish always has the wall at the same side along the whole path of the rose: to its left during clockwise motion, and to its right during anticlockwise motion. This allows us to cumulate the experiments performed in both directions and reveal the lateral positioning of the virtual fish with respect to the real one. Both PDFs show that the real fish tends to position itself so that the virtual fish remains in front of it, as shown by the peaks at
. More interestingly, both PDFs are quite asymmetric, revealing a robust behavioral preference: the real fish tends to position itself on the inner side of the path of the virtual fish. In other words, regardless of the rotation direction, the real fish places itself on the side of the virtual fish towards which the latter is turning.
S3 Fig shows the cross-correlation between the velocity vectors of the real fish,
, and the virtual fish,
+
, in both rhodonea conditions. As already observed, despite the complexity of the trajectories, the correlation values remain high, only slightly lower than those observed for circular trajectories. The maximum correlation is higher for Rose 1 (
) than for Rose 2 (
), confirming that Rose 2 is more difficult for the real fish to follow. The respective maxima are reached with a slight negative delay at t1 = −0.23 s ( + nT1, T1 = 24.6 s) and t2 = −0.1 s ( + nT2, T2 = 13.4 s), for
, indicating that in both cases, the real fish tends to follow the virtual one. In Rose 1, this delay is longer (almost a quarter of a second) but yields a stronger correlation, whereas in Rose 2, the delay is shorter (one tenth of a second), but the correlation is weaker.
4 Discussion
Understanding how animals interact socially is a long-standing goal in behavioral research [1,2]. However, studying these processes in real time is often difficult. Traditional methods struggle to capture the complexity of live interactions and cannot easily manipulate environmental factors. In this work, we present and validate an open-source and low-cost immersive virtual reality (VR) system designed specifically to investigate real-time social interactions between fish. We show that this system allows a real fish to swim freely while interacting with a computer-generated virtual fish whose behavior can be precisely controlled. The VR setup uses closed-loop feedback, meaning that the virtual fish responds to the real fish’s movements in real time.
Our results show that real fish adjust their swimming speed depending on the speed of the virtual fish. When the virtual fish swims slowly, the real fish often moves faster, sometimes circling and waiting for it. At intermediate speeds, the real fish synchronizes more closely, and at higher speeds, it sometimes struggles to keep up, resulting in shortcuts or pauses. This shows that real fish are sensitive to the movement speed of their virtual partners and modify their own behavior accordingly. It also suggests that real fish prefer to maintain proximity to conspecifics moving at naturalistic speeds. Previous studies in live groups also indicated that matching speed is critical for coordination, and these results strengthen that conclusion by isolating the interaction to a single partner [39,40].
We also find that real fish tended to swim at about the same depth as the virtual fish, usually slightly above it. Even when the virtual fish swims deeper or closer to the surface, the real fish adapts remarkably well. However, when the virtual fish moves very close to the bottom, real fish have less vertical room to maneuver, resulting in slightly reduced matching. These results demonstrate the real fish’s strong ability to track vertical position, even under spatial constraints. To determine their swimming depth, it has been shown that fish use the rate in change of pressure mediated by changes in swim-bladder volume, together with their vertical speed [41].
Our results further reveal that the spatial context, particularly the distance of the virtual fish to the tank wall, significantly shapes real fish behavior. When the virtual fish swims closer to the wall, real fish follows it tightly. When the virtual fish swims farther from the wall, the real fish maintains the same close distance but sometimes maneuvers between the virtual fish and the wall. This shows that real fish perceive and adapt to environmental constraints while trying to stay close to a social partner. Moreover, these results highlight the effectiveness of the anamorphic projection: fish behaved as if the virtual conspecific was physically present in the water, even when distortions could have been expected. This spatial flexibility echoes findings from studies on collective navigation in open water, where environmental features modulate but do not eliminate social attraction [42].
Finally, our results demonstrate that real fish were able to follow complicated paths such as rhodonea (non-circular) trajectories surprisingly well. When petals were broad and smooth, fish tracked the virtual fish closely. When the path required sharper turns, fish performance declined somewhat, with real fish lagging or showing wider turns. Still, the basic tendency to follow remained strong. This indicates that the VR setup can be used to study not only simple pursuit but also more intricate forms of social tracking. These results suggest that fish can integrate more complex motion cues than previously thought. In natural settings, this ability likely supports following in rapidly changing environments, such as evading predators or tracking group members in turbulent flows.
5 Conclusions
In summary, our study shows that a simple, low-cost, and open-source VR system can effectively create realistic social interactions between real and virtual fish. Real fish responded dynamically to changes in virtual fish speed, depth, distance from the wall, and trajectory complexity. They showed remarkable behavioral flexibility and consistency, even under visual distortions or challenging movement patterns. The system’s closed-loop design was essential, ensuring that virtual stimuli adapted to real fish movements, making the interactions genuinely interactive.
The current work focuses on open-loop validation, but the design also supports closed-loop integration with three-dimensional behavioral models, enabling causal tests of social interaction rules. Moreover, the ability to project multiple virtual fish offers new opportunities to study how individuals integrate information from several neighbors and how local rules scale to group dynamics [28,43,44].
This platform opens new possibilities for studying how sensory inputs drive social decisions at fine scales. It may also serve as a model for future work combining VR, brain imaging, and genetic tools to uncover the neural basis of collective behavior [45,46]. It demonstrates how low-cost, accessible technologies can significantly advance our understanding of animal social cognition. Ultimately, our findings may have broader implications for the fields of robotics, artificial intelligence, and swarm intelligence, where biological principles of collective motion inspire novel engineering solutions [47,48].
Supporting information
S1 Text. Experimental setup, software architecture, and trajectory design for the closed-loop virtual reality system.
https://doi.org/10.1371/journal.pone.0339909.s001
(PDF)
S1 Fig. Experimental setup.
Descriptive diagram (left) and dimensions (right).
https://doi.org/10.1371/journal.pone.0339909.s002
(PDF)
S2 Fig. Cross-correlation of the velocity vector of the real fish with that of the virtual fish at different virtual fish speeds.
Virtual fish conditions correspond to C1, C2, and C3, with (orange),
(blue), and
cm/s (green). The trajectories of the virtual fish are circles of radius R = 15 cm at depth z = 5 cm corresponding to a distance to the wall of
cm. Maximum correlation is reached at t1 = 0 s + nT1, t2 = 0.4 s + nT2, and t3 = −0.267 s + nT3, where
s,
s, and
s, for
.
https://doi.org/10.1371/journal.pone.0339909.s003
(PDF)
S3 Fig. Cross-correlation of the velocity vectors of the real and virtual fish for the two rhodonea trajectories.
In both cases, the swimming speed is cm/s, the minimum distance to the wall is
cm, and the swimming depth is z = 5 cm (corresponding to a maximum radius of R = 19 cm). The maximum correlation for Rose 1 (purple) is reached at t1 = −0.23 s + nT1, with T1 = 24.6 s, and for Rose 2 (brown) at t2 = −0.1 s + nT2, with T2 = 13.4 s, for
. Vertical dashed lines indicate the respective period lengths T1 and T2.
https://doi.org/10.1371/journal.pone.0339909.s004
(PDF)
S1 Table. Main frame materials of the Virtual Reality setup.
https://doi.org/10.1371/journal.pone.0339909.s005
(PDF)
S2 Table. Hellinger distance between the probability distribution function (PDF) of the 3 observables used to quantify the impact of virtual fish’s swimming speed on real fish’s behavior.
We list the distances obtained for the three conditions compared pairwise. The real fish maintained the same swimming distance and depth relative to the virtual fish across the three swimming speeds tested (i.e., no significant differences were found between the PDFs). However, the swimming speeds of the real fish were significantly different between the three conditions C1, C2, and C3 (i.e., when the virtual fish moved at 5, 10, and 15 cm/s respectively). Values of Hellinger distance are shown in bold font when H > 0.2 (high dissimilarity of the PDFs).
https://doi.org/10.1371/journal.pone.0339909.s006
(PDF)
S3 Table. Hellinger distance between the probability distribution function (PDF) of the 3 observables used to quantify the impact of virtual fish’s swimming depth on real fish’s behavior.
We list the distances obtained for the three conditions compared pairwise. The swimming distance of the real fish to the virtual fish was significantly smaller when the virtual fish was at 8 cm depth. There was also a significant difference in the swimming depth of the real fish between the three conditions C1, C8, and C9. Values of Hellinger distance are shown in bold font when H > 0.2 (high dissimilarity of the PDFs).
https://doi.org/10.1371/journal.pone.0339909.s007
(PDF)
S4 Table. Hellinger distance between the probability distribution function (PDF) of the 3 observables used to quantify the impact of the virtual fish’s distance to the wall on the real fish’s behavior.
We list the distances obtained for the three conditions compared pairwise. The swimming speed and depth of the real fish were not influenced by the virtual fish’s proximity to the tank wall. Only when the virtual fish was swimming close to the wall at 2.4 cm in condition C7, the real fish maintained a higher distance in comparison with condition C6, in which the distance to the wall of the virtual fish was much larger (10.4 cm). Values of Hellinger distance are shown in bold font when H > 0.2 (high dissimilarity of the PDFs).
https://doi.org/10.1371/journal.pone.0339909.s008
(PDF)
S1 Video. Comparison of a fish’s behavior in the presence of a virtual fish and then a virtual ball under control conditions.
Video excerpts of an experiment with a real fish interacting with the anamorphic projection of a virtual conspecific during 30 minutes and then with a non-social control stimulus consisting of a black sphere during the next 30 minutes in the experimental bowl of radius 250 mm. Both the virtual fish and virtual sphere move along a uniform circular trajectory with a constant speed of 10 m/s, at a constant depth of 5 cm below the water surface, and at a constant distance of 10.4 cm to the edge of the tank. The video shows, after one minute, the replacement of the three-dimensional fish model with the spherical model. Top-Left: user interface allowing the real-time visualization of the trajectories of the real and virtual fish (in the xy- and xz-planes) and the instantaneous modification of the parameters of the model driving the virtual fish. Bottom-Left: real-time 3D tracking of the real fish. Right panel: anamorphic rendering of the virtual fish projected onto the acrylic bowl by the rendering application according to the 3D position of the real fish.
https://doi.org/10.1371/journal.pone.0339909.s009
(MP4)
S2 Video. Behavioral response of a real fish to the circular movement of a virtual fish.
Video excerpts of an experiment with a real fish interacting with the anamorphic projection of a virtual conspecific in the experimental bowl of radius 250 mm. The virtual fish moves along a uniform circular trajectory with a constant speed of 10 cm/s, at a constant depth of 5 cm below the water surface, and at a constant distance of 5.4 cm to the edge of the tank. Top-Left: user interface allowing the real-time visualization of the trajectories of the real and virtual fish (in the xy- and xz-planes) and the instantaneous modification of the parameters of the model driving the virtual fish. Bottom-Left: real-time 3D tracking of the real fish. Right panel: anamorphic rendering of the virtual fish projected onto the acrylic bowl by the rendering application according to the 3D position of the real fish.
https://doi.org/10.1371/journal.pone.0339909.s010
(MP4)
S3 Video. Behavioral response of real fish to the motion of a virtual fish along a rose-like trajectory.
Video excerpt of an experiment with a real fish interacting with the anamorphic projection of a virtual conspecific in the experimental bowl of radius 250 mm. The virtual fish moves along a rhodonea (rose-like) trajectory with a constant speed of 10 cm/s and at a constant depth of 5 cm below the water surface. The rhodonea trajectory is given by the parametric equation , where
are the polar coordinates, with
cm, n = 3 and d = 1. Top-Left: user interface allowing the real time visualization of the trajectories of the real and virtual fish (in the xy- and xz-planes) and the instantaneous modification of the parameters of the model driving the virtual fish. Bottom-Left: real-time 3D tracking of the real fish. Right panel: anamorphic rendering of the virtual fish projected onto the acrylic bowl by the rendering application according to the 3D position of the real fish.
https://doi.org/10.1371/journal.pone.0339909.s011
(MP4)
Acknowledgments
The authors warmly thank Zhen Kang for his help in processing the control condition data.
References
- 1.
Camazine S, Deneubourg JL, Franks NR, Sneyd J, Theraulaz G, Bonabeau E. Self-organization in biological systems. Princeton, NJ: Princeton University Press; 2001.
- 2.
Sumpter DJT. Collective animal behavior. Princeton, NJ: Princeton University Press; 2010.
- 3. Couzin ID, Krause J, Franks NR, Levin SA. Effective leadership and decision-making in animal groups on the move. Nature. 2005;433(7025):513–6. pmid:15690039
- 4. Lopez U, Gautrais J, Couzin ID, Theraulaz G. From behavioural analyses to models of collective motion in fish schools. Interface Focus. 2012;2(6):693–707. pmid:24312723
- 5. Parrish JK, Edelstein-Keshet L. Complexity, pattern, and evolutionary trade-offs in animal aggregation. Science. 1999;284(5411):99–101. pmid:10102827
- 6.
Krause J, Ruxton GD. Living in groups. Oxford: Oxford University Press; 2002.
- 7. Katz Y, Tunstrøm K, Ioannou CC, Huepe C, Couzin ID. Inferring the structure and dynamics of interactions in schooling fish. Proc Natl Acad Sci U S A. 2011;108(46):18720–5. pmid:21795604
- 8. Herbert-Read JE, Perna A, Mann RP, Schaerf TM, Sumpter DJT, Ward AJW. Inferring the rules of interaction of shoaling fish. Proc Natl Acad Sci U S A. 2011;108(46):18726–31. pmid:22065759
- 9. Gautrais J, Ginelli F, Fournier R, Blanco S, Soria M, Chaté H, et al. Deciphering interactions in moving animal groups. PLoS Comput Biol. 2012;8(9):e1002678. pmid:23028277
- 10. Calovi DS, Litchinko A, Lecheval V, Lopez U, Pérez Escudero A, Chaté H, et al. Disentangling and modeling interactions in fish with burst-and-coast swimming reveal distinct alignment and attraction behaviors. PLoS Comput Biol. 2018;14(1):e1005933. pmid:29324853
- 11. Stowers JR, Hofbauer M, Bastien R, Griessner J, Higgins P, Farooqui S, et al. Virtual reality for freely moving animals. Nat Methods. 2017;14(10):995–1002. pmid:28825703
- 12. Fry SN, Rohrseitz N, Straw AD, Dickinson MH. TrackFly: Virtual reality for a behavioral system analysis in free-flying fruit flies. J Neurosci Methods. 2008;171(1):110–7. pmid:18405978
- 13. Jouary A, Haudrechy M, Candelier R, Sumbre G. A 2D virtual reality system for visual goal-driven navigation in zebrafish larvae. Sci Rep. 2016;6:34015. pmid:27659496
- 14. Huang K-H, Rupprecht P, Frank T, Kawakami K, Bouwmeester T, Friedrich RW. A virtual reality system to analyze neural activity and behavior in adult zebrafish. Nat Methods. 2020;17(3):343–51. pmid:32123394
- 15. Vidal M, Mills SC, Gairin E, Bertucci F, Lecchini D. Validation of a novel immersive virtual reality set-up with responses of wild-caught freely moving coral reef fish. Anim Behav. 2023;206:99–123.
- 16. Naik H, Bastien R, Navab N, Couzin ID. Animals in virtual environments. IEEE Trans Vis Comput Graph. 2020;26(5):2073–83. pmid:32070970
- 17. Brookes J, Warburton M, Alghadier M, Mon-Williams M, Mushtaq F. Studying human behavior with virtual reality: The unity experiment framework. Behav Res Methods. 2020;52(2):455–63. pmid:31012061
- 18. Froese T, Iizuka H, Ikegami T. Embodied social interaction constitutes social cognition in pairs of humans: A minimalist virtual reality experiment. Sci Rep. 2014;4:3672. pmid:24419102
- 19. Oscar L, Li L, Gorbonos D, Couzin ID, Gov NS. A simple cognitive model explains movement decisions in zebrafish while following leaders. Phys Biol. 2023;20(4):10.1088/1478-3975/acd298. pmid:37141898
- 20. Harpaz R, Nguyen MN, Bahl A, Engert F. Precise visuomotor transformations underlying collective behavior in larval zebrafish. Nat Commun. 2021;12(1):6578. pmid:34772934
- 21. Li L, Nagy M, Amichay G, Wu R, Wang W, Deussen O. Reverse engineering the control law for schooling in zebrafish using virtual reality. Sci Adv. 2021;7:eabe7960.
- 22.
Butail S, Chicoli A, Paley DA. Putting the fish in the fish tank: Immersive VR for animal behavior experiments. In: Proceedings of the IEEE international conference on robotics and automation (ICRA); 2012. p. 4251–6.
- 23. Trivedi CA, Bollmann JH. Visually driven chaining of elementary swim patterns into a goal-directed motor sequence: A virtual reality study of zebrafish prey capture. Front Neural Circuits. 2013;7:86. pmid:23675322
- 24. Takalo J, Piironen A, Honkanen A, Lempeä M, Aikio M, Tuukkanen T, et al. A fast and flexible panoramic virtual reality system for behavioural and electrophysiological experiments. Sci Rep. 2012;2:324. pmid:22442752
- 25. Torigoe M, Islam T, Kakinuma H, Fung CCA, Isomura T, Shimazaki H, et al. Zebrafish capable of generating future state prediction error show improved active avoidance behavior in virtual reality. Nat Commun. 2021;12(1):5712. pmid:34588436
- 26. Jiang L, Giuggioli L, Perna A, Escobedo R, Lecheval V, Sire C, et al. Identifying influential neighbors in animal flocking. PLoS Comput Biol. 2017;13(11):e1005822. pmid:29161269
- 27. Hinz RC, de Polavieja GG. Ontogeny of collective behavior reveals a simple attraction rule. Proc Natl Acad Sci U S A. 2017;114(9):2295–300. pmid:28193864
- 28. Lei L, Escobedo R, Sire C, Theraulaz G. Computational and robotic modeling reveal parsimonious combinations of interactions between individuals in schooling fish. PLoS Comput Biol. 2020;16(3):e1007194. pmid:32176680
- 29. Puy A, Gimeno E, Torrents J, Bartashevich P, Miguel MC, Pastor-Satorras R, et al. Selective social interactions and speed-induced leadership in schooling fish. Proc Natl Acad Sci U S A. 2024;121(18):e2309733121. pmid:38662546
- 30.
Intel Corporation. RealSense D435 Depth Camera; 2025. https://www.intelrealsense.com/depth-camera-d435/
- 31.
Intel Corporation. RealSense SDK 2.0; 2025. https://www.intelrealsense.com/sdk-2/
- 32.
OpenCV Foundation. OpenCV Image Processing Routines; 2025. https://opencv.org/
- 33. Tuncer B, Ozkan E. Random matrix based extended target tracking with orientation: A new model and inference. IEEE Trans Signal Process. 2021;69:1910–23.
- 34. Feldmann M, Franken D, Koch W. Tracking of extended objects and group targets using random matrices. IEEE Trans Signal Process. 2011;59(4):1409–20.
- 35.
The Qt Company. Qt Framework for C Applications. https://www.qt.io/. 2025.
- 36.
Unity Technologies. C# Rendering Application; 2025. https://unity.com/fr
- 37.
Basu A, Harris IR, Basu S. 2 Minimum distance estimation: The approach using density-based distances. Handbook of statistics. Elsevier; 1997. p. 21–48. https://doi.org/10.1016/s0169-7161(97)15004-0
- 38. Beran R. Minimum Hellinger distance estimates for parametric models. Ann Statist. 1977;5(3).
- 39. Schaerf TM, Dillingham PW, Ward AJW. The effects of external cues on individual and collective behavior of shoaling fish. Sci Adv. 2017;3(6):e1603201. pmid:28691088
- 40. Jolles JW, Weimar N, Landgraf T, Romanczuk P, Krause J, Bierbach D. Group-level patterns emerge from individual speed as revealed by an extremely social robotic fish. Biol Lett. 2020;16(9):20200436. pmid:32933404
- 41. Taylor GK, Holbrook RI, de Perera TB. Fractional rate of change of swim-bladder volume is reliably related to absolute depth during vertical displacements in teleost fish. J R Soc Interface. 2010;7(50):1379–82. pmid:20190038
- 42. Berdahl A, Torney CJ, Ioannou CC, Faria JJ, Couzin ID. Emergent sensing of complex environments by mobile animal groups. Science. 2013;339(6119):574–6. pmid:23372013
- 43. Wang W, Escobedo R, Sanchez S, Sire C, Han Z, Theraulaz G. The impact of individual perceptual and cognitive factors on collective states in a data-driven fish school model. PLoS Comput Biol. 2022;18(3):e1009437. pmid:35235565
- 44. Xue T, Li X, Lin G, Escobedo R, Han Z, Chen X, et al. Tuning social interactions’ strength drives collective response to light intensity in schooling fish. PLoS Comput Biol. 2023;19(11):e1011636. pmid:37976299
- 45. Yu J-H, Napoli JL, Lovett-Barron M. Understanding collective behavior through neurobiology. Curr Opin Neurobiol. 2024;86:102866.
- 46. Zada D, Schulze L, Yu JH, Tarabishi P, Napoli JL, Milan J, et al. Development of neural circuits for social motion perception in schooling fish. Curr Biol. 2024;34:3380–91.
- 47.
Verdoucq M, Theraulaz G, Escobedo R, Sire C, Hattenberger G. Bio-inspired control for collective motion in swarms of drones. In: 2022 international conference on unmanned aircraft systems (ICUAS); 2022. p. 1626–31. https://doi.org/10.1109/icuas54217.2022.9836112
- 48.
Verdoucq M, Sire C, Escobedo R, Theraulaz G, Hattenberger G. Bio-Inspired 3D flocking algorithm with minimal information transfer for drones swarms. In: 2023 IEEE/RSJ international conference on intelligent robots and systems (IROS); 2023. p. 8833–8. https://doi.org/10.1109/iros55552.2023.10341413