General task conditions.

Animals were moved from the animal house to the arena near the end of the light period and remained there for the duration of the experiment (2 to 3 weeks). The arena contained a shelter, a water dish, and a basking area. Before starting the experiment, each animal was cleaned inside an open box filled with shallow water to prevent excessive dirt from reducing position-tracking accuracy. Subsequently, animals were placed in the dark arena, simulating nighttime, and the experiment protocol began the following day (at 7 AM). During all experiments, lizards received a vegetable meal every 7 days. They were given live food manually in case they did not reach a minimum of 10 worms every 3 days. The experiment was fully automated by the experiment python module (see system/experiments/loclearn2.py in the ReptiLearn repository). The LED strip and the heat lamp were automatically turned on from 7 AM to 7 PM daily.

Task structure.

The task consisted of 3 blocks. In the first block, lizards were rewarded for entering the feeder area, and in the second and third blocks for entering area 1 and 2, respectively (Fig 4B). Specifically, upon entering the rewarded location and staying there for 2 s (wait period), the cue LED started to blink and 5 s later it was turned off, and a command was sent to dispense the reward. In pilot tests, we found that lizards positioned themselves on the edge of rewarded areas such that slight movements resulted in repeated crossing into the rewarded areas. To solve this, we required animals to exit the cooldown area (and wait 20 s starting from LED blink onset) before entering the rewarded area again to receive another reward. Rewarded locations were only defined in the video analysis algorithm and were not physically marked in the arena. The transition between blocks occurred after 60 rewards were received, or after 6 days, whichever occurred first.

Logging task data.

Lizard position was constantly tracked throughout the task. Entrance to the rewarded location was determined in real time by the experiment module (see software) that received input from the YOLOv4 ImageObserver (see software) tracking the animal’s head position from the top camera video stream. The ImageObserver output was stored in CSV files for later analysis (see software). The event logger tracked the time of reward releases, their resupply times, and the moments when the animal entered or exited the reinforced area. Switching between the 3 reinforced areas was done manually using the Web UI by moving to the next block in the session UI section (see software). Switching was always done at the beginning of a day so that the same area was rewarded throughout each day. The experiment module tracked the number of available rewards in each stacked feeder and switched between feeders automatically when one of them was empty. This information was also relayed to the experimenter through the Web UI so that the feeders could be manually refilled when necessary.

Closed-loop latency test.

We estimated the latency of delivering a closed-loop stimulus by combining the processing duration of the YOLOv4 algorithm with the processing time of all other arena components including camera acquisition. The average duration of YOLOv4 processing was measured directly (mean = 7.92 ms, SD = 0.36 ms, Fig 2B) and the delays associated with the rest of the arena components were measured by turning on an LED in the arena and measuring the time until the system turned it automatically off while the entire arena was running. Specifically, we placed an LED on the arena floor and turned it on. We measured the intensity of a single pixel in the LED light spot in each top-view camera frame and sent a command to the arena controller to turn the LED off once it was on (its intensity reached a certain threshold). We measured the latency by counting the number of frames (at 60 Hz frame rate) in which the LED was turned on. Thus, the latency measurement is quantized to 17 ms bins and represents an upper bound of the real latency (Fig 2C).

Animal head position tracking.

To track the lizards head and determine if it is inside the reinforced area, we trained YOLOv4 [51]—a light-weight convolutional neural network. The model was used to detect the animal’s head bounding box in each top-view camera frame (Fig 2A). Training was performed using a custom dataset consisting of approximately 2,000 grayscale images of P. vitticeps lizards from various angles, cameras, and different backgrounds, as well as approximately 1,400 grayscale images that did not contain a lizard, which were added to reduce the number of false-positive detections. Approximately 800 of these negative examples were images of birds and insects from the Imagenet dataset [87], and the rest were images taken from several empty arenas. The resulting model was able to detect the lizard head in a wide range of images with various individual lizards, camera angles, and arenas (Fig 2D). Each top-view camera frame was resized to 416 × 416 pixels and then processed with the trained model. The output of the model provides a confidence score for each bounding box. Non-max suppression with a 0.6 threshold was used to remove overlapping detected bounding boxes with lower scores [51]. Additionally, detections with scores lower than 0.8 were discarded. When the output of the non-max suppression algorithm resulted in multiple bounding boxes, we used the box closest to the previous detected one and ignored the others. We used the bounding box’s center point to estimate the animal’s head position in the arena.

Validation of fine-tuned YOLOv4 model.

We evaluated the model’s performance by comparing manually annotated head bounding boxes and model-derived bounding boxes. We used a validation set of 400 images sampled uniformly from 800 h of experimental videos from 4 lizards. Using a confidence threshold of 80%, the model achieved a recall of 100% (no false positives) and a precision (true positive/total positive) of 78% (Fig 2D). To estimate bounding box accuracy, we divided the intersection area of the 2 boxes by the area of their union (Intersection over Union, IoU). Out of the true positive detections, the average IoU between the predicted and manually annotated bounding boxes was 73% (S1 Fig) indicating a good accuracy.

Travel distance measurements.

We measured the travel distance of the animal by cumulatively summing the distances between head position coordinates along pairs of consecutive video frames. To bridge frames with poor YOLOv4 annotation (see above), we linearly interpolated the position in these frames using values in neighboring frames. Lizards can spend long periods of time in the same location. During these times, small random fluctuations in YOLOv4 position estimation are accumulated resulting in increased travel distance. To remove such contributions, we used Gaussian filtering on the position coordinates (window size = 51; SD = 17). The filter window size was chosen empirically by plotting travel distance as a function of window size for a range of values during periods of limited movement: Travel distance was observed to drop quickly for small window sizes and nearly saturate for windows larger than 51. To transform the measurements from pixels to centimeters, we multiplied the values by a constant factor, which was calculated by computing the distance between several pixel pairs with known physical distances and averaging the resulting multiplication factors.

Segmentation into periods of movement and quiescence.

We segmented each experiment day into periods in which the animal moved in the arena or stayed in place based on the position-tracking data generated by the YOLOv4 model. First, we used time-based linear interpolation to replace missing position values. We applied a Gaussian filter (window size = 51, see travel distance section above) to reduce the jitter of the model output, resulting in a matrix of Tx2, where T is the number of video frames recorded daily. Then, we created a time-delay embedding from this data (windows size = 50, gap size = 2) [88] and calculated the differences between consecutive columns for each row, resulting in a matrix of Tx100 (each row containing 50 interleaved 2D velocity vectors). We conducted a principal component analysis on this matrix and kept the first 6 principal components (explaining more than 90% of the data variance in an animal), which resulted in a Tx6 matrix. We then converted this matrix into a binary vector of size T by calculating the L2-norm of each row and applying a threshold. A value of 1 indicated a movement sample and 0 indicated a stationary sample. This vector was then used as an input to Kleinberg’s burst detection algorithm [89] to segment into periods of stationarity and movement.

Entry rate statistics.

To measure whether animals significantly increased their entry rate to the reinforced area compared to the previous block (ΔER), we defined 96 areas (including both second and third reinforced areas) in an evenly spaced grid across the arena floor, each with the same size and shape as the reinforced areas (S5 Fig). We used the animal’s position data to estimate the entry rate for each area in each block, resulting in ΔER values for each area and block transition. To calculate the entry rate for each area, we used offline simulations, assuming each time that a different area was the reinforced one and incorporating all rules for releasing a reward used in the actual task (as specified in the Results section). For correlation analysis, areas were classified as either neighboring the feeder area, the first reinforced area, the second reinforced area, or not neighboring any reinforced areas (green, orange, blue, and gray areas in S5 Fig, respectively). This was determined based on whether areas had overlapping sections with the reinforced areas; however, the classification was slightly altered to ensure areas were only considered neighbors of a single reinforced area and maintaining an equal number of neighbors for the second and the third reinforced areas. The simulation output was the number of rewards per block. We divided this value by the number of light hours in each block to derive reward rates.

Basking periods estimation.

Animals’ basking periods were determined based on position data (from YOLOv4). Animals were considered basking when their head bounding-box centroid resided within a radius of 20 cm around the heat lamp center.

Camera thermal measurements.

We used a thermal camera (see hardware) to estimate ambient and animal skin temperatures during experiment sessions. We recovered temperature data for each pixel by decoding video files and linearly scaling the decoded 8bit images to Celsius values. We then calculated the ambient arena temperature for each frame by averaging over all image pixels. This measurement exhibited the same trend as the ambient temperature sensor on the arena wall but was 1.67°C higher on average (SD = 0.16) due to the inclusion of the basking area in the calculation.

Animal thermal measurements.

We estimated animals’ body temperature as follows. We found the temporally closest regular camera frame for each thermal video frame and extracted the animal head bounding box centroid using YOLOv4. We used the centroid coordinates as an input prompt to the SAM segmentation model [53] and produced a segmentation mask containing all animal pixels. We then linearly transformed coordinates to shift from the visual camera to the thermal camera arena coordinates. This transformation was based on a set of 112 manually labeled reference point pairs to align the centroid and mask with the coordinate space of the thermal image. We used SAM again to generate a second segmentation mask of the animal in the thermal image using the transformed centroid as input. The transformed mask was then passed through 2 iterations of erosion and was finally used to determine the estimated body temperatures by computing the median intensity across all mask pixels.

To analyze temperature dynamics over entire days, we extracted thermal video frames in 3-s intervals and executed the procedure described above for each frame. To make sure body temperature measurements are accurate, masks that were unrealistically large or small were discarded. Specifically, only areas of 800 to 4,000 pixels for thermal image masks and 11,000 to 30,000 pixels for regular camera image masks were kept. Additionally, due to the low acquisition frequency of thermal images, pairs of regular and thermal images were occasionally spatially non-overlapping, especially during fast movement bouts. To solve this issue, we calculated the IoU between each pair of masks and removed frames in which the IoU was below 0.3 (2% of the frames; M = 0.5; SD = 0.11). Finally, we filtered the sequence of estimated skin temperatures extracted from valid frames using a moving average (window size = 51). To illustrate the conversion from skin temperature (ST) to core temperature (CT), we used the calibration in [64]. Specifically, core temperature was estimated (S2C Fig) using a second order polynomial fit (CT(T) = p1* ST ² + p2* ST + p3, p1 = 0.0159; p2 = 1.7066, p3 = −6.1034).

Validation of thermal body masks.

We validated the generated thermal body masks by examining the temperature gradients across mask edge points. Since lizards were warmer than ambient temperature, these gradients are expected to sharply decrease at lizard edges. We used a uniform sample of body masks comprising approximately 10% of the total masks generated during the measurement day. Near the heat lamp, the ambient temperatures are high and it is harder to evaluate gradients. We therefore removed frames in which lizards partially overlapped with the basking area (24% of the original sample). We proceeded by calculating the temperature gradient along line segments originating from the animal mask center of mass, and extending outwards such that each edge point was at the center of its respective segment (S2A Fig). Line segments were discarded when their inner part (from animal center to edge) was not fully contained in the mask or when their outer part (from the edge until the line endpoint) overlapped with mask points. For each segment, we calculated the intensity of each pixel according to the thermal image. We aligned the segments according to their middle points and averaged to produce a mean gradient for each frame. Finally, we calculated the distribution of intensities as a function of distance along the line (S2B Fig). We aligned the average mask segments in the same way, normalized their intensities to z-score units, and calculated the median gradient across the masks (S2B Fig, red line).

Arena cage.

The arena was shaped like a box without its top face. It consisted of a frame built with square aluminum profiles and walls and floor made from 3 mm thick aluminum composite panels. The floor dimensions were 70 cm by 100 cm, and its height was 45 cm. An additional 100 cm long profile was placed 1 m above the centerline of the arena floor using additional profiles to support the top-view and thermal cameras, as well as the grid of heat lamps (described below). S1 Table contains the price list of arena components.

Arena computer.

The arena hardware was connected to a desktop computer placed next to the arena, on which the ReptiLearn software was running (Intel Core i7-11700K CPU, 32GB DDR4 memory, NVIDIA GEFORCE RTX 3080 Ti GPU, 500GB SAMSUNG 980 M.2 NVME SSD, and a 2TB 7200 RPM HDD). An Ubuntu 22.04 Linux operating system was used.

Video cameras.

Five cameras were used to acquire video and thermal data in the arena. They were attached to the arena using short adjustable arms (Noga Engineering & Technology, LC6100). Three FLIR Firefly S USB3 monochrome cameras (FFY-U3-16S2M-S) with 6 mm lens (Boowon BW60BLF) were attached to profiles at the top edges of the arena walls. A top-down view of the arena floor was captured using a FLIR Blackfly S USB3 color camera (BFS-U3-16S2C-CS) with a 2.8 to 10 mm varifocal lens (Computar A4Z2812CS-MPIR). Although this is a color camera, only monochrome data was acquired. The camera was attached to the middle of the center-top profile and was used for real-time position tracking. To prevent IR light emitted by heat lamps from flooding images, we covered the Firefly camera lens with an IR cutoff filter attached using custom 3D-printed holders. Video was mainly recorded at a frame rate of 50 Hz.

Thermal camera.

We used a thermal IR camera (FLIR A70) positioned next to the Blackfly camera above the arena to measure temperature dynamics during daily activity periods. The camera captures images in a resolution of 640 × 480 pixels at 16 bits per pixel (bpp), representing temperatures in up to 10 mK resolution. The camera thermal resolution is advertised as 45 mK or less; however, we linearly scaled each image to 8 bpp and encoded it into video files to store the thermal data. We chose a temperature range between 20°C and 45°C and scaled accordingly, which resulted in approximately 0.1°C resolution. The camera supports up to 30 frames per second; however, images were taken at a frequency of about 3 Hz due to technical issues, which FLIR developers eventually solved (but only after we finished conducting the experiments).

Touch screen.

A touch screen (ELO Touch Solutions AccuTouch 1790L 17” LCD open frame) was attached to one of the shorter arena walls using screws. The touch screen was connected to the arena computer using HDMI and USB cables. A cardboard was fitted around the screen to prevent animals from using it to leave the arena.

Arduino microcontroller boards.

Three Arduino boards (Arduino Nano Every) were used to control the arena lighting, food dispensers, temperature sensors, and heat lamps and send TTL signals to synchronize the video cameras (Figs 1B and S6). The boards were connected to the arena computer using USB cables and placed inside a box attached to the external side of an arena wall.

Light.

The arena was lit using an LED strip (12V, 6500K white LEDs, approximately 3.4 m long) that was attached using adhesive to profiles at the top edge of the 4 arena walls. This provided relatively uniform lighting across the arena, minimizing shadows. The strip was controlled using a relay module (based on an Omron G5LE-14-DC5 5VDC SPDT relay). The module’s EN, VDD, and GND control ports were connected to one of the Arduino boards, and it was used to control the DC output of the LED strip 12V, 5A power supply unit (S6A Fig).

Live prey dispenser.

We used a widely available aquarium food dispenser (EVNICE EV200GW) for rewarding animals with live prey. It was attached to the arena frame using the included clamp and released rewards into a small dish placed on the arena floor below it. Commercially available aquarium dispensers are suitable for providing live food. However, they are slow and do not provide means for external control. To control the food dispenser and reduce its latency, we modified its control circuit. We connected a ULN2003 stepper motor driver board to the feeder’s 28BYJ-48 stepper motor (see S6A Fig for circuit details). An Arduino board controls the motor driver, and Arduino code, integrated into the arena controller, implements an alternative motor control sequence, significantly reducing its latency to 3.21 s (SD = 0.54 s) (Fig 2F). This was measured by calculating the time difference from sending a command to the worm dispenser until the dispensed worm was detected hitting the arena floor in the video stream. Feeders were filled with worms each kept in a small compartment (15 in total) with food for gut loading the worms.

To keep animals motivated, smaller worms are more suitable. We therefore used T. molitor larvae as rewards. However, these worms may turn to pupa in experiments with low reward rate and long durations of a few days. In such cases, using young Zophobas atratus can extend larval stages [90]. In some experiments, a larger quantity of worms was needed between refills. This was solved by vertically and horizontally stacking the feeders (up to 30 worms in 2 vertically stacked feeders and 150 for 5 horizontally stacked feeders). We vertically aligned stacked feeders such that the top feeder released its reward through the release hole of the feeder below it. The experiment module tracked the number of available rewards in each feeder and determined which device should discharge a reward accordingly. After resupplying the feeders, researchers can notify the software remotely using the Web UI. Multiple feeders can be placed in the arena and they require no manual intervention except loading every few days.

Heat grid.

Twelve infrared halogen heat lamp bulbs (24 volts, 50 watts) were attached above the arena, arranged in an equally spaced grid of 3 rows by 4 columns. We attached 8 flat slotted steel bars to the center-top profile to hold the lamps. Lamp heating can vary between different manufacturers resulting in different ground temperatures. To control heating intensity, we modified lamp height by attaching 20 cm steel M6 spacer to the steel bars. The lamp was attached to a ceramic G6.35 socket using appropriate holders and screws. Lowering the lamps towards the arena floor, increased temperature and reduced the heating area of each lamp. The lamps were connected to a dedicated Arduino board through a 16-channel relay module board (S6B Fig). The relay board routed the output of a 12A, 24V DC power supply unit to specific lamps, which we used to simultaneously operate up to 2 lamps. We selected this high maximum current capacity to prevent potential problems caused by the high inrush current of halogen filaments. Another relay module (based on an Omron G5LE-14-DC5 5VDC SPDT relay) controlled the power supply unit output as an additional safety measure (S6A Fig). The grid enabled targeted heating of specific arena regions by more than 10°C, covering nearly the entire arena floor (Fig 3A).

Temperature sensors.

We used 2 plastic-covered digital DS18B20 temperature sensors to measure temperature conditions in the arena. One sensor was attached to the back wall of the arena and measured ambient temperatures. The second sensor was placed on the bottom of the opposite wall under the basking area heat lamp. The sensors were connected to one of the Arduino boards using a 1-Wire connection (S6A Fig).

General design.

The ReptiLearn software, written in Python, provides a toolkit for automating closed-loop behavioral tasks, collecting behavioral data, and extracting basic behavioral features. It includes a customizable real-time image processing pipeline that can be used to process and record synchronized video data from multiple cameras or other image sources. An arena controller program provides a generic way to integrate custom hardware components into the system without writing code by communicating with any number of Arduino microcontroller boards (S7 Fig). Controlling and monitoring the system can be done remotely through a web-based user interface (Web UI; Fig 1D). Users can implement new automated experiments by writing Python scripts and linking them to experiment sessions. These scripts can automate any part of the system based on real-time information gathered from arena sources. Non-programmers can customize existing scripts by modifying session parameters through the Web UI. A scheme of blocks and trials makes it possible to design complex experimental sequences.

Parallelism and synchronization.

To overcome limitations in Python’s concurrency model, the software makes extensive use of separate OS processes (S7 Fig). These processes are synchronized using a central state store, which holds the current state of all system components in one place. The Web UI and other external applications can receive updates whenever the state data changes by making a WebSocket connection to the system HTTP server. The server also provides an API to control the system remotely, and the MQTT protocol is supported for communicating with external devices and software (described below).

Supported operating systems and license.

The software can run on a wide range of operating systems thanks to Python’s cross-platform support. It was tested on Ubuntu 20.04, Ubuntu 22.04, and recent versions of Microsoft Windows and macOS. The software is licensed under the open-source GPL-3.0 license. Source code, detailed installation instructions, and guides for running and adapting the system are available at https://github.com/EvolutionaryNeuralCodingLab/reptiLearn.

Web-based user interface.

The Web UI is implemented as a separate JavaScript application using the React framework (code is available at /ui in the GitHub repository). After an initial build process (described in the “Getting Started” guide), the system HTTP server can be used to access it from any device that includes a modern web browser (Figs 1D and S8). It provides live video streaming from multiple sources and communicates with all other system components through a WebSocket connection and an HTTP API. The various features of the interface are described below in relevant sections.

Video acquisition software and image sources.

Image data from multiple sources can be acquired using the system for real-time processing and offline analysis (S9 Fig). ImageSource classes collect raw data from cameras or other imaging devices and make it available for further processing. Support for FLIR cameras and potentially other GenICam cameras are provided by the FLIRImageSource class using the Spinnaker SDK Python bindings. Allied Vision cameras are also supported through the AlliedVisionImageSource class, which utilizes the Vimba SDK. Additionally, video files can be used as sources for simulation and debugging purposes using the VideoImageSource class. This class also supports capturing images from standard webcams. Additional sources can be supported by writing new ImageSource classes and storing them in Python modules (inside the /system/image_sources folder).

Image observer.

ImageObserver classes provide an interface for further processing image data acquired by ImageSource classes. Each ImageObserver is attached to an ImageSource and is notified whenever a new image is acquired. Similarly to image sources, observer classes found inside the /system/image_observers folder are automatically recognized. The repository includes a YOLOv4ImageObserver class that performs object detection and can generate bounding boxes for objects or animals in the arena (the YOLO model weights are not included; see “Getting Started” guide). Additional simple observer classes are included as examples for implementing new processing algorithms. Multiple ImageObserver classes can be attached to the same ImageSource, and an additional VideoWriter observer is attached by default to each source for recording and encoding video. Both sources and observers can be added and configured using the Web UI “Video Settings” window (S8A Fig). When adding new sources and observers, all available classes found in their respective folders are listed, and users can modify classes and reload them without needing to restart the software. ImageObserver output can be accessed from the experiment script using a provided API.

Parallel image processing.

To improve performance and utilize multiple CPU cores or GPUs, each ImageSource and ImageObserver instance runs in a separate OS process and communicates with other components through shared memory buffers to avoid expensive data copying (S9 Fig). There is no limit to the number of sources and observers that can be used concurrently. Using this architecture, data from each image source can be simultaneously used for encoding video, streaming video over HTTP, and real-time processing by multiple algorithms with minimal latency.

Video encoding.

Video is encoded by the FFmpeg library using the ImageIO python library. Encoding profiles can be defined in the system configuration file and selected in the “Video Settings” window. For encoding using NVIDIA GPUs, we used the NVENC encoder; however, any encoder can be configured by setting FFmpeg parameters accordingly. Specific FFmpeg builds can be used by pointing the ImageIO library to a particular FFmpeg executable (see “Getting Started” guide). In addition to video files, each recording includes a CSV file containing a timestamp for each frame and a JSON file containing metadata about the recording. Timing data is supplied by the ImageSource for each frame and, in supporting sources, represents the camera exposure start time.

Camera synchronization.

Camera synchronization is accomplished by connecting the output pin of an Arduino board to the GPIO input of supporting cameras (S6C Fig). A trigger interface is provided by the arena controller for this purpose (see Arena controller) and does not require programming the Arduino manually. The video system automatically identifies an existing trigger and provides manual control through the Web UI. Additionally, the trigger is automatically paused for 1 s before starting a recording to simplify synchronization with external sources, such as electrophysiological data.

Arena controller.

Communicating with the various arena electronic components (described above) is done using the arena controller program (found in the /arena folder of the repository). The program is integrated into the ReptiLearn system; however, it can also be used standalone or even on a different computer. It maintains two-way communication between the rest of the system and any number of Arduino boards by relaying commands received over an MQTT connection to serial protocol over USB, as well as forwarding data received from Arduino boards over designated MQTT topics using a simple JSON-based protocol. A single Arduino program includes all the code necessary to operate a wide range of devices, avoiding the need to program the boards manually. The “Arena Settings” window in the Web UI provides an interface to configure the controller, identify connected boards, and upload the unified Arduino program to each board.

Each device connected to an Arduino board is controlled by an interface class. Several interface classes are implemented, and more can be added by implementing them using C++. For example, the LineInterface is used to control a single digital output (e.g., for switching light sources), the FeederInterface controls the automatic feeder (described above), and the TriggerInterface is used for sending TTL pulses at a selected frequency to synchronize camera acquisition. Each interface provides specific configuration parameters, has a current state value (e.g., a measurement or whether it is turned on or off), and can respond to multiple commands (see docs/arena_interfaces.md for more details).

Once configured, the ReptiLearn system integrates with the controller in several ways. The arena module (at system/arena.py) is responsible for executing the controller program on startup and provides functions for communicating with it, which can be used to control interfaces from experiment scripts. It also maintains a list of all current interface values in the state store, which is updated after each interface command is sent, and by polling the interfaces at a fixed interval (once a minute, by default). Additionally, it can be configured to store interface values (such as temperature sensor measurements) in CSV files or a database using a data logger (see data collection below). The Web UI provides an Arena menu (S8B Fig) for manual interaction with the controller, where individual interfaces are listed with their current state. It can also be used to control the execution of the controller program and to send commands to interfaces, for example, to trigger a reward feeder or toggle a digital output manually. Once the controller is configured, the same configuration file is automatically used to generate this menu.

Touchscreen interface.

A web application is included to display stimuli on screens and receive touch input into the system (at /canvas). The application uses the Konva.js 2d canvas JavaScript library and exposes large parts of its API over a bidirectional MQTT connection. The canvas module (/system/canvas.py) provides all the necessary functionality to communicate with the web application through the Canvas class. This architecture supports multiple screens by communicating with multiple web app instances, possibly running on different computers. Canvas classes can be used in experiment scripts to display various objects, such as shapes, images, or videos, and manipulate and animate properties of these objects (e.g., their position or color). Scripts can be notified of various events, such as screen touches of specific objects, video and animation progress, and other events supported by Konva.js. Example experiment scripts that use this module are available in the repository (see /system/experiments/canvas_shapes.py, for example).

Data collection.

The software provides several methods for collecting and storing data for offline analysis. When starting a new session, a session folder is created inside the session root folder (as defined in the configuration file), in which all video, image, and data files are stored. Data loggers, implemented by the DataLogger class (see /system/data_log.py), can be used to store data in CSV files as well as in TimescaleDB database tables, which can also be used to provide real-time visualization of data using third-party applications. Each data logger runs in a separate OS process to ensure that data collection does not interfere with other parts of the system.

Several specialized data loggers are provided: (1) An event logger (see /system/event_log.py) automatically keeps track of session events and can be further configured to log changes of specific state store values or incoming MQTT messages with specific topics. Additionally, experiment scripts can use the event logger to log any event relevant to the experimental paradigm. (2) ImageObserver loggers (ObserverLogger class) can be configured to collect the output of a specific observer (for example, to record animal position for each video frame). (3) One can define custom loggers within scripts using a generic API to record data from any source, such as screen touches or the position of objects on the screen (for example, see /system/experiments/canvas_video.py). An offline analysis module (at /system/analysis.py) provides classes and functions for analyzing the data stored in session folders, simplifying tasks such as finding video frames matching a specific session event or reading time-series data created by data loggers.

Failure recovery.

Ensuring the ability to quickly recover in the face of unavoidable interruptions, such as power or system failures, is crucial for the success of long-term experiments. Consequently, data loggers were designed to instantly save all gathered data to the session folder. The session state is periodically recorded in JSON format, triggered by significant events like the initiation and conclusion of trials and blocks. Furthermore, sessions can be paused and resumed at a later time without introducing any disruptions or complexities to the analysis process.

Creating and running automated experiments.

As described above, each system feature can be automated by implementing and using experiment scripts. The experiment module (at /system/experiment.py) includes an Experiment class that provides hooks for triggering code at various events: when the session is loaded (setup method), when a trial or block begins or ends (run_trial, end_trial, run_block, and end_block methods), and when the session is closed (release method). Sessions can be created and configured to run a specific script using the “New Session” dialog in the Web UI (S8C Fig). Any Experiment class found inside the experiments folder (at /system/experiments) is automatically listed and can be used in a session. Example scripts and scripts that were used in this study can be found in this directory of the GitHub repository.

Scripts can define default parameters, which can then be modified using the session section of the UI once a session is opened (S8D Fig). The session section also allows defining experiment blocks, each having different parameter values. Additionally, several built-in parameters are provided for controlling the number and duration of trials in each block, as well as block and inter-trial interval durations ($num_trials, $trial_duration, $block_duration, and $inter_trial_interval, respectively). Experiment classes can also define custom actions that can be triggered using the Web UI.

The system simplifies the process of developing and testing experiment scripts. Log messages generated by the code become instantly accessible within the Web UI, and updating the code following modifications does not necessitate a system restart. This results in a swift and efficient code-test-debug feedback cycle. Since ReptiLearn is developed purely in Python, scripts can access all of the system code and interface with additional Python packages as needed.

Task scheduling.

Controlling the timing of code execution is an integral part of running automated experiments. Consequently, the system provides several ways to schedule tasks. Scripts can utilize the schedule module (at /system/schedule.py) to set timers and trigger functions to run at specific times each day or at regular intervals. The asyncio python library is also supported and can be used for similar purposes. Additionally, task functions can be defined inside the tasks folder (at /system/tasks) to be scheduled manually using the Schedule menu in the Web UI.