Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

A Novel Robot System Integrating Biological and Mechanical Intelligence Based on Dissociated Neural Network-Controlled Closed-Loop Environment

  • Yongcheng Li ,

    Contributed equally to this work with: Yongcheng Li, Rong Sun

    Affiliations State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, Liaoning, P. R. China, University of Chinese Academy of Sciences, Beijing, P. R. China

  • Rong Sun ,

    Contributed equally to this work with: Yongcheng Li, Rong Sun

    Affiliation Hefei National Laboratory for Physical Sciences at the Microscale, University of Science and Technology of China, Hefei, Anhui, P. R. China

  • Yuechao Wang ,

    ‡ These authors also contributed equally to this work.

    Affiliation State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, Liaoning, P. R. China

  • Hongyi Li ,

    ‡ These authors also contributed equally to this work.

    Affiliation State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, Liaoning, P. R. China

  • Xiongfei Zheng

    zhengxiongfei@sia.cn

    ‡ These authors also contributed equally to this work.

    Affiliation State Key Laboratory of Robotics, Shenyang Institute of Automation, Chinese Academy of Sciences, Shenyang, Liaoning, P. R. China

A Novel Robot System Integrating Biological and Mechanical Intelligence Based on Dissociated Neural Network-Controlled Closed-Loop Environment

  • Yongcheng Li, 
  • Rong Sun, 
  • Yuechao Wang, 
  • Hongyi Li, 
  • Xiongfei Zheng
PLOS
x

Abstract

We propose the architecture of a novel robot system merging biological and artificial intelligence based on a neural controller connected to an external agent. We initially built a framework that connected the dissociated neural network to a mobile robot system to implement a realistic vehicle. The mobile robot system characterized by a camera and two-wheeled robot was designed to execute the target-searching task. We modified a software architecture and developed a home-made stimulation generator to build a bi-directional connection between the biological and the artificial components via simple binomial coding/decoding schemes. In this paper, we utilized a specific hierarchical dissociated neural network for the first time as the neural controller. Based on our work, neural cultures were successfully employed to control an artificial agent resulting in high performance. Surprisingly, under the tetanus stimulus training, the robot performed better and better with the increasement of training cycle because of the short-term plasticity of neural network (a kind of reinforced learning). Comparing to the work previously reported, we adopted an effective experimental proposal (i.e. increasing the training cycle) to make sure of the occurrence of the short-term plasticity, and preliminarily demonstrated that the improvement of the robot’s performance could be caused independently by the plasticity development of dissociated neural network. This new framework may provide some possible solutions for the learning abilities of intelligent robots by the engineering application of the plasticity processing of neural networks, also for the development of theoretical inspiration for the next generation neuro-prostheses on the basis of the bi-directional exchange of information within the hierarchical neural networks.

Introduction

Robot intelligence has been investigated for several decades. However, robots that recognize, express emotion, perform logical reasoning like humans are still a far cry from realization. The challenge relates to the lack of a plastic developmental phase essential for biological neural network [16]. Therefore, scientists proposed connecting the biological and mechanical intelligence using a closed-loop method as an effective strategy to develop a powerful and intelligent controller for novel intelligent robots [7].

In order to employ additional biological capabilities, bioactuators with more effective driving performance were recommended in novel intelligent robots. In some studies, flagellated bacteria that generate collective fluid motion via chemical or electrical stimuli were used to provide the driving energy to robots [812]. Recently, dissociated muscle cells were used as effective actuators. The contraction of cardiomyocytes [1316] or skeletal muscle fibers [1720] controlled by electrical or optical stimuli has been reported to generate the driving force. The use of bioactuators in novel intelligent robots is feasible.

In the recent past, two preliminary solutions were reported for the integration of biological and mechanical intelligence. In addition to the integration of the mechanical with the living nervous systems, consisting of human brains [2123] and animal nervous systems [2426], the use of dissociated neural networks [2731] coupled to different robot systems received increased attention. We paid attention to the robot system coupled to cultured neural networks to execute specific tasks, but also the observe data transfer within the biological neural network. However, the dissociated neural network was not connected only to the mechanical equipment in our novel intelligent robots.

To our knowledge, several systems connected the dissociated neural network to the robotic agents. However, these earlier studies [30] used a hybrid neural-algorithmic controller rather than a neural controller. Studies reported adverse results due to lack of a learning phase (in Warwicks work [27], limited correct turning (67%) and chance level (50%). In Pizzis work [29], the percentages of correct turning was 42.86% with a chance level of 25%). In some studies, a modular interconnected neural network was employed to control a virtual robot, with a low-level performance (many hits) even after tetanus training [32].

In order to further integrate the biological and mechanical intelligence, we proposed the architecture of a novel intelligence robot, and built a closed-loop hybrid robot system interfacing with a population of dissociated neurons. In this paper, we developed flexible software and home-made hardware for the closed-loop experiments. Based on previous studies [33], the dissociated hierarchical neural networks provided a significant impact on the performance of this neural-robot hybrid system. So a complex hierarchical neural network was first used as the neural controller of robots in this paper. We successfully designed an actual mobile robot to move and eventually accomplish object-searching tasks efficiently. In the effective experiments, we designed an efficient experiment proposal to make sure the occurrence of plasticity of neural networks in response to the external environment, and found the performance of the robot system was enhanced with persistent training only due to the short-term plasticity. Finally, based on these designs, we further improved the performance of the neuro-robot hybrid system (average percentage of correct turning over 80% in both directions, chance level 33%). We also initially demonstrated the feasibility of novel intelligent robots.

Architecture of the neuro-robot system

The novel robot system proposed here drastically reduces the use of artificial components. It consists of a more effective neural interface, a flexible biological interface, powerful bioactuators, and artificial or biological agents. The representative architecture of this kind of novel mechanical-biological hybrid robot system is detailed in Fig 1. The dissociated neural network may serve as an integrated and flexible controller of the robot system. It not only represents a simplified model to investigate the underlying principles of neuroscience, but also simulates the functional specificity of brain [3436] with stronger information processing abilities. The electrical signals of neural network are acquired and processed. The output information from the neural network is decoded based on the rules of neural activities [1, 3, 5, 37]. Three types of agents are used as the actuators controlled by the decoded output. The first type includes the mechanical humanoid/general robot agents. The framework of this system is described in our paper. The second type includes bioactuators, which are embedded into the current robot system to effectively drive the robot. The third type involves the central or peripheral nervous system of the organism. We considered this challenge based on restoration of the functional disability and injury to the nervous system. A set of sensors was used to detect the agent environment as well as transmit the data to the information integration system. The neural network analyzes these data and transforms them into optical or electrical stimuli, which were used to generate the training strategy and effectively formulate the learning and memory phases of the neural network.

thumbnail
Fig 1. Architecture of the novel robot system.

The dissociated neurons are cultured on the multichannel electrode array (MEA). The multichannel recording system includes the multichannel amplifier and data acquisition card. The electrical stimulus is generated by the electrical stimulation generator system. The optical stimulus is generated by the optical modulation system including digital mirror device (DMD) and laser.

http://dx.doi.org/10.1371/journal.pone.0165600.g001

Current experimental framework

Based on the architecture of the novel robot system, we developed a neural-robot hybrid system including a differential mobile robot as the agent. Two parts constituted the overall closed-loop system, they were the main PC and slave PC components (Fig 2).

thumbnail
Fig 2. Framework of the neuro-robot hybrid system.

(i) Main PC components, employed to run the software tools, including the stimulator, the amplifier, the data acquisition card, and the MEA coupled to live neurons; (ii) Slave PC, hosting the robot control software to command the robot system. The main PC communicates with the slave PC via network port running the TCP/IP protocol. An offline analysis further analyzes the experimental data.

http://dx.doi.org/10.1371/journal.pone.0165600.g002

The main PC components included a standard 60-electrode multi-electrode array (MEA) with cultured hippocampal neurons. A culture with complex hierarchical construction was used. The cultured neurons exhibited unique morphology and were artificially divided into four interconnected components looked like four quadrants, so we designated it as‘4Q’ culture.

The hardware of main PC included: a MEA1060-Inv-BC amplification system (Multichannel Systems, MCS, Reutlingen, Germany) used to record the neural electrical signal from the MEA (multi-electrodes array) device; the home-made stimulating hardware, which could generate arbitrary electrical stimulation to the 60 electrodes on the MEA in real time; and a directly linked workstation, which conducted computationally expensive neural data analysis.

One part constituted the main PC software. We modified a toolbox written by Daniel Wagenaar (http://www.danielwagenaar.net/res/software/meabench/) and let it run on the directly linked workstation in a Linux operating system. This modified software, which included all the functions of the toolbox, also had new functions controlling the parameters of stimulation for the required data processing, such as communication with the robot control system, implementation of the coding/decoding, and short- term plasticity schemes.

The hardware of the slave PC comprised a separate workstation running the robot control interface and the robot system including a two-wheel differential robot and a camera. We compiled the software in the separate workstation turning into a GUI interface to control the robot. Various components of the framework communicated via TCP/IP sockets. This architecture enabled distributed processing of the computational loads via multiple machines throughout the internal network.

In addition to the real-time system, offline analyses including spike detection, spike sorting, firing rate calculation, spatiotemporal motif detection, and statistical analysis were written with MATLAB to further analyze the raw data recorded in the disk. These analyses were mainly intended to further demonstrate the altered plasticity of the neural controller in the neural-robot hybrid system.

Ethics Statement

This study was carried out in strict accordance with the recommendations in the Guide for the Care and Use of Laboratory Animals. The animal experimental facilities were approved by the Anhui Provincial Department of Science and Technology [Approval ID: SYXK (Anhui) 2005-004]. All protocols involving animal work were approved by the Ethics Committee of Animal Experiments of the University of Science and Technology of China (Permit Number: USTCACUC1201051). All surgery was performed under anesthesia. The adult pregnant rats were anesthetized with 99% carbon dioxide. After death by cervical dislocation, we removed the embryo from the uterus of the pregnant rat. The neurons were directly extracted from the hippocampus of embryo. All efforts were made to minimize animal suffering.

Culture preparation

A SU-8 mold was employed to creat the PDMS (polydimethylsiloxane) mask to culture the ‘4Q’ neural network with the hierarchical structure. The SU-8 mold was fabricated using UV lithography (Fig 3A). It consisted of two layer SU-8 photoresist (MicroChem). A 10μm-thick first layer SU-8 3005 photoresist was spin-coated on the cleaned Si wafer. The coated wafer was baked on the hot plate at 95°C for 2 min (soft bake). The SU-8 3005 photoresist was exposed to UV light under the photo mask containing a microchannel. After UV exposure, the coated wafer was baked on the hot plate at 95°C for 2 min (after exposure bake). Next, the exposed SU-8 photoresist was developed by SU-8 developer. After finishing the first layer, SU-8 2150 was spin-coated as the second layer with a thickness of 150μm and soft baked at 95°C for 30 min. The chamber photo mask was aligned to the microchannel patterned on the wafer followed by UV exposure and baked at 95°C for 15 min. Finally the SU-8 2150 wafer was developed by SU-8 developer.

thumbnail
Fig 3. Manufacture of the ‘4Q’ neural network with the tailor-made PDMS.

(A). schematic of PDMS device fabrication. (B). schematic of the PDMS device attached on ‘4Q’ MEA (C). physical map of the PDMS device attached on ‘4Q’ MEA [33].

http://dx.doi.org/10.1371/journal.pone.0165600.g003

The mixture of PDMS (polydimethylsiloxane) prepolymer and curing agent (10:1 (w/w), Dow Corning Corp) was poured on the SU-8 mold. PDMS was cast on the SU-8 mold and cured on the hot plate at a temperature of 80°C for 40min. After curing, the PDMS device was removed from the SU-8 mold and attached to the MEA platform (Fig 3B and 3C).

Cultures of embryonic rat hippocampus were prepared to produce the dissociated ‘4Q’ neural network. Low dense cultures of embryonic rat hippocampus were prepared as previously described [38] with minor modifications. Briefly, hippocampi were removed from embryonic day 17-18 (E17-18) rats and were treated with trypsin for 15 min at 37°C, followed by washing and gentle trituration. Dissociated cells were plated on the MEA chips with ‘4Q’ PDMS at a density of 50-80 cells per mm2. MEA chips were coated with poly-L-lysine to facilitate cell adherence. The culture medium comprised DMEM (BioWhittaker) supplemented with 10% heat-inactivated bovine calf serum (HyClone), 10% Hams F-12 with glutamine (Bio-Whittaker), 50 units ml penicillin streptomycin (Sigma), and 1×B-27 supplement (Invitrogen Gibco). Twenty-four hours after plating, one-third of the culture medium was replaced with a fresh medium supplemented with 20 mM KCl. Cytosine arabinoside (Sigma) was added to the culture dish (final concentration, 5 μM) around 7-10 days in vitro (div) to prevent overgrowth of glial cells. Cultures of 14 to 20 days were used in the experiments.

Home-made stimulator

A home-made stimulator was developed to provide real-time electrical stimulation. This home-made stimulator was developed around a micro-controller, which yielded digital value for digital-to-analog converter (DAC) and transmitted necessary control signals to other circuits. an arbitrarily defined biphasic voltage waveform with a time resolution of 3 μs and amplitude resolution of 12 bits could be generated by the accomplished stimulator for multi-electrodes simultaneously. We integrated the stimulator into the existing commercial recording system including MEA1060-Inv-BC amplification system and a data acquisition card (Multichannel Systems, MCS, Reutlingen, Germany). The whole hardware system is illustrated in Fig 4. In our work, a pair-pulse electrical stimulation with an interval of 50 milliseconds and an amplitude ±300 mv (negative first, 1.5 milliseconds in duration) was provided by this stimulator to the neural network to serve as coding information input.

Software of neural network signal processing

In order to control the whole electrical stimulation and recording system in real time, a modified software program based on an open source toolbox MEABENCH written by Daniel Wagenaar was developed. The software which ran on the Linux OS preserved most of the original functions including spike detection, artifact filter, recording, monitoring and so forth. The modified software also ran coding/decoding algorithms, processed the stimulation and training parameters, and communicated with the robot system. The signals of cultured neurons were processed adequately to support the robotic motion based on a set of command line and GUI utilities from the modified software.

Robot control system

The robot system utilized in our experiment included a two-wheeled differential mobile robot turning with a speed difference of two wheels, a camera monitoring the robot and objects, and a video acquisition card. We labeled on the robot with two colored papers to confirm its position, and offered the basic control via a GUI interface written with MFC. As illustrated in Fig 5, the GUI monitored the position of the robot and objects in real time, controlled the motion of the robot manually, and displayed the data exchange between the robot and the cultured neurons. In our experiment, we designed the neuro-robotic hybrid system to carry out the search-object task. Therefore, this interface was ultimately employed to transmit the relative position between the robot and the objects to the coding algorithm to decide the stimulation signal in real time and receive the motion commands from the decoding algorithm simultaneously.

Neural signal processing

Electrophysiological signals acquired from the MEA electrodes were preprocessed in order to remove the stimulus artifact and isolate spikes from the noise. Before spike detection, a first order band pass filter with cutoffs at 150Hz and 2.5KHz was used to filter most of high-frequency noise. The stimulus artifact suppression algorithm has been described in a previous study [39].

Spike detection

The spike detection algorithm always split the data stream into 10ms windows to perform a two-step test. Subsequently, spikes were detected whenever the absolute value of the voltage exceeded the current threshold, which was calculated from all the ‘no spike’ windows through a low-pass filter with a time constant of 100 windows (1 second if all were ‘no spike’). This algorithm adapted rapidly to changing noise conditions. The threshold was proportional to RMS (root mean square) noise and was calculated separately for each individual channel (typically four to six times RMS noise) before and during the actual experiment.

Decoding and coding strategy

Decoding scheme.

Based on previous theories of information propagation within the biological neural network [1, 4042], a rate-based algorithm [31, 43] was implemented as the decoding scheme in the hybrid system. In this scheme, we selected a group of electrodes (i.e., a subpopulation of neurons) on the MEA and defined them as the ‘output areas’ for each stimulus before commanding the robot to move. The number of spikes occurring over all the parts in 1000ms after stimulation were used to calculate the motion signal for the corresponding external information. In the current architecture, we built a binomial relation between robot direction and motor signal: if the evoked firing rate per second calculated from the defined left (or right) output area exceeded the firing rate from the other area with a threshold (80 per second in our experiments), the robot turned left (or right) immediately after the stimulation. However, in the absence of firing activities evoked by the electrical stimulation in the neural network, the robot received a ‘stop’ command. For the robot, the direction was therefore defined as:

(1)

Where Sl, Sr and So represented the total electrodes in the corresponding ‘output areas’ and ‘other areas’, respectively; fi or fj denoted the number of spikes in the ith or jth electrodes belonging to the corresponding areas, and t was the sampling time.

Coding scheme.

We initially defined two groups of electrodes as ‘input areas’ and assigned to the robot on the left and right sides of the objects. Based on different evoked firing activities, at least two electrodes were necessary to be confirmed to represent the ‘input areas’ (left and right). Subsequently, we defined the areas where the neural activities evoked by the corresponding electrodes were intense as the ‘output areas’. As long as the relative position between the robot and the objects was confirmed and delivered to the main PC, the coding algorithm translated the data received to command the home-made stimulator to generate a pair-pulse electrical stimulation with an amplitude ±300 mv into the appropriate areas.

Experimental design

The typical experimental protocol followed in this work included:

  1. Extraction of the dissociated neural cultures from the culture medium to the extracellular solution (ESC);
  2. Selection of the I/O areas by supplying a stimulus to a set of electrodes;
  3. Connection with the robot run for 10 minutes without training;
  4. Optimization of robot parameters after 2 to 4 min rest;
  5. A ten-minute run with training;
  6. Repetition of steps 4 and 5 at least twice.

During the first step, we changed the culture medium to the ECS so as to reduce the spontaneous activities of the neural network. In the second step of the experiment, we delivered a stimulus described above to each of the selected candidate electrodes every 30 seconds until we found at least two electrodes which were identified to be able to evoke the network firing. Based on the activities evoked by different electrodes, two or more electrodes were empirically selected as the ‘input areas’. These electrodes should try to evoke the firing activities which were different from each other. Areas that responded the most to stimulus from the corresponding input area were selected as the ‘output areas’ suggesting that one electrode/input area corresponded to one ‘output areas’. While each ‘output areas’ was confirmed, the remaining areas of neural network were defined as the corresponding ‘other areas’.

At the step 3, the robot commanded by the neural network moved free from any corner of the arena to the objects in the middle of the arena. Due to the refractory period of the neurons, at least a 20-second time interval was required between two consecutive stimulations. Probably, since we cultured the neurons with a low density (50-80 cells per mm2), and wanted to encode the neural activities using the neural reverberation [38], the refractory period was little longer than those work previously reported. At the step 4, we just adjusted the parameters including rotation angle, speed of robot and color threshold to keep the movement of robot stable. During step 5, a high-frequency stimulus, containing a series of pulses with an amplitude of ±300 mv, frequency of 20 Hertz and duration of 1000 milliseconds, was used to act as the training input as the high-frequency stimulus can effectively induce the plasticity in the neural network [4447]. The training input was provided to the corresponding ‘input areas’ of neural network when the neural network made a wrong turning decision. The high-frequency training inputs in this work were employed to play a key role on promoting the occurrence of plasticity of neural network, while the inputs just were used to serve as the test stimuli. The trial was not stopped if the robot reached the objects, but, the robot was reset to the start position to resume the trial. Step 6 was an effectively improved experimental proposal in contrast to previous studies. This step utilizing the continuous repeated training phases fully ensured the occurrence of the plasticity phase of neural network. Finally, the robot trajectory and the activities of the dissociated neural networks were recorded in the hard disk.

In this paper, 6 groups of successful experiment provided the sufficient dataset. Generally, an experiment from beginning to end took around 2 hours.

Results

Evoked activities in ‘4Q’ neural network

In order to simulate the animal brain to improve the information processing capacity, a PDMS template was developed to culture the neural network with a hierarchical structure. which was more complex in our neuro-robot system than those in previous studies. Since we had four components to be connected, that made the neurons in these components having more opportunities to generate different connection motifs in our work than those works previously reported. Hence, we called our neural networks ‘complex’. The image of this hierarchical neural network designated as ‘4Q’ is illustrated in Fig 6. The hierarchical network presented the diversity in various time scales with different stimuli like the functional specificity of brain. The spontaneous activities, which were inherent to the neural network [4851], showed a powerful potential in processing data in the scale of minutes in the ‘4Q’ neural network. At least two kinds of network bursting firing represented the pattern of information propagation in our ‘4Q’ neural network (Fig 7). Most of the neurons (the whole neural network) in the neural network fired in the red circles. However, only few neurons (subnetwork) in the neural network fired in the green circles suggesting that at least two different circuits in the ‘4Q’ neural network were used to process external information. Further evidence was presented in the evoked activities in the scale of hundreds of milliseconds. The neural network activities evoked by different stimuli appeared to be different in our experiments, with the firing spikes in Fig 8A and 8B different from each other, demonstrating that the network in the ‘4Q’ culture employed various neural circuits to respond to different data. We observed these firing phenomenons within all of 6 experiments.

thumbnail
Fig 6. Four-well neural network on ‘4Q’ MEA.

(A). We turned the dissociated neurons coupled to MEA into ‘4Q’ shape with a PDMS. The white raised dots (marked by the arrow) are the cell bodies, and black dots represent the electrodes. (B). One of the four quadrants is magnified to illustrate the details of neural network. The light raised dots indicate the cell bodies.

http://dx.doi.org/10.1371/journal.pone.0165600.g006

thumbnail
Fig 7. Spontaneous activity in the ‘4Q’ hippocampal neural network.

The activity of 59 electrodes is displayed. Each small vertical bar represents a spike. Each line represents an electrode. 190 seconds of activity is acquired from a 15-day dissociated neural network.

http://dx.doi.org/10.1371/journal.pone.0165600.g007

thumbnail
Fig 8. Activities evoked by different stimuli in the ‘4Q’ neural network.

Raw electrode data of the ‘4Q’ neural network are showed. Two seconds of activities evoked by the electrodes in the column 3 row 8 (A) and column 7 row 2 (B) in 59 electrodes were acquired from 15-day dissociated neural network. The augmented amplitudes recorded in these electrodes suggest a firing spike.

http://dx.doi.org/10.1371/journal.pone.0165600.g008

Closed−loop robot control

We built a closed-loop novel robot system using a differential mobile robot. This hybrid system was used to initially integrate biological and mechanical intelligence. The signal propagation of this hybrid system is shown in Fig 9. The camera in the robot system was used to confirm the relative position of the robot and objects. The position information was translated by the coding function into the corresponding electrical stimulus applied to the neural network in order to evoke the activities of neural network. After the neural signal was preprocessed, the decoding function extracted the decision of the neural network and output to command the robot movement. A high-frequency training stimulus described above (at section Experimental design) was provided to the neural network immediately as long as the robot took a wrong turning. The closed-loop hybrid neuro-robot system was employed to complete the object-searching task. The signal propagation of a successful example is illustrated in Fig 10. The average evoked spike firing rates in this experiment were 327 ± 33.58 (left areas, mean ± S.D.) and 495.29 ± 134.09 (right areas, mean ± S.D.), respectively. The firing rate of the right was greater than that of the left ‘output areas’. However, the stimulus from left input electrodes attenuated the spike firing rate of the right ‘output areas’ when turning left (two out of three). No correct left turning was observed in the absence of high-frequency training in this experiment suggesting that the stimulus from left input areas failed to strengthen the signal of left ‘output areas’ compared with that of ‘other areas’. However, after the addition of the high-frequency training phase, the successful rate of left turning reached a level that enabled the robot reach the objects (Fig 11) suggesting that training facilitated the neural network to use different circuits to respond to different data. This ‘trial and error’ learning process of the neural network reflected the application of biological intelligence.

thumbnail
Fig 9. Block diagram of signal propagation in the closed-loop hybrid neuro-robot system.

http://dx.doi.org/10.1371/journal.pone.0165600.g009

thumbnail
Fig 10. Signal propagation in the closed-loop hybrid neuro-robot system.

From top to bottom, the vertical bars in the Figs 1 and 2 represent the relative position of robot and objects; Figs 3 and 4 show the stimulus provided to the neural network; Figs 5 and 6 show the spike firing rates evoked by each electrical stimulus; and the vertical bars in the Figs 7 and 8 represent the turns executed by the robot.

http://dx.doi.org/10.1371/journal.pone.0165600.g010

thumbnail
Fig 11. Histogram of the correct turning percentage of the robot in an example.

Blue bar represents left turn (ratios are 0, 33%, 67%, and 67% sequentially), and red bar represents right turn (ratios are 77%, 67%, 83%, and 100% sequentially).

http://dx.doi.org/10.1371/journal.pone.0165600.g011

Learning ability of the neuro-robot system

In order to further demonstrate that the application of reinforced learning effectively improved the robot performance, we statistically calculated the correct turning ratios in both directions, in four different phases (no high-frequency training, first 10-min high-frequency training, second 10-min high-frequency training, and third 10-min high-frequency training). We compared the correct turning ratios in no high-frequency training phase to that of the other three high-frequency training phases to obtain the correct turning ratios in second 10-min and third 10-min training phases, which were significantly larger than in no high-frequency training phase (Fig 12, P<0.05, T-test). Interestingly, the correct ratios bidirectionally were counted, which suggested that the dissociated neural network simultaneously adapted to two different data inputs. This learning ability embodied in the overall robot’s performance was reflected by its frequency of reaching objects, which increased under experimental conditions (Fig 13). Meanwhile, we found that the number of tetanus stimuli decreased from the first training to third training phases (average were 7,4,2 for these three training phases, respectively).

thumbnail
Fig 12. Histogram of the correct turning percentage of the robot.

We calculate the statistics for the correct turning of the two directions in four different phases, respectively. Red bars are right turn (mean± S.D), and blue bars represent left turn (mean ± S.D). Students T-test for test, asterisks indicate the significant differences between two datasets, and the significance level = *p <0.05.

http://dx.doi.org/10.1371/journal.pone.0165600.g012

thumbnail
Fig 13. Box plot of arrival times in four different phases.

Box range: percentile 2575; box whiskers: percentile595; line: median.

http://dx.doi.org/10.1371/journal.pone.0165600.g013

Spike firing rates of ‘output areas’ and ‘other areas’ were statistically calculated, respectively, to further prove that the improvement of robot system was completely induced by the learning of ‘4Q’ neural network. The difference in value, denoted by the spike firing rate of ‘output areas’ subtracted from that of the corresponding ‘other areas’, obviously displayed occurrence of reinforced learning induced by the firing rate of neural network (Fig 14(A)). As illustrated in Fig 14(A), we found the negative values in difference value diminished from no high-frequency training phase to third 10-min high-frequency training phase suggesting that a wrong turn was avoided on the basis of the decoding function. Further, we drew the learning curve using the difference values to detail the occurrence of learning. A 2-minutes time window was used to count the difference values. The mean of difference values from all the experiments was calculated in each time window and illustrated in Fig 14(B). Although the difference values were not completely increased within one phase, which perhaps meant the neural networks always continuously adjusted themselves to adapt the stimuli during the experiments, the growth among the phases was significant. Especially, compared to the no training phase, the growth began to become significant in second 10-min training phase, and was gradually stable in third 10-min training phase. The analyses of changes in our hybrid system demonstrated that the dissociated ‘4Q’ neural networks were able to dynamically ‘recognize’ the different stimuli input to strengthen its learning abilities in order to eventually accomplish the specified task of the robot.

thumbnail
Fig 14. The variabilities of spike firing rate in the neural networks used for the hybrid neural-robots system.

(A). Box plots and histograms of spike firing rates in four different phases. Box range: percentile 25–75; box whiskers: percentile595; line: median. Students T-test for test, asterisks indicate the significant differences between two datasets, and the significance level = *p<0.05, ***p<0.001. (B). Variation curve for the difference values of spike firing rate. The difference values of spike firing rate in each 2-min time window are statistically calculated (mean ± S.E.M).

http://dx.doi.org/10.1371/journal.pone.0165600.g014

Discussion

Consistent with previous studies, especially Tessadori’s work [32], we found that the use of hierarchical neural network enables construction of obviously different neural circuits to increase the level of input data, with outstanding results (Figs 7, 8 and 10). Due to inadequate utilization of altered plasticity of dissociated neural network in previous studies, we used continuous and repetitive training phases to effectively improve the experimental results. We ensured altered plasticity and boosted the robot’s performance (Figs 11 and 12).

Increasing the network hierarchy allows the dissociated cultures to process more complex behavior. Nevertheless, we expected data representation would be enhanced with effective decoding/coding algorithms. The decoding and coding algorithms used in early experiments were rate-based, which may explain data processing within the biological neural network as well as an easier method used in the hybrid system. Despite obvious specific response of dissociated ‘4Q’ neural network in the firing rate scale, the variation caused by different frequency stimulation from the same spatial position may not be detected due to synchronous firing of the neural network. Therefore, a spatiotemporal decoding and coding algorithms on a scale of tens of milliseconds will be investigated to improve the resolution of the input/output data in the hybrid system.

In our studies, a training strategy with tetanic stimulation was used to demonstrate effective improvement in robot performance (Figs 12 and 13). Though the plastic developmental phase was eventually employed in our hybrid system, it lasted nearly 30 min for a satisfying performance. Therefore, in order to effectively elicit neural plasticity, a more productive learning scheme is needed. Considering the spatiotemporal decoding/coding algorithm simultaneously, the STDP (spike-timing dependent plasticity) learning mechanism detailed in [52, 53] is a promising candidate. The application of the STDP training phase is a useful method to investigate the mechanism of learning depending on specific stimulation protocols and more effectively enhance the learning efficiency.

Our work primarily integrates mechanical and biological intelligence systems. The final performance of the robot was worse than the result without including biological components in the closed-loop. Nonetheless, a long-term objective which is worth expecting for us is to employ neural plasticity in more complex hybrid systems to perform closed-loop experiments for effective biological, computational and learning properties even with relatively simple neural preparations. This neuro-robot framework also opens avenues for robotic applications in designing architecture (Fig 1). The neural controller was developed to be able to control the bioactuators and the living animals. The former system provides more powerful driving capabilities resulting in highly intelligent robots that leverage biological intelligence. The latter can be used to provide new hope for investigation into neural prosthetics to assist paralyzed persons and amputees.

Conclusion

We presented the architecture for a novel robotic system and successfully connected dissociated hierarchical neural networks originating in the hippocampus of embryonic rats to a mobile robot system in a bi-directional closed-loop. We successfully transformed the spike frequency into directions for the robot using a rate-based decoding strategy, and also translated the positional information of the robot and objects into electrical signals to stimulate the neurons. The robot system, comprising a mobile robot and a camera, was designed to move in a static arena to execute its tasks of finding the objects. In general, we demonstrated successful control of an external agent by a high-performance in vitro network of neurons. We introduced a neural network with a more complex hierarchy than previous studies based on a bi-directional interface. Our improved experimental approach successfully elicited a reinforced learning response of the ‘biological brain’. We preliminarily demonstrated successful improvement of robot performance in a hybrid system using biological intelligence.

Supporting Information

S1 Video. Video of improvement process in the robot’s performance.

This video of a differential robot run is running at 3 × real speed. The yellow papers in the middle of the arena are acted as the objects. In the no training phase, the robot controlled by the dissociated neural network makes no correct turn. The increase of correct turn occurs when the high frequency stimuli are added into the dissociated neural network. But there are still some mistakes existing in the first 10 minutes training phase. Further increase of correct turn is observed in the second 10 minutes training phase, which makes the robot closer to the objects as the experiment goes on. All of the improvement in the robot’s performance is independently caused by the plasticity of the dissociated neural network as demonstrated in the Fig 14.

doi:10.1371/journal.pone.0165600.s001

(WMV)

Acknowledgments

The authors want to thank the Prof. Guoqiang Bi and Prof. Pakming Lau in University of Science and Technology of China. We appreciate their useful instruction and support of the setup for our experiments. The work has been supported by the Research Equipment Development Program of the Chinese Academy of Sciences (YZ201545), National Natural Science Foundation of China (51305438), National High Technology Research and Development Program of China (863 Program 2015AA020312), and General Reserve Department of PLA (9140A26060214ZK63424).

Author Contributions

  1. Conceptualization: YCL RS YCW XFZ.
  2. Data curation: YCL.
  3. Formal analysis: YCL RS.
  4. Funding acquisition: XFZ HY.
  5. Investigation: YCL HYL.
  6. Methodology: YCL RS.
  7. Project administration: YCL YCW XFZ.
  8. Resources: XFZ YCW.
  9. Software: YCL.
  10. Supervision: XFZ HYL.
  11. Validation: YCL HYL XFZ.
  12. Visualization: YCL.
  13. Writing – original draft: YCL RS.
  14. Writing – review & editing: YCL.

References

  1. 1. van Pelt J, Wolters PS, Corner MA, Rutten WL, Ramakers GJ. Long-term characterization of firing dynamics of spontaneous bursts in cultured neural networks. Biomedical Engineering, IEEE Transactions on. 2004;51:2051–2062. doi: 10.1109/TBME.2004.827936. pmid:15536907
  2. 2. Maeda E, Robinson H, Kawana A. The mechanisms of generation and propagation of synchronized bursting in developing networks of cortical neurons. The Journal of Neuroscience. 1995;15:6834–6845. pmid:7472441
  3. 3. Shahaf G, Marom S. Learning in networks of cortical neurons. The Journal of Neuroscience. 2001;21:8782–8788. pmid:11698590
  4. 4. Jarosiewicz B, Chase SM, Fraser GW, Velliste M, Kass RE, Schwartz AB. Functional network reorganization during learning in a brain-computer interface paradigm. Proceedings of the National Academy of Sciences. 2008;105:19486–19491. doi: 10.1073/pnas.0808113105. pmid:19047633
  5. 5. Ikegaya Y, Aaron G, Cossart R, Aronov D, Lampl I, Ferster D, et al. Synfire chains and cortical songs: temporal modules of cortical activity. Science. 2004;304:559–564. doi: 10.1126/science.1093173. pmid:15105494
  6. 6. Bi GQ, Rubin J. Timing in synaptic plasticity: from detection to integration. Trends in Neurosciences. 2005;28:222–228. doi: 10.1016/j.tins.2005.02.002. pmid:15866196
  7. 7. Warwick K, Ruiz V. On linking human and machine brains. Neurocomputing. 2008;71(13):2619–2624. doi: 10.1016/j.neucom.2007.06.017.
  8. 8. Kim MJ, Breuer KS. Microfluidic Pump Powered by Self Organizing Bacteria. Small. 2008;4:111–118. doi: 10.1002/smll.200700641. pmid:18085723
  9. 9. Al-Fandi M, Jaradat M, Fandi K, Beech J, Tegenfeldt J, Yih T. Nano engineered living bacterial motors for active microfluidic mixing. Nanobiotechnology, IET. 2010;4:61–71. doi: 10.1049/iet-nbt.2010.0003. pmid:20726672
  10. 10. Martel S, Mohammadi M. A robotic micro-assembly process inspired by the construction of the ancient pyramids and relying on several thousand flagellated bacteria acting as micro-workers. In: Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJ International Conference on. IEEE; 2009. p. 426–427.
  11. 11. Steager EB, Sakar MS, Kim DH, Kumar V, Pappas GJ, Kim MJ. Electrokinetic and optical control of bacterial microrobots. Journal of Micromechanics and Microengineering. 2011;21:035001. doi: 10.1088/0960-1317/21/3/035001.
  12. 12. Francis NR, Sosinsky GE, Thomas D, DeRosier DJ. Isolation, characterization and structure of bacterial flagellar motors containing the switch complex. Journal of molecular biology. 1994;235:1261–1270. doi: 10.1006/jmbi.1994.1079. pmid:8308888
  13. 13. Kim J, Park J, Yang S, Baek J, Kim B, Lee SH, et al. Establishment of a fabrication method for a long-term actuated hybrid cell robot. Lab on a Chip. 2007;7:1504–1508. doi: 10.1039/b705367c. pmid:17960278
  14. 14. Chan V, Jeong JH, Bajaj P, Collens M, Saif T, Kong H, et al. Multi-material bio-fabrication of hydrogel cantilevers and actuators with stereolithography. Lab on a Chip. 2012;12:88–98. doi: 10.1039/C1LC20688E. pmid:22124724
  15. 15. Feinberg AW, Feigel A, Shevkoplyas SS, Sheehy S, Whitesides GM, Parker KK. Muscular thin films for building actuators and powering devices. Science. 2007;317:1366–1370. doi: 10.1126/science.1146885. pmid:17823347
  16. 16. Jiangzhong X, Schmidt JJ, Montemagno CD. Self-assembled microdevices driven by muscle. Nature Materials. 2005;4:180–184. doi: 10.1038/nmat1308. pmid:15654345
  17. 17. Sakar MS, Neal D, Boudou T, Borochin MA, Li YQ, Weiss R, et al. Formation and optogenetic control of engineered 3D skeletal muscle bioactuators. Lab on a Chip. 2012;12:4976–4985. doi: 10.1039/c2lc40338b. pmid:22976544
  18. 18. Cvetkovic C, Raman R, Chan V, Williams BJ, Tolish M, Bajaj P, et al. Three-dimensionally printed biological machines powered by skeletal muscle. Proceedings of the National Academy of Sciences. 2014;111:10125–10130. doi: 10.1073/pnas.1401577111. pmid:24982152
  19. 19. Nawroth JC, Lee H, Feinberg AW, Ripplinger CM, McCain ML, Grosberg A, et al. A tissue-engineered jellyfish with biomimetic propulsion. Nature biotechnology. 2012;30:792–797. doi: 10.1038/nbt.2269. pmid:22820316
  20. 20. Neal D, Asada HH. Engineered muscle systems having individually addressable distributed muscle actuators controlled by optical stimuli. In: Engineering in Medicine and Biology Society (EMBC), 2013 35th Annual International Conference of the IEEE. IEEE; 2013. p. 326–329.
  21. 21. Yanagisawa T, Hirata M, Saitoh Y, Kishima H, Matsushita K, Goto T, et al. Electrocorticographic control of a prosthetic arm in paralyzed patients. Annals of Neurology. 2012;71:353–361. doi: 10.1002/ana.22613. pmid:22052728
  22. 22. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, Caplan AH, et al. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature. 2006;442:164–171. doi: 10.1038/nature04970. pmid:16838014
  23. 23. Leuthardt EC, Schalk G, Wolpaw JR, Ojemann JG, Moran DW. A brainCcomputer interface using electrocorticographic signals in humans. Journal of Neural Engineering. 2004;1:63–71. doi: 10.1088/1741-2560/1/2/001. pmid:15876624
  24. 24. Santhanam G, Ryu SI, Byron MY, Afshar A, Shenoy KV. A high-performance brainCcomputer interface. Nature. 2006;442:195–198. doi: 10.1038/nature04968. pmid:16838020
  25. 25. Schwartz A, Cui X, Weber D, Moran D. Brain-Controlled Interfaces: Movement Restoration with Neural Prosthetics. Neuron. 2006;52:205–220. doi: 10.1016/j.neuron.2006.09.019. pmid:17015237
  26. 26. ODoherty JE, Lebedev MA, Ifft PJ, Zhuang KZ, Shokur S, Bleuler H, et al. Active tactile exploration using a brainCmachineCbrain interface. Nature. 2011;479:228–231. doi: 10.1038/nature10489. pmid:21976021
  27. 27. Warwick K, Xydas D, Nasuto SJ, Becerra VM, Hammond MW, Downes JH, et al. Controlling a Mobile Robot with a Biological Brain. Defence Science Journal. 2010;60:5–14. doi: 10.14429/dsj.60.11.
  28. 28. Novellino A, D’Angelo P, Cozzi L, Chiappalone M, Sanguineti V, Martinoia S. Connecting Neurons to a Mobile Robot: An In Vitro Bidirectional Neural Interface. Computational Intelligence and Neuroscience. 2007;2007:1–13. doi: 10.1155/2007/12725. pmid:18350128
  29. 29. Pizzi RMR, Rossetti D, Cino G, Marino D, Vescovi AL, Baer W. A cultured human neural network operates a robotic actuator. Biosystems. 2009;95:137–144. doi: 10.1016/j.biosystems.2008.09.006. pmid:18983888
  30. 30. Potter A, DeMarse TB, Bakkum DJ, Booth MC, Brumfield JR, Chao Z, et al. Hybrots: hybrids of living neurons and robots for studying neural computation. Proceedings of Brain Inspired Cognitive Systems. 2004;.
  31. 31. Bakkum DJ, Chao ZC, Potter SM. Spatio-temporal electrical stimuli shape behavior of an embodied cortical network in a goal-directed learning task. Journal of Neural Engineering. 2008;5:310–323. doi: 10.1088/1741-2560/5/3/004. pmid:18714127
  32. 32. Tessadori J, Bisio M, Martinoia S, Chiappalone M. Modular neuronal assemblies embodied in a closed-loop environment: toward future integration of brains and machines. Frontiers in neural circuits. 2012;6:1–16. doi: 10.3389/fncir.2012.00099. pmid:23248586
  33. 33. Li Y, Sun R, Zhang B, Wang Y, Li H. Application of Hierarchical Dissociated Neural Network in Closed-Loop Hybrid System Integrating Biological and Mechanical Intelligence. Plos One. 2015;10. doi: 10.1371/journal.pone.0127452. pmid:25992579
  34. 34. Jun SB, Hynd MR, Dowell-Mesfin N, Smith KL, Turner JN, Shain W, et al. Low-density neuronal networks cultured using patterned poly-l-lysine on microelectrode arrays. Journal of Neuroscience Methods. 2007;160:317–326. doi: 10.1016/j.jneumeth.2006.09.009. pmid:17049614
  35. 35. Erickson J, Tooker A, Tai YC, Pine J. Caged neuron MEA: A system for long-term investigation of cultured neural network connectivity. Journal of Neuroscience Methods. 2008;175:1–16. doi: 10.1016/j.jneumeth.2008.07.023. pmid:18775453
  36. 36. Feinerman O, Rotem A, Moses E. Reliable neuronal logic devices from patterned hippocampal cultures. Nature Physics. 2008;4:967–973. doi: 10.1038/nphys1099.
  37. 37. Xu S, Jiang W, Poo Mm, Dan Y. Activity recall in a visual cortical ensemble. Nature Neuroscience. 2012;15:449–455. doi: 10.1038/nn.3036. pmid:22267160
  38. 38. Lau PM, Bi GQ. Synaptic mechanisms of persistent reverberatory activity in neuronal networks. Proceedings of the National Academy of Sciences of the United States of America. 2005;102:10333–10338. doi: 10.1073/pnas.0500717102. pmid:16006530
  39. 39. Wagenaar DA, Potter SM. Real-time multi-channel stimulus artifact suppression by local curve fitting. Journal of Neuroscience Methods. 2002;120:113–120. doi: 10.1016/S0165-0270(02)00149-8. pmid:12385761
  40. 40. Raichman N, Volman V, Ben-Jacob E. Collective plasticity and individual stability in cultured neuronal networks. Neurocomputing. 2006;69:1150–1154. doi: 10.1016/j.neucom.2005.12.064.
  41. 41. Wagenaar DA, Pine J, Potter SM. An extremely rich repertoire of bursting patterns during the development of cortical cultures. BMC Neuroscience. 2006;7:11. doi: 10.1186/1471-2202-7-11. pmid:16464257
  42. 42. Sakamoto K, Kawaguchi N, Yagi K, Mushiake H. Spatiotemporal patterns of current source density in the prefrontal cortex of a behaving monkey. Neural Networks. 2015;62:67–72. doi: 10.1016/j.neunet.2014.06.009. pmid:25027732
  43. 43. Martinoia S, Sanguineti V, Cozzi L, Berdondini L, Van Pelt J, Tomas J, et al. Towards an embodied in vitro electrophysiology: the NeuroBIT project. Neurocomputing. 2004;58:1065–1072. doi: 10.1016/j.neucom.2004.01.167.
  44. 44. Fujii S, Saito K, Miyakawa H, Ito Ki, Kato H. Reversal of long-term potentiation (depotentiation) induced by tetanus stimulation of the input to CA1 neurons of guinea pig hippocampal slices. Brain research. 1991;555:112–122. doi: 10.1016/0006-8993(91)90867-U. pmid:1681992
  45. 45. Figurov A, Pozzo-Miller LD, Olafsson P, Wang T, Lu B. Regulation of synaptic responses to high-frequency stimulation and LTP by neurotrophins in the hippocampus. Nature. 1996;381:706–709. doi: 10.1038/381706a0. pmid:8649517
  46. 46. Lamsa K, Heeroma JH, Kullmann DM. Hebbian LTP in feed-forward inhibitory interneurons and the temporal fidelity of input discrimination. Nature Neuroscience. 2005;8:916–924. doi: 10.1038/nn1486. pmid:15937481
  47. 47. Hyman JM, Wyble BP, Goyal V, Rossi CA, Hasselmo ME. Stimulation in hippocampal region CA1 in behaving rats yields long-term potentiation when delivered to the peak of theta and long-term depression when delivered to the trough. The Journal of Neuroscience. 2003;23:11725–11731. pmid:14684874
  48. 48. Luczak A, Barth P, Marguet SL, Buzski G, Harris KD. Sequential structure of neocortical spontaneous activity in vivo. Proceedings of the National Academy of Sciences. 2007;104:347–352. doi: 10.1073/pnas.0605643104. pmid:17185420
  49. 49. Chiappalone M, Boue M, Vato A, Tedesco M, Martinoia S. Dissociated cortical networks show spontaneously correlated activity patterns during in vitro development. Brain research. 2006;1093:41–53. doi: 10.1016/j.brainres.2006.03.049. pmid:16712817
  50. 50. Feber Jl, Rutten WLC, Stegenga J, Wolters PS, Ramakers GJA, Pelt Jv. Conditional firing probabilities in cultured neuronal networks: a stable underlying structure in widely varying spontaneous activity patterns. Journal of Neural Engineering. 2007;4:54–67. doi: 10.1088/1741-2560/4/2/006. pmid:17409480
  51. 51. Chen C, Chen L, Lin Y, Zeng S, Luo Q. The origin of spontaneous synchronized burst in cultured neuronal networks based on multi-electrode arrays. Biosystems. 2006;85:137–143. doi: 10.1016/j.biosystems.2005.12.006. pmid:16533555
  52. 52. Bi Gq, Poo Mm. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. The Journal of Neuroscience. 1998;18:10464–10472. pmid:9852584
  53. 53. Rubinsky L, Raichman N, Lavee J, Frenk H, Ben-Jacob E. Spatio-temporal motifs remembered in neuronal networks following profound hypothermia. Neural Networks. 2008;21:1232–1237. doi: 10.1016/j.neunet.2008.06.008. pmid:18657391