Towards a more general understanding of the algorithmic utility of recurrent connections
Fig 4
Performance comparison of uniform and hybrid networks on the edge-connected pixel task.
(A) Schematic of a general hybrid network combining a recurrent and feedforward model. For any time-step t, the switching network combines the current output of the recurrent network yr and the output of the feedforward network yf to form the hybrid output yh. Note that the feedforward output can only be used by the switching network when t is greater than or equal to the number of layers in the model. (B) Performance of the best solution after hyperoptimization type plotted against the number of neurons to motivate why combining these two architectures into a hybrid networks can provide better performance across a range of computational times. The feedforward architecture (input augmented) provides a superior neuron/performance tradeoff at low computational time while the recurrent architecture (masked neighbors) provides a superior neuron/performance tradeoff at high computational time. Colors correspond to different network architectures. Small circles correspond to recurrent networks run for five timesteps and feedforward networks with five layers. Large circles correspond to architectures that use 25 timesteps or layers. Solid colored lines connect models of the same architecture with different layers or time steps. Dotted lines with arrows highlight the masked and feedforward architectures with equivalent layers or timesteps for ease of comparison. (C) Performance of hybrid architecture designed to be able to switch on-the-fly based on how many time steps are available to output a labelling. Here we consider the simplest switching network which can choose to either to output yr or yh at a given time step. The model combines the tag propagation solution and trained input augmented networks; the budget for each curve is the number of neurons the full model requires. The blue curve is the error profile for tag propagation alone while the orange curve shows the result of combining tag propagation with the 2-layer feedforward (abbreviated FF) model. The green curve shows tag propagation combined with the 10-layer feedforward network and the purple curve shows the combination with both the 2-layer and 10-layer networks. Note that the figure only shows path error; the feedforward solutions will also have some error on the distractor pixels. (D) Three distributions over computation time: the blue distribution is evenly split over short, medium, and long computation times (2, 5, 10, and 30 steps) and the orange distribution is evenly split over medium and long computation times (10 and 30 steps). (E) Performance of hybrid networks on three distributions of allowed computation time illustrated in panel D, meaning the fraction of runs the network is limited to a certain number of steps. The x-axis corresponds to the number of neurons allowed when constructing the hybrid solution. The first point on all curves allows only a fully recurrent architecture and thus is not hybrid. This is indicated by a square marker. The rest of the x-axis corresponds to a neuron budget allowing hybrid solutions with increased feed-forward composition. Additional hybrid models were only included in the plots if they improved the performance over models with fewer neurons (see Methods). (F) Per-pixel switching hybrid network learned from observing the output of the four networks (tag propagation; input augmented with 2, 5, and 10 layers) on 10,000 test samples. The heatmap shows the percent of times a given network correctly classified a pixel.