Dictionary learning on cellular images

In this section, we use Dictionary Learning [39,41] (see Materials and Methods) to classify images of Dictyostelium cells as a function of the surface nano-topography (flat or ridged) and the presence or absence of the electric field. Each video is parsed into its constituent image frames and each frame is analyzed separately. Specifically, 300 randomly chosen patches are extracted from each image; each patch may or may not overlap with the other patches that originate from the same frame. Thus, in the absence of the electric field the data matrix X in Eq (2) consists of columns of length , corresponding to frames listed in Table 1. Because the matrix is so large, it is processed in batches of size 15. We choose dictionary atoms and set λ = 1 in Eq (3). The list of 100 predicted dictionary atoms is subsequently reduced to 30 by ranking all atoms by the reconstruction error: , where the sum runs over all the patches in the dataset and is given by in Eq (3), but with column removed from D and element removed from (j = 1…M). Only the atoms with 30 largest reconstruction errors are retained in the basis set (Fig 1). These images are conceptually similar to the filters learned during CNN training [47,48,54,56].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 1. Visualization of dictionary atoms as a set of filters for sparse image reconstruction.

Each dictionary atom is a image. Highlighted are the features corresponding to top 3 positive (red squares; cells on ridged surfaces) and negative (blue squares; cells on flat surfaces) SVM weights in Fig 3B.

https://doi.org/10.1371/journal.pone.0318036.g001

In order to demonstrate that the top 30 dictionary atoms can be used to reconstruct the entire image, we extract all possible overlapping patches from an image and represent each patch via sparse encoding in Eq (2) with . The values at each pixel in the final reconstructed image are obtained by averaging over the pixel’s value in all patches that contain that pixel. The result of the reconstruction is shown in Fig 2.

Next, we construct a measure that succinctly characterizes the entire image frame. Let us denote a set of vectorized image patches, where N is the total number of all overlapping patches extracted from a given frame. Then denote the corresponding set of sparse codes for these patches. We construct a feature vector v such that (k = 1…M), where denotes the component of the M-dimensional vector . A set of 30-dimensional feature vectors computed for a suitably chosen set of frames provides an input dataset for the Support Vector Machine (SVM) binary classifier [54,57].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 2. Sparse coding reconstruction of an image frame showing a single Dictyostelium cell.

Panel (A) shows the original image, panel (B) shows the same image reconstructed using sparse coding, and panel (C) shows a colormap of the pixel-by-pixel differences between the original and the reconstructed images. The frame was extracted from a video of a Dictyostelium cell on the flat surface in the absence of the electric field. Note that all image frames in this study are in size.

https://doi.org/10.1371/journal.pone.0318036.g002

We use the SVM classifier implemented in scikit-learn, with the penalty on the fitting weights and . The SVM scores are defined as

(1)

where v is the feature vector, w is the corresponding M-dimensional vector of weights, and b is the intercept [54]. Note that the components of v are non-negative in our case. First, we focus on differentiating between images of Dictyostelium cells moving on flat vs. ridged surfaces in the absence of the external electric field. In this case, the positive dataset is given by all image frames from 5 videos of cells on ridged surfaces and the negative dataset is given by all image frames from 12 videos of cells on flat surfaces (Table 1). Both positive and negative datasets are randomly divided into training and test sets in a way that prevents frames from the same video from appearing both in the training and the test set. Namely, we randomly divide the available videos into 3 subsets of  ( 2 , 1 , 2 )  videos for ridged surfaces and  ( 4 , 4 , 4 )  videos for flat surfaces. One of the 3 subsets then constitutes the test set, while the other 2 groups make up the training set. Note that this procedure results in 3 splits into test/training sets (labeled splits 1 , 2 , 3) depending on which of the 3 subsets is chosen as the test set. In the end, the test set consists of 4 videos on flat surfaces and 2 videos on ridged surfaces in split 1, 4 videos on flat surfaces and 1 video on ridged surfaces in split 2, and 4 videos on flat surfaces and 2 videos on ridged surfaces in split 3.

Finally, the SVM is trained on the training set and the resulting classification accuracy on the test set is represented by the confusion matrix in Fig 3A (see S3A and S4A Figs for the classification results on the other two training/test splits; all confusion matrices shown in this work correspond to the test data unless explicitly indicated otherwise). We observe that the set of dictionary atoms shown in Fig 1 leads to high SVM classification accuracy. The top 3 dictionary atoms that contribute to differentiating images of cells on flat vs. ridged surfaces are highlighted in Fig 1. The top 3 atoms that are found more frequently in videos on ridged surfaces all appear to contain horizontal or nearly horizontal bands. In contrast, the top 3 features from videos on flat surfaces contain inclined edges and, in one case, a nearly uniformly lit square with a black vertical bar to one side. This is consistent with a previous observation that cells on nano-ridges exhibit one-dimensional actin waves guided along the horizontally-oriented ridges, in a process known as esotaxis, whereas cells on flat substrates are characterized by two-dimensional actin waves [30].

Download:

• PNG
  larger image
• TIFF
  original image

Fig 3. SVM classification of feature vectors obtained by dictionary learning.

The features characterize images of Dictyostelium cells moving on flat vs. ridged surfaces in the absence of the external electric field. Shown are results for training/test set split 1. Panel (A) is the confusion matrix for the SVM classification into two nano-topography types (flat/ridged surfaces). The true labels are on the y-axis in all confusion matrices presented in this work. Panel (B) shows SVM weights assigned to each dictionary atom. The dictionary atoms corresponding to the top three positive and negative SVM weights are highlighted in Fig 1; the 1–30 numbering on the x-axis corresponds to the rows of atoms in Fig 1, starting from the upper left corner. Note that dictionary atoms characterized by positive/negative SVM weights describe features of cells on ridged/flat surfaces, respectively. Panel (C) is a histogram of SVM scores (Eq (1)) evaluated on all video frames from the training set; panel (D) is a histogram of SVM scores (Eq (1)) evaluated on all video frames from the test set.

https://doi.org/10.1371/journal.pone.0318036.g003

To analyze which atoms contribute the most to the classification, we plot the components of the weight vector w in Fig 3B. Note that positive weights correspond to the atoms that are more prominent in the image frames obtained from the videos on ridged surfaces. Likewise, negative weights correspond to the atoms that are found more readily in the image frames from the videos on flat surfaces. Finally, we show histograms of SVM scores for the training set (Fig 3C) and the test set (Fig 3D). As expected, the separation between SVM scores corresponding to the flat and ridged surfaces is higher for the training set than for the test set. However, even in the latter case the scores are well-separated, consistent with the high classification accuracy observed in Fig 3A. The corresponding results for training/test splits 2 and 3 are shown in S3B–S3D Fig and S4B–S4D Fig, respectively. We also note a high degree of reproducibility between SVM weights obtained on splits 1 and 2 (compare Fig 3B and S3B Fig), although the sets of SVM weights obtained on splits 1 and 3 are somewhat less correlated (compare Fig 3B and S4B Fig).

Classification of individual image frames into flat or ridged can be used to classify entire videos containing these frames by majority voting. In the split 1 test set, no frames depicting cells on flat surfaces were misclassified, while for the two videos of cells on ridged surfaces, 4 out of 31 frames were misclassified in one video and 12 out of 122 frames were misclassified in the other (Fig 3A). Thus, for split 1 the accuracy of video classification is 100% by majority voting. Similarly, the split 2 test set produces 100% accurate classification of each frame and therefore of entire videos (S3A Fig). In the split 3 test set, 26 out of 60 frames were misclassified for one of the two videos of cells on ridged surfaces and all other frames were classified correctly, again yielding 100% accurate video classification. Visual inspection revealed that the video with misclassified frames appears to have fewer horizontal features that are characteristic of cells on ridged surfaces. Despite this, the overall video accuracy classification by majority voting is 100% on all test sets.

Since the nano-ridges are oriented horizontally in all image frames (Materials and Methods), it is conceivable that our classification is aided by the advance knowledge that the ridge orientation is fixed in one direction, which would not be the case in more realistic microcellular environments. To investigate this issue, we have created an enlarged dataset in which the original images were combined with the same images rotated counterclockwise by , , and . The dataset created in this way contains both horizontally and vertically oriented ridges, potentially making the classification a more challenging task. However, we do not observe a significant loss of accuracy on this combined dataset, indicating that dictionary learning is sufficiently flexible to accommodate the loss of knowledge of the nano-ridge orientation (S5 Fig). Note that the most informative features for differentiating between flat and ridged surfaces have changed (cf. red and blue boxes in Fig 1 and S5C, S5F, S5I Fig), consistent with the fact that dictionary learning produces a set of adaptive, data-dependent visual features. The new features tend to be more oblique than before, probably to account for the mixture of vertical and horizontal nano-ridge orientations.

Next, we divided all videos in Table 1 into two groups: 15 videos on flat surfaces and 9 videos on ridged surfaces, regardless of the presence or absence of the external electric field. These videos were randomly split into 3 subgroups of  ( 5 , 5 , 5 )  videos for flat surfaces and  ( 3 , 3 , 3 )  videos for ridged surfaces, again ensuring that no frames from the same video appear in both the training and the test sets. The subgroups of videos were used to construct 3 equivalent training/test splits as before, with one subgroup in the test set and the other two in the training set. The SVM classifier was trained on these data as described above, using the same scikit-learn implementation and the same parameter settings.

The resulting confusion matrix indicates a significant loss of accuracy when the videos of cells moving under the influence of the external electric field are included into the dataset (S6 Fig). The lack of accuracy is even more pronounced when SVM classification is done only on cells in the external electric field (S7 Fig); in this case, the videos were randomly split into subgroups of  ( 1 , 1 , 1 )  for flat surfaces and  ( 1 , 1 , 2 )  for ridged surfaces, with 3 equivalent training/test splits constructed as before. Overall, these results show that the set of dictionary atoms shown in Fig 1 can be used to differentiate between cells moving on flat vs. ridged surfaces in the absence of the electric field, but not when the field is on. Apparently, the dependence of actin wave shapes on the nano-topography type diminishes or disappears when the external electric field is present.

Finally, we have checked whether SVM classification on dictionary atom-based feature vectors can be employed to distinguish cells in the presence vs. absence of the external electric field, regardless of the nano-topography type. In this case, the videos are randomly subdivided into groups of  ( 2 , 2 , 3 )  with the electric field and into groups of  ( 5 , 6 , 6 )  without the electric field; the rest of the procedure is as described above. The classification accuracy is also rather low in this case (S8 Fig), confirming our previous conclusion that the set of dictionary atoms in Fig 1 does not provide an adequate basis for classifying cellular images along the electric field axis.

Application of scattering transforms to cellular images

For each video frame in the original videos, we compute zeroth- and first-order scattering coefficients: , where I represents the set of pixel values in the original 2D image (see Materials and Methods for details). Altogether, there are filters, where J = 4 is the total number of scale parameters and is the total number of angular orientation parameters. Applying a zeroth- or first-order scattering transform to the original M × M image results in an output image of size . Since in our case , the output images are pixels. The set of wavelet filters in the scattering transform is designed to capture the presence of various features in the images. If the feature corresponding to a filter is present in the image, it is highlighted in the output image after applying that filter (see S9 and S10 Figs for representative examples).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 4. SVM classification of scattering transform feature vectors.

The feature vectors are obtained from the images of Dictyostelium cells moving on flat vs. ridged surfaces in the absence of the external electric field. Shown are results for the training/test set split 1. Panel (A) is the confusion matrix for the SVM classification into the two nano-topography types (flat vs. ridged surfaces). Panel (B) shows SVM weights assigned to each scattering transform; the 1–49 numbering on the x-axis corresponds to the zeroth-order transform followed by the first-order transforms in the order displayed in S12 Fig. Note that positive/negative SVM weights describe features of cells on ridged/flat surfaces, respectively. Panel (C) is a histogram of SVM scores (Eq (1)) evaluated on all video frames from the training set; panel (D) is a histogram of SVM scores (Eq (1)) evaluated on all video frames from the test set.

https://doi.org/10.1371/journal.pone.0318036.g004

In order to obtain feature vectors for image classification, we construct an n-dimensional vector v with components (). Here, denotes the value of a pixel  ( i , j )  in the output image obtained after applying wavelet filter α to the original image. After collecting feature vectors for all image frames in a given dataset, we carried out binary SVM classification as was done before for dictionary learning. Namely, we used the SVM classifier implemented in scikit-learn, with the L₁ penalty on the fitting weights and C = 5000. First, we focused on classifying the videos by nano-topography type (flat or ridged) in the absence of the electric field. We used the same 3 training/test set splits as in the dictionary learning section. The results of the SVM classification on split 1 are shown in Fig 4A and on splits 2 and 3 in S11 Fig. We observe that the classification accuracy is somewhat lower than than obtained with dictionary learning. Note that all components of the feature vector v are non-negative due to the absolute magnitude operations in Eq (6). Thus, wavelet features corresponding to positive SVM weights are more prominent in the video frames from ridged surfaces, whereas wavelet features corresponding to negative SVM weights are more prominent in the video frames from flat surfaces (Fig 4B). The separation of SVM scores is excellent in the training set (Fig 4C) but less good in the test set (Fig 4D), leading to the misclassification of a subset of video frames on flat surfaces observed in Fig 4A. Examination of wavelets corresponding to top 3 positive and negative SVM weights (S12 Fig) shows that cells on ridged surfaces are characterized by horizontal or nearly horizontal features, whereas cells on flat surfaces are characterized by vertical or nearly vertical features. This is consistent with the dictionary learning analysis.

Similar to dictionary learning, we used classification of individual image frames to label entire videos by majority voting. For the split 1 test set, for videos of cells on flat surfaces, 3 out of 30 frames were misclassified in one video, all 31 frames were misclassified in another video, and no misclassification was observed in the remaining 2 videos. For videos of cells on ridged surfaces, 100% of the image frames were classified correctly. Thus, 5 out of 6 test set videos were classified correctly. For the split 2 test set, the only misclassification instance was in one video of cells on ridged surfaces, where 30 out of 120 frames were misclassified. Thus, all 5 videos are classified correctly by majority voting. Finally, in the split 3 test set significant misclassification was observed in both videos of cells on ridged surfaces (all 62 frames were misclassified in one video and 35 out of 60 frames were misclassified in the other video) and no misclassification was observed elsewhere, leading to 4 out of 6 videos classified correctly. This is again due to the relative lack of horizontal wave features in the two videos in question; it appears that scattering transforms are more sensitive to this than dictionary learning. We conclude that the classification accuracy of entire videos is somewhat lower with scattering transforms compared to dictionary learning.

As with dictionary learning, we have extended SVM classification into flat vs. ridged surfaces by including the frames from the videos with the electric field turned on into training and test sets. As expected, doing so has made the classification accuracy significantly worse (S13 Fig). The accuracy decreases again when only videos with the electric field present are considered (S14 Fig), showing that, similarly to dictionary learning, scattering transforms cannot be used to distinguish cells moving under the influence of the electric field by the nano-topography type. This conclusion is supported by the fact that classification of the cell videos by the presence or absence of the external electric field also shows relatively low accuracy (S15 Fig).

Next, we consider the product of the vector of weights and the feature vector in the SVM scoring function: , where is the SVM-reconstructed image: the sum of n output images from the scattering transform, weighted by their SVM weights. Two representative examples of for cells moving on flat or ridged surfaces in the absence of the electric field are shown in Fig 5. Since , ∀ ⁡ α, the negative values of , shown in blue in Fig 5B, 5D, identify pixel positions corresponding to flat-surface features. Conversely, the positive values of , shown in red in Fig 5B, 5D, identify pixel positions corresponding to the features that are more prominent in ridged-surface frames. Note that the ridged-surface (red) features tend to be horizontal whereas the flat-surface (blue) features tend to be vertical in these examples, in line with the overall findings described above.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 5. SVM-reconstructed cell images.

The cells move on the flat (A,B) and ridged (C,D) surfaces, in the absence of the electric field. (A): A representative frame from a flat-surface video, (B): the corresponding SVM reconstruction . (C): A representative frame from a ridged-surface video, (D): the corresponding SVM reconstruction .

https://doi.org/10.1371/journal.pone.0318036.g005

Download:

• PNG
  larger image
• TIFF
  original image

Fig 6. Angular plot of the SVM weight vector components for the Morlet wavelet filters.

The filters are classified by different scales j = 1…J (J = 4) and angular orientation indices ℓ = 1…L (). Each scale j corresponds to a separate panel (A)–(D). The angular orientation indices label angles between and in increments (cf. Materials and Methods). The inner and outer circles show negative and positive SVM weights, respectively.

https://doi.org/10.1371/journal.pone.0318036.g006

Since the wavelet filters are labeled by the scale index j and the angular orientation index ℓ which labels angles in the range, it is convenient to display the SVM vector components on an angular plot (Fig 6). We observe that positive SVM weights, which are statistically more prominent in the frames from videos of cells on ridged surfaces, tend to correspond to angular orientations in the range and thus to horizontal and nearly horizontal features (S12 Fig). In contrast, negative SVM weights that characterize cells on flat surfaces correspond to more vertical features.

Finally, as was done earlier with dictionary learning, we checked the scattering transform performance on the enlarged dataset in which the original image frames were combined with the same images rotated counterclockwise by , , and (S16 Fig). In contrast to dictionary learning, we observe an appreciable loss of performance (cf. Fig 4A and S11 Fig on the one hand and S16A, S16D, and S16G Fig on the other). Apparently, the fixed set of features provided by the scattering transform is less suitable in this case than the adaptive set of dictionary learning features. The top features picked out by the SVM classifier are no longer vertical or horizontal—similarly to dictionary learning, the most discriminative features are oblique, likely to account for the mixture of horizontal and vertical nano-ridge orientations (S16 C, S16F, and S16I Fig).

Classification using optical flow and tracking algorithm

We have implemented corner tracking using Lucas-Kanade optical flow algorithm (see Materials and Methods for details). We have found that tracking corners with optical flow is preferable to tracking all image pixels, allowing us to focus on a relatively sparse set of well-defined actin wave features. We observe that cells exhibit distinct patterns of motion with and without the external electric field (Fig 7; S1 and S2 Movies): there is no preferred direction of motion without the electric field (Fig 7A–7C), in contrast to the drift towards the left with the electric field (Fig 7D–7F). Therefore, we have focused first on the binary classification along the electric field axis, irrespective of the nano-topography type.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 7. Mapping cell motion caused by the application of the external electric field: corner tracking and optical flow in videos of Dictyostelium cells.

(A): The first frame from a video depicting cell motion on a flat surface with no external electric field. Colored circles indicate corners found using the Shi-Tomasi corner detection algorithm with the quality level α = 0 . 4 (Materials and Methods), resulting in 28 corner points detected. (B): The middle frame from the same video. (C): The last frame from the same video. Colored lines in (B) and (C) indicate optical flow trajectories for the set of corners shown in (A). (D): Same as (A), but for the first frame from a video depicting cell motion on a flat surface in the presence of the external electric field pointing to the left. Colored circles indicate corners found using the Shi-Tomasi corner detection algorithm with the quality level α = 0 . 4 (Materials and Methods), resulting in 28 corner points detected. (E): Same as (B), but for the middle frame from the video in (D). (F): Same as (C), but for the last frame from the video in (D). Colored lines in (E) and (F) indicate optical flow trajectories for the set of corners shown in (D). Each video frame is in size. Full videos of the cell motion and optical tracking are available as Movies S1 and S2.

https://doi.org/10.1371/journal.pone.0318036.g007

Each video in our dataset yields a set of optical flow velocity vectors , one for each tracked corner and each pair of consecutive frames. In order to summarize these velocities in a low-dimensional representation, we compute and by averaging and over all the tracked points and all pairs of consecutive frames in a given video. We also compute . We observe that even by itself, the average velocity magnitude is an informative descriptor for separating videos with and without electric field (Fig 8A). Therefore, we construct a 3D feature vector for each video and use a collection of these vectors in an SVM binary classifier. Our dataset includes both flat and ridged surfaces and in total consists of 17 videos in the absence of the electric field and 12 videos in the presence of the electric field (Table 1).

We divide all videos randomly into training and test sets in the 2 : 1 ratio (or the closest approximation to it); this procedure is repeated 3 times, resulting in 3 equivalent splits into test/training sets (labeled splits 1 , 2 , 3). As before, we use the SVM classifier implemented in scikit-learn, with the penalty on the fitting weights and . We find that, as expected, classifying videos by the presence or absence of the external electric field using optical flow velocities as input yields reasonably accurate results, with 90% classification accuracy in split 1 (Fig 8B) and 80%, 100% classification accuracy in splits 2 and 3, respectively (S17 Fig). Interestingly, and consistent with Fig 8A, using as a sole feature results in the SVM classification accuracy that is nearly as high: 90%, 90% and 80% for splits 1,2 and 3, respectively. This is noteworthy because, although the electric field is applied horizontally in all the videos, v is independent of the image rotation – it is the non-zero magnitudes of the velocities and not their orientations that enable the successful detection of the electric field.

Download:

• PNG
  larger image
• TIFF
  original image

Fig 8. SVM classification of optical flow feature vectors.

The feature vectors were obtained from the videos of Dictyostelium cells moving in the absence vs. presence of the external electric field, regardless of the nano-topography type. (A): A Histogram of average velocity magnitudes for all the videos in our dataset. (B): Confusion matrix and classification accuracy for 10 videos in the test set (training/test set split 1). (C): Plot of the mean classification accuracy versus the number of sub-video frames n. The green vertical lines denote standard deviations.

https://doi.org/10.1371/journal.pone.0318036.g008

Using entire videos for classification along the electric field axis results, for any given training/test split, in only 19 videos in the training set and 10 videos in the test set. We sought to increase the number of datapoints in both sets by subdividing each original video into smaller videos with a fixed number of frames. Specifically, we divided each original video into consecutive n-frame segments starting from the first frame; in the cases where the total number of frames was not exactly divisible by n, we also added the last n frames from each video to the dataset. For videos which have fewer than n frames, we added the entire video to the dataset. The resulting set of sub-videos was divided randomly into training and test sets in the 2 : 1 ratio (or the closest approximation to it); this procedure was repeated 8 times, resulting in 8 equivalent splits into test/training sets.

Using the same strategy as before, we computed a 3D feature vector s for each sub-video and carried out binary SVM classification, with the same settings as above, on the presence or absence of the external electric field. We find that the mean classification accuracy, averaged over the 8 splits, is high for longer videos () and becomes markedly lower when the videos are shorter () (Fig 8C). With , the average classification accuracy is 76.8% when is the sole classification feature, only a little worse than using the full 3D feature vector s. Thus, the magnitudes of the corner velocities are again the primary factor in detecting the motion of the cell under the influence of the external electric field. Overall, we conclude that optical flow-based feature vectors can be used to reliably detect the presence of electric fields from the patterns of actin wave motion in Dictyostelium cells. The detection can be based either on entire videos or on shorter sub-videos; there is no appreciable degradation of the performance, as long as the sub-videos contain sequences of at least 60 consecutive frames.

In contrast, the optical flow-based feature vectors cannot be used to predict the surface type (flat or ridged) when the videos both with and without external electric field are mixed together in the dataset. Using 3 random training/test splits in the approximately 2 : 1 ratio, we obtain a relatively low classification accuracy by the nano-topography type (S18 Fig). Thus, optical flow analysis is complementary to single-frame analysis employing either dictionary learning or scattering transforms.

Video microscopy dataset

All the data used in this work is available in our previous publication [29], which provides full experimental details of electrofusing giant cells, surface preparation, and electrotaxis. S19 Fig provides an overview of the experimental setup used to collect the video microscopy data. Briefly, the following procedures were employed:

Cell lines: Aggregation adenylyl cyclase null (ACA-) -LimE RFP cell lines, which do not have chemotaxis signal relay and express filamentous actin biosensors, were grown axenically in HL5 buffers. Before imaging, cells were washed twice with 17 mM Sorensen buffer (15 mM KH₂PO₄ and 2 mM Na₂HPO₄) to remove nutrients, and rolled in a 15 mL conical tube for 30 minutes before electrofusion. Afterwards, cells were seeded into an electrotaxis chamber for 2 hours before experiments.

Surfaces: Both flat and ridged nano-topographies were fabricated using the same photopolymerizable resin. The ridged nano-topography surfaces were covered with 1.6 μm-spaced nano-ridges at 1 μm height; the ridges were aligned horizontally with respect to the image frame (Fig 9).

Download:

• PNG
  larger image
• TIFF
  original image

Fig 9. Schematic representation of the ridged nano-topography surface and its alignment with the image frame.

https://doi.org/10.1371/journal.pone.0318036.g009

Electrotaxis: 20 V/cm DC electric field was applied to our customized 3D-printed electrotaxis chamber (dimension: 20 mm  ×  5 mm  ×  0.25 mm). The field always pointed horizontally in Fig 9, with several right-to-left reversals in the course of each experiment. Agar bridges were used to minimize toxic electrochemical products and maintain a stable pH environment.

Spinning disk microscopy: Time-lapse images of the RFP channel were recorded using a PerkinElmer spinning-disk microscope at a frame rate of 0.1 frames per second.

Video dataset: Our dataset consists of videos of giant electrofused Dictyostelium cells moving on flat or ridged surfaces, in the presence or absence of the external electric field. Each video frame in this study is in size. In the presence of the electric field, there are 3 videos of cells on flat surfaces, with 444, 460, and 126 frames; and 4 videos of cells on ridged surfaces with 675, 348, 699, and 656 frames. In the absence of the electric field, there are 12 videos of cells on flat surfaces and 5 videos of cells on ridged surfaces (the number of frames in each video is listed in Table 1). Altogether, we have 4551 video frames in our dataset. Each video frame is a grayscale image of the size pixels.

Note that the original videos with the electric field turned on contain several reversals of the electric field direction; after each change, there is a small time lag between the onset of the electric field and the onset of the cellular response to the electric field. Therefore, we have additionally subdivided each of these videos into sets of continuous frames where the cell motion has stabilized in response to the electric field with a fixed magnitude and direction. Each resulting sub-video has the number of frames ranging from 30 to 181; for videos without electric field, no subdivision was necessary and therefore the original video sequences were used (Table 1). The sub-videos listed in Table 1 were used for the optical flow analysis; the longer original videos were used for the dictionary learning and the scattering transform analysis.

Dictionary learning

Dictionary learning (sparse coding) is a computational method which aims to find a sparse representation of data vectors in terms of a set of basis vectors that are learned from the data itself [39,41]. The idea is to reconstruct each data vector as faithfully as possible while employing very few active (i.e., non-zero) elements in the basis set.

Let us define a d × N data matrix X, where each of N columns of X represents a d-dimensional data vector . Here, for an image (or an image patch) of the size pixels. The goal of dictionary learning is to factorize the data matrix X as follows:

(2)

where D is a d × M dictionary matrix with each column representing one of M d-dimensional dictionary atoms, or basis vectors. S is an M × N matrix where each column contains the sparse code representation for the data vector .

Dictionary learning is implemented by defining the following objective function:

(3)

where the first term is the penalty for deviations between the data vectors and their dictionary learning representations (reconstruction error) and the second term is the penalty which induces sparseness in the code representation. In order to produce an optimal sparse representation, the objective function needs to be minimized with respect to the elements of both D and S:

(4)

under the constraint which prevents scaling transformations of the dictionary atoms. Such transformations could lower the penalty in Eq (3) through the trivial reduction of the overall magnitude of the matrix elements of S instead of inducing sparseness. Note that while L in Eq (4) is a convex function of either D or S separately, it is not a convex function of both variables together. Thus, to minimize L alternate optimization is performed by (i) keeping D fixed and minimizing L with respect to each ; (ii) keeping the latent representation S fixed and minimizing L with respect to D (see Supporting information (SI) Methods for details).

Implementation. We have implemented dictionary learning using the Dictionary Learning module in scikit-learn, with the minimization of L carried out by Block Coordinate Descent [60]. Note that the only parameter needed to be specified explicitly is the sparsity-inducing parameter λ in Eq (3).

The scattering transform

The scattering transform is an image analysis technique which allows to extract a relatively compact and interpretable set of summary statistics starting from the raw image [44–46]. On the one hand, the scattering transform can be viewed as an extension of the power spectrum analysis, with expansions in terms of wavelets [43] rather than plane waves. On the other hand, it is conceptually related to convolutional neural networks (CNNs) which are routinely used in image analysis and classification [47,48,54,56]. However, unlike CNNs, the scattering transform yields more interpretable results and does not require training because wavelet convolutions do not have any fitting parameters. The scattering transform can be viewed as a non-linear transformation which decomposes a signal into a collection of translation-invariant coefficients (summary statistics). It is widely used in signal processing and representation learning tasks in many areas of science [46].

The scattering transform has been recently reviewed and contrasted with other methods such as CNNs [46]. Briefly, at the second-order the transform yields a set of scattering coefficients:

(5)

where I represents the original 2D image and denote scale/orientation wavelet indices, as defined below.

The scattering coefficients are given by

(6)

Here,  ∗  denotes the 2D convolution operation: (u = ( x , y )  is the position in the image), ϕ is a low-pass filter such as the spatial average, and is a wavelet kernel which acts as a band-pass filter and which is labeled with the scale/orientation index . Note that in practice, discrete convolutions are used and thus integrals are replaced with sums.

In Eq (6), the zeroth-order scattering coefficient is simply a convolution of the original image I with the low-pass filter. The first- and second-order scattering coefficients are obtained by iteratively convolving I with a wavelet kernel, taking an absolute magnitude of the result and convolving with the low-pass filter at the end (extension to higher-order scattering coefficients is straightforward). Taken together, the set of scattering coefficients serves as a translation-invariant, compact representation of the image I. For 2D images, the wavelet index λ has two components: a scale index 1 ≤ j ≤ J, where the integer J > 0 specifies the overall averaging scale of the filter, and an angular orientation index 1 ≤ ℓ ≤ L, where L denotes the total number of angles used in the wavelet transform. For simplicity, we limit our analysis to the first-order transform.

Implementation. We used the wavelet scattering transform implemented in the Kymatio Python package (https://www.kymat.io) [61]. We employed 2D Morlet wavelets [43,45] with J = 4 and in the scattering transform (S12 Fig). Thus, there are 4 distinct spatial scales that determine the degree of localization of a given wavelet and 12 distinct angular orientations that correspond to (note that the scattering coefficients are invariant with respect to a rotation through π).

The Morlet wavelets are defined as follows [45]. Let be two vectors in describing  ( x , y )  positions in the image, and let and  | u |  denote their inner product and the vector norm, respectively. The base Morlet wavelet ψ is given by

(7)

where α is a normalization constant, β is adjusted so that  ∫ ψ ( u ) du = 0, and ξ and σ are constants fixed within the Kymatio package.

Let G be a discrete group of finite rotations in . Each group element r can be represented as a 2 × 2 matrix denoted by

where denotes distinct angular orientations (ℓ = 1…L).

Two-dimensional directional Morlet wavelets are obtained by rotating ψ in Eq (7) by r ∈ G and dilating it by for j ∈ ℤ:

(8)

where λ = ( j , ℓ )  is the composite wavelet index.

Optical flow

Lucas-Kanade algorithm. Optical flow is a technique designed to track moving objects given a sequence of images (video frames) that are consecutive in time [49,50]. The idea is to estimate the velocity vectors for each pixel, or a group of pixels, using spatiotemporal gradients of pixel intensity. Let us denote the image intensity field at time t by I ( x , y , t ) , where x and y denote the 2D location of the pixel in the image. Assuming that the pixel’s brightness remains constant between t and t + dt, we have:

(9)

where  ( x + dx , y + dy )  are the pixel coordinates at time t + dt. If dt is small and therefore  ( dx , dy )  are also small, we can use the first-order Taylor expansion to obtain the optical flow equation:

(10)

where and are the x and y components of the optical flow velocity that is our quantity of interest. The optical flow velocity cannot be directly estimated from Eq (10) because it is underconstrained. The Lucas-Kanade algorithm [50] is a standard approach to resolving this problem. It aims to estimate by considering windows of fixed size (image patches) and assuming that the optical flow velocity is constant over that window:

(11)

where i = 1…N enumerates all the pixels in the window and , , are the spatial and temporal partial derivatives of the intensity for pixel i.

In order to estimate for the window, the following objective function is minimized:

(12)

where the sum extends over all the pixels in the window. Repeated for all windows and all consecutive pairs of video frames, the Lucas-Kanade algorithm yields time-dependent optical flow vector field for the entire image (see SI Methods for details).

Corner Detection. Corners are regions in the image with an appreciable variation of intensity in all directions [51]. Computationally, corners can be found by applying a displacement vector to pixel intensities: . The weighted difference between the original and the displaced images is given by:

(13)

where the sum is over all the pixels in the image and w ( x , y )  is a rectangular or a Gaussian window function which assigns weights to the contribution of each pixel. If the displacement is small, Eq (13) can be simplified using the first-order Taylor expansion:

(14)

where

(15)

is a 2 × 2 matrix with the eigenvalues and . Here, ∂I ∕ ∂x and ∂I ∕ ∂y denote the first derivatives of the pixel image intensity I ( x , y )  in the x and y directions, respectively. These derivatives are calculated using discrete approximations similar to the ones used in the optical flow calculations (see SI Methods for details). The magnitudes of the eigenvalues of M determine whether a region is a corner, an edge, or a flat area. Specifically, for corner regions one expects that the quantity E(a) is large for an arbitrary shift a, implying that both eigenvalues of M should be large.

In order to decide whether a window contains a corner or not, we use the Shi-Tomasi scoring function for corner detection [52,53]:

(16)

If R is greater than a pre-specified threshold value, the region is considered to contain a corner. Other choices of the scoring function are available in the literature, such as that employed in the Harris corner detector [51].

Implementation. We use an optical optical flow-based feature tracking algorithm from OpenCV (https://docs.opencv.org/2.4.13.7/). Specifically, we employ the OpenCV function cv2.calcOpticalFlowPyrLK() to track features such as corner points in a video. To decide which features to focus on, we use the OpenCV function cv2.goodFeaturesToTrack(). This function uses the Shi-Tomasi corner-detection method described above [52,53] to find highest-scoring corner points in a video frame. For a grayscale image, we provide the number of corners we want to find, the quality level α (a number between 0 and 1 which denotes the minimum quality threshold below which all corners are rejected), and the minimum Euclidean distance between the detected corners. Here, we use , α = 0 . 2, and pixels (except in Fig 7, where α = 0 . 4 was used to produce fewer corners in the interests of better visualization). To identify corners, we used a pixel window in Eq (13). The function produces a list of all corners above the quality threshold, sorted by the quality score in the descending order. The resulting list is traversed from the top and for each corner all lower-quality corners closer than the Euclidean distance threshold are removed from the list. Finally, the function returns top entries from the list, or the entire list if it has remaining entries.

We use the above procedure to find up to highest-scoring corners in the first video frame. These corners are tracked using Lucas-Kanade optical flow algorithm [50] implemented in cv2.calcOpticalFlowPyrLK(). This function requires the current frame, the corners detected in the current frame and the next frame as inputs, and produces the corner positions in the next frame accompanied by corner status indicators (1 if the corner is found in the next frame, and 0 otherwise). The procedure is then repeated iteratively over the entire video using a progressively smaller set of corners with corner status equal to 1; for corners with corner status equal to 0, the optical flow trajectory terminates. The entire procedure yields a set of optical flow trajectories with variable lengths. To increase the robustness of our predictions, the optical flow calculations are done using image pyramids – a successively downsampled collection of images derived from a single original image (https://docs.opencv.org/3.4/d4/d1f/tutorial_pyramids.html). The number of levels in the image pyramids is controlled via the parameter maxLevel in the function cv2.calcOpticalFlowPyrLK(), which we set equal to 3. The window size in the Lucas-Kanade algorithm is pixels, centered on each corner point.