Ethics statement

This study was conducted in accordance with the Declaration of Helsinki and was approved by the Internal Review Board of Albert Einstein College of Medicine and Montefiore Medical Center (IRB number: 2019–10297). Participants gave online signed consent to participate in the experiment, and upon completion of the experiment they were compensated in accordance with institutional guidelines.

We first present the experimental procedure to measure the same/different judgments of human participants who were instructed to segment the image either into a predefined number of segments, or freely. We then explain how we reconstruct the segmentation maps from the same/different judgments. For this reconstruction, we highlight the important practical constraints (e.g. on the minimal number of trials), and we provide expressions for the loss functions involved in the reconstruction problem. We also propose a regularization method to robustly recover the segmentation maps and explain how to perform reconstruction based on different parametric models.

Experimental procedure

All the experiments presented in this paper were conducted online on naive participants. At the beginning of an experimental session, the screen displays some text instructing the participant to partition the image in K segments and some additional text to precisely define “partition” and “segments” (see S1 Video). We also conducted separate experiments where we did not specify the value of K, and instead instructed the participants to freely partition the image into segments.

After performing a few practice trials, the participants started the main experiment, which is divided in N_b blocks of N_t trials (criteria to choose N_b and N_t are discussed in the following sections, and specific values are provided below). At the beginning of each block, an image to be partitioned is presented on the screen for 3 seconds during which participants are free to visually explore and decide the segmentation of the image. Then the experiment proper starts. On each trial, two points on the image are selected, and participants report whether the two points belong to the same segment or not. Each point corresponds to the center of an element of a predefined grid of size N × N (this grid covers the image but is coarser than the pixel grid and N ⩾ 3). The two elements of the grid are selected pseudo-randomly. First, two small red circles at the selected locations are shown in isolation on a gray background for 300 ms. Immediately afterwards, the two same circles are superimposed on the to-be-segmented image for 300 ms (those durations could be different and reduced when the experiment is conducted in the lab). Thereafter, the image and the points disappear, and a response screen is presented prompting participants to report whether the two cued locations belonged to the same segment or not (Fig 1 bottom-left). The response screen remains visible until the participant reports their choice with a key press, which triggers the beginning of the next trial.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Inference of segmentation maps from pairwise same/different judgments.

Top: Reconstruction of a deterministic segmentation map from simulated data (simulation details in section Materials and methods, subsection Implementation and algorithm). The leftmost panel shows the ground-truth probability map, namely the probability that each pixel belongs to the segment labeled ‘A’ (blue), and similarly for the second (segment ‘B’, green) and third (segment ‘C’, yellow) panel. The fourth panel from the left shows the full segmentation map, namely, for each pixel, the label of the segment with the highest probability. The four panels on the right show the corresponding maps reconstructed with the numerical procedure described in section Materials and methods, subsection Inference of probabilistic segments. Bottom-left: outline of a trial of the segmentation experiment: the participant reports whether the two locations indicated by the red dots belong to the same segment. Bottom-right: for one participant, the reconstructed probability maps (left) and corresponding segmentation map (right), obtained using spatial regularization (see section Materials and methods, subsection Spatial regularization).

https://doi.org/10.1371/journal.pcbi.1011483.g001

Experimental participants

Adult participants were recruited on the online platform Prolific (www.prolific.co). From this website, they were redirected to our experiment page produced with jsPsych 6.3 (www.jspsych.org/6.3/, [53]). Then, after calibrating the size of images to be shown on the screen of the participants by estimating their viewing distance [54] and correcting for their monitor gamma [55], they started to perform the experiment as described above.

In the experiments of involving artificial texture stimuli, we recruited 30 participants in total. They were divided in two groups of 15 participants, and each group performed the experiment on a different image.

In the experiments involving natural image stimuli, we recruited 64 participants. We collected data for 8 different natural images, and data for each image were collected over 8 sessions (as explained above).

Stimuli

For the experiments with natural images, we used cropped natural images from the database BSD500 [29]. For the experiments involving artificial texture stimuli, we used composite textures as follows. Stimuli are images divided in two random areas which are filled with two different (but close) bandpass Gaussian noise textures (oriented textures). Image synthesis is achieved by convolving a white Gaussian noise image with a spatially-dependent filter giving the desired spectral content in each areas. Additional details are in Appendix B in S1 Text.

Detailed choices for each experiment

The minimal number of trials N_t that is necessary for the reconstruction of the segmentation map of an image is related to the grid size N and the expected number of segments in the image K, and is precisely N_t = (K − 1)N². The explanation is given in section Inference of probabilistic segments paragraph Choosing the tested pairs. Here, we report the numbers that were used for each experiment during the development of the proposed method.

In the example experimental session of Fig 1 (bottom), the number of segments K was fixed to 3. We used N_b = 1 and grid size N = 19.

In the psychophysical experiments involving artificial texture stimuli, K was fixed to 2. We used N_b = 5, a grid size N = 11. We collected N_t = KN² = 242 trials (notice that this is more than the strict minimum, N_t = (K − 1)N²). The median duration of each trial was 1.33 s (95% c.i. [0.98, 1.69]), including the presentation time (600 ms) and the median reaction time. The median total duration of the experiment was approximately 40 minutes to measure the segmentation map of one image for one participant. Notice that this is substantially longer than the time during which participants were engaged with the task (median time is 27 minutes), because it includes voluntary breaks that are notoriously difficult to control in an online setting. The analyses presented in section Results were performed by reconstructing the segmentation maps and probability maps of each individual participant (shown in Fig E in S1 Text), and summarized in the main figures as the average maps and inferred features.

In the experiments involving natural image stimuli, K was not constrained. We used N_b = 1 and a grid of size N = 16, and we collected the minimal number of trials needed to reconstruct up to K = 5 segments, that is we collected responses to N_t = (K − 1)N² = 1024 trials. In these experiments, to limit the duration of each sessions, we divided the number of trials by 8 and collected responses to 128 trials per session, thus completing one image along 8 experimental sessions. The participants completed a session in approximately 30 minutes, including voluntary breaks. Therefore, the maps for each image were reconstructed from the aggregate data of 8 participants, not from an individual participant. See section Discussion for further considerations on the duration of the experimental session.

In all the simulations K was fixed to 3 except where noted. Other simulation parameters were varied as detailed in Results.

Inference of probabilistic segments

Given an image, a number of segments K, and the participant’s responses, our goal is to reconstruct both the segmentation map and K probability maps. Probability maps are maps that assign the probability that each pixel belongs to each of the K segments, where K ⩾ 2. The segmentation map assigns, for each pixel, the label of the segment with the highest probability. These maps are defined on a grid of size N × N with N ⩾ 3. Intuitively, this requires finding the maps that are most consistent with the set of N_bN_t binary responses from the participant. In turn, this involves relating the participant’s judgments about whether two pixels belong to the same segment or not, to the probability that each pixel belongs to one of the K segments. In this section we explain how to perform the reconstruction while treating the probability values at each pixel as free parameters. Then in the section Parametric models, we describe two approaches to parametrize the maps more concisely.

Formally, we use the notation for the grid, i.e. the set of (x, y) coordinates of the centers of all the elements of the grid (each element is a square, if the image length and width are equal, as in all our experiments). We use the notation for the set of the coordinates of all the pairs of points ((x₁, y₁) and (x₂, y₂)). At each block n ∈ {1, …, N_b}, we denote by the set of unordered tested pairs of dots presented in each trial (i.e. a pair and its symmetric pair count as a single element). Note that we include in each block multiple distinct pairs, and the notation includes the possibility that the set of pairs tested in each block is different (we will discuss further below how to optimize the choice of the pairs). Because each pair is distinct from all other pairs in the same block, the variability of the responses of one participant can only be assessed by running multiple blocks. The response of a participant at block n and for a pair of pixels is denoted (note that it will be uninformative to test pairs of identical points, therefore in our experiments we exclude such pairs). We assume that participant responses are independent samples of a Bernoulli random variable , with p_i,j denoting the probability that pixels (i, j) are perceived as belonging to the same segment. The negative log-likelihood of the dataset is (1) where, BCE is the Binary Cross-Entropy or simply the negative log-likelihood of a Bernoulli sample r (the participant’s response) knowing the parameter p i.e. BCE(r|p) = −r log (p) − (1 − r) log (1 − p). Next, because our ultimate goal is to estimate segmentation maps, we need to relate this negative log-likelihood to individual pixels rather than pairs of pixels.

In our setting, an image is assumed to have K segments, and a pixel i belongs to segment k with probability p_i[k] ∈ [0, 1]. We also assume that the assignment of a pixel to a segment is independent of the assignments of the other pixels, given the probabilities for all the pixels (i.e. conditionally on ). Under these assumptions, the probability p_i,j that two pixels (i, j) belong to the same segment is given by (2) where p_i = (p_i[1], …, p_i[K]) ∈ Δ_K (the K −dimensional simplex), and p_i ⋅ p_j denotes the dot product. The collection is called probabilistic segmentation maps. Therefore, by plugging Eq (2) into Eq (1) the negative log-likelihood with parametrization given by Eq (2) is (3) The probabilistic maps can be estimated by minimizing the negative log-likelihood (4) under the constraints (5)

Eq (4), as for other clustering methods such as K-means or mixture models, is invariant to label permutation. Therefore, the labels found when solving the problem of Eq (4) will depend on the solver and its initialization. It is well-known that ℓ₀ is minimized when the probability p_i,j is equal to the empirical mean of the responses . As for Generalized Linear Models (GLMs) [56], it is worth knowing under which conditions ℓ is minimized when the probability p_i ⋅ p_j is equal to the empirical mean of the responses . The answer is given by the following proposition.

Proposition 1. Suppose that for all tested pixels the family is a sub-family of critical point of ℓ and of ℓ_s and is linearly independent ( reads “j such that (i, j) belongs to ”). Then, the optimization problem (4) is equivalent to the following least square optimization (6) under constraints (5) and where k_i,j is the proportion of same-segment responses for the pair (i, j) and is the set of tested pixel pairs (different set of pairs can be tested at each block).

Under the conditions of Proposition 1, minimizing ℓ or ℓ_s is equivalent. The proof of Proposition 1 can be found in Appendix A in S1 Text.

Quantifying the accuracy of the inferred probabilistic segmentation maps

In the following, we refer to the loss ℓ defined by Eq (1) as the Binary Cross-Entropy (BCE) and to the loss ℓ_s defined by Eq 6 as the Squared Error (SE). In practice, we will always use the SE loss ℓ_s as it corresponds to the classical non-linear least-square regression.

We illustrate numerically the theoretical result established by Proposition 1 in Fig 2. Experiments were run with N_b = 10 blocks. In practice, we observe that the equivalence of SE and BCE losses holds even if the linear independence condition is not exactly obtained.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Equivalence of loss functions and effects of regularization.

Top left: value of the BCE loss when we optimize for BCE (dashed lines) or for SE (continuous lines). Top center: same but for SE loss. Bottom left: value of the reconstruction MAE. In all panels, the shaded areas represent 95% bootstrap error bars over 1000 simulations. Right: ground truth (GT) probabilistic maps and reconstructed probabilistic maps for each objective function indicated in the legend. The mention “10 Reg.” means that we use regularization with λ = 10.

https://doi.org/10.1371/journal.pcbi.1011483.g002

First, we compare the SE and BCE numerical optimizations. Both methods find solutions with comparable values of the cost function (light gray lines in top-left and top-middle panels), although convergence is marginally slower for the SE loss function compared to the BCE loss function (note that slower here refers simply to the number of iterations of the numerical optimization, which is distinct from the number of trials N_t collected in an experiment).

As an important additional quantitative comparison between the two methods, we also compute the Mean Absolute Error (MAE) with the ground truth maps. The MAE is defined as the L¹ norm of the differences between the K-tuple of the ground truth and reconstructed probability at each pixel, averaged over pixels. Because it is measured on the probabilistic maps, the MAE reflects the accuracy of the estimation of uncertainty. Again we find similar values (light gray lines in bottom-left panel). Lastly, the reconstructed maps are identical (bottom-right panels). Notice that the MAE increases as the optimization of SE or BCE progresses, confirming the visual impression that both the reconstructed segmentation map and the probability maps are noisy and quite different from the ground truth. In a later section, we show that spatial regularization is an effective solution to this problem.

Choosing the tested pairs

The probabilistic maps consist of a set of (K − 1)N² unknowns , thus at least (K − 1)N² pairs have to be tested to infer the unknowns (the choice of the number of segments K and the grid size N is further discussed in Appendix C in S1 Text). Proposition 1 narrows the choice of the pairs to be tested: To preserve the relation between the MLE estimates of Bernoulli random variables and the MLE estimate of the probabilistic maps, it is sufficient that for each tested pixel i the family of probability vectors is linearly independent.

To gain some intuition about this constraint, consider the deterministic case where the probability vectors are one-hot vectors (i.e. one element equals 1, and all others equal 0). In this case, to preserve the linear independence of the family, a pixel i must not be tested against more than K other pixels. In addition, it also indicates that the optimal choice of tested pixels is the following: one pixel must be in the same segment as i, the K − 1 other pixels must belong to every other segments (see Fig 3). As practical guidance for real experiments with natural images, where we do not know the ground-truth segments, a pixel should be tested against a total of K other pixels ensuring that they are sufficiently scattered across the image, given that Gestalt rules suggest nearby pixels are more likely to belong to the same segment than distant pixels. To summarize, we collect enough data to form KN² equations which is more than the minimal amount that is required ((K − 1)N²) but not more to possibly preserve the linear independence of the tested families.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Optimal choice of tested pairs.

Red dots denote the optimal choice of pixels to be paired with the pixel i, in the case of a deterministic segmentation map.

https://doi.org/10.1371/journal.pcbi.1011483.g003

So far, we have treated the probability vectors at each pixel as free parameters, therefore our approach to reconstruct the probability maps requires optimizing a large number ((K − 1)N²) of unknowns. In practice, we find that with limited amounts of data as can be collected in realistic experiments, the reconstructed maps are noisy (illustrated in section Results). In the following two sections, we describe two distinct approaches to tackle this problem.

Spatial regularization

One of the basic Gestalt rules of perceptual segmentation is that spatial proximity encourages grouping [1]. Therefore, although it is still an open question whether human perception uses this rule for grouping and segmentation of complex natural images, we can assume that nearby pixels have a high prior probability of belonging to the same segment when the grid is sufficiently fine. We show in the section Results that adding such a prior (or regularization) to Eq (6) is a powerful method to reduce noise in the recovered probabilistic maps. The regularized problem writes (7) where G * q = ∑_j G_jq_i−j is the discrete convolution (in practice, edge values of p[k] are repeated to ensure images size consistency), G is a local kernel and λ > 0. For example, G can be a Gaussian kernel or, as we chose in this paper, a Laplacian kernel. Intuitively, this regularization simply imposes a cost for p being different than the local average over a neighborhood.

In Fig 2 we have illustrated that Proposition 1 still holds, at least approximately, when using regularization; that is, the results for the two loss functions remain numerically equivalent. The effects of regularization are further examined in section Results.

Parametric models

The model proposed in the previous sections is a parametrization of the probability p_i,j that pixels i and j belong to the same segment. We write (8) where with being the space of parameters. Here , the Cartesian product of two K-dimensional simplexes. Despite being a parametric model for p_i,j, it is the maximally non-parametric model under the assumption of Eq (2). Indeed, we can further consider parametric versions of the underlying class probabilities i.e. (9) where (with being an arbitrary parameter space). Here, it is unknown if the result stated in Proposition 1 holds under such parametric assumptions. However, we illustrate this approach numerically in section Results.

Specifically, we consider feature maps associated to the image (for instance, x_i could be the RGB values of the image pixel i, as in section Numerical simulation; or the activation of a bank of visual filters centered at pixel i, as in section Human participants). We then define the set of parameters (ω, β) with and (where D denotes the feature dimensionality e.g. D = 3 for RGB features), and consider the following multinomial logistic model for the class probabilities (10)

The fitting procedure finds the model parameters that best associate the feature map to the empirical mean of the observed samples (where k_i,j is defined in Proposition 1). See section Discussion for future work on more expressive parametrizations.

General case

The most general approach is to consider a parameter space and to look for a maximum of the likelihood ℓ defined in Eq (1) in the space . Such a problem has been previously explored in the more general case of multinomial distributions but with a single dimensional parameter space i.e. [57]. With our level of generality it is not known under which conditions the results stated in Proposition 1 hold.

Implementation and algorithm

We implemented the models described above in Python using PyTorch. In the non-parametric case defined by Eq (8), we use exponentiated gradient descent to perform the inference [58]. The pseudo code implementing this model is described in Algorithm 1. In the parametric case, defined by Eq (10), we use a quasi-Newton gradient descent (PyTorch implementation of the L-BFGS algorithm).

Algorithm 1: Inference of probabilistic segmentation maps

input : dataset , number of segments K, learning rate λ_r, stopping criterion ε

output : probabilistic maps

begin

 Initialize u ← 0

 Initialize the probabilistic maps p ← p^(u)

 Initialize the loss values and ℓ^(u) ← ℓ^(u+1) + 1

 while |ℓ^(u+1) − ℓ^(u)| > ε do

  ℓ^(u) ← ℓ^(u+1)

  u ← u + 1

 end

end

Simulation details

To validate our methods, we generate synthetic data as follows. Synthetic probabilistic segmentation maps are generated according to the method described in Appendix B in S1 Text. To simulate binary responses , we first randomly selected a set of pairs ensuring that it contains at least once each pixel of the grid. We used the same set of pairs at each block i.e. for any block n ∈ {1, …, N_b}, . Then, for each pair of pixels , we sampled N_b Bernoulli variables with parameter p_i,j.

In numerical experiments, we re-sampled 1000 times the set of pairs in order to show the sampling variability using error bars corresponding to 95% confidence intervals.

Accurate inference of segmentation maps from synthetic and experimental data

Reconstruction of segmentation maps works perfectly in the absence of uncertainty (i.e. each pixel is assigned to a specific segment with probability equal to 1) as illustrated in the top of Fig 1. Conversely, Fig 2 (right panels) shows that when there is uncertainty about the assignment of pixels to segments (which, in the simulated data, translates into variable same/different judgments across blocks), the reconstructed probabilistic maps are less accurate when they are estimated from limited data, as is typical in real experiments. Therefore, we studied in simulations how the accuracy of our approach depends on the level of uncertainty and on the number of blocks N_b. Furthermore, the reconstruction algorithm requires specifying a number of segments K, therefore we also studied how to deal with experimental data in which K might not be known.

Robust reconstruction with limited data.

We generated synthetic data with moderate underlying uncertainty, and studied how the accuracy of the inferred maps depends on the dataset size and on the use of regularization (see the section Spatial regularization). First, we found that regularization substantially improves the accuracy, i.e. it reduces the mean absolute error (MAE) between the ground truth (GT) and inferred maps (Fig 2, bottom left). Importantly, the MAE is measured on the probabilistic maps, therefore it reflects the accuracy of the estimation of uncertainty. This is also appreciable by visual inspection of the reconstructed maps (Fig 2, right panels).

Next, in additional simulations, we studied how the accuracy depends on the number of data points collected. We observed (Fig 4, left) that reconstruction accuracy improved at approximately the same rate with or without regularization, but was 2 to 3 times better on average when using regularization, regardless of dataset size. Upon visual inspection of the maps, regularization afforded near–perfect reconstruction even with only N_b = 1 block (i.e. corresponding to a single measurement per tested pair; Fig 4 right, example 3), although the MAE shows that accuracy increased quantitatively for larger numbers of blocks, as expected. When using regularization, we observed that the increase in accuracy started leveling off after N_b = 10, which can provide a reference for experimental design (for instance, with a grid resolution N = 10 and K = 4 segments, each block lasts approximately 5 minutes, therefore N_b = 10 blocks may be collected in a single session but more blocks might be prohibitive). We also note that this improvement comes at the cost of an increase in variability across simulated experiments (larger error bars with than without regularization, in Fig 4, left), due to the reconstruction bias induced by the regularization.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Accurate inference of segmentation maps from limited data.

Left : the MAE between reconstructed maps and ground truth (GT) as a function of the number of blocks (with and without regularization, light and dark gray respectively). Shaded areas represent 95% bootstrap error bars. Top–Right: ground truth maps. Center–Right: reconstructed maps without regularization from 1 block (left) and 128 blocks (right). Bottom–Right: same as Center–Right but with regularization. The mention “10 Reg.” means that we use regularization with λ = 10.

https://doi.org/10.1371/journal.pcbi.1011483.g004

Robust reconstruction across levels of uncertainty.

Intuitively, the accuracy of the estimates of uncertainty depends on the estimation of across-block variability, and therefore it could be affected by the ground-truth uncertainty level. Thus, we studied the performance of our reconstruction method for systematic changes in ground-truth uncertainty, with a fixed number of blocks N_b = 10. Fig 5 illustrates that the MAE generally increases with uncertainty, because higher ground-truth uncertainty implies noisier observations. When no regularization is used, the MAE rapidly plateaus on average as the uncertainty increases, whereas the MAE variability across experiments decreases. In contrast, when using regularization, the MAE first decreases before increasing strongly for medium levels of uncertainty and then decreasing slightly. The MAE variability is very small for low levels of uncertainty and it is maximal for medium level of uncertainty. Lastly, the reconstruction quality for the two methods is equivalent in the deterministic case, but the reconstructions are 2–5 times better with regularization across all uncertainty levels. These results demonstrate that the regularization enables to robustly capture uncertainty (at least when the uncertainty map has a range of spatial frequency that is similar to the one of the regularization kernel G).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Accurate inference of segmentation maps from variable data.

Left: the MAE between reconstructed maps and ground truth (GT) as a function of the uncertainty (with and without regularization, light and dark gray respectively). Shaded areas represent 95% bootstrap error bars. Top–Right: ground truth maps. Center–Right: reconstructed maps without regularization from low (left) and high (right) uncertainty. Bottom–Right: same as Center-Right but with regularization. The mention “10 Reg.” means that we use regularization with λ = 10.

https://doi.org/10.1371/journal.pcbi.1011483.g005

Robust reconstruction with unconstrained number of segments.

So far we have considered the case where we either know the number of segments K in the ground-truth synthetic data, or, in the real experiments, we ask the participants to partition the image using a specific value of K. However, in some variations of our experiment, we would like to measure segmentation maps without specifying the number of segments. For instance, this is relevant for natural images where there is no obvious ground truth, or for artificial images with high uncertainty, where the perceived number of segments could be an additional source of variability.

Therefore, we first verified in simulations that when uncertainty is moderate and when using regularization, the value of K for the reconstruction can be determined with a straightforward approach: We generated data using K = 5 segments, and reconstructed the maps using K = 3, 4, 5, 6 and 7. In the reconstructions using K = 6 and 7, the superfluous probabilistic maps were automatically set to zero. Therefore the correct K can be inferred from the reconstructed maps, as the maximum value of K that produces no empty maps (see Appendix C in S1 Text).

Next, we conducted experiments with human participants segmenting natural images. Participants were not instructed about the number of segments, and instead were informed that the level of detail in segmenting the images was up to them. The results are presented in Fig 6. We performed the reconstruction assuming K = 5 segments, but we recovered only 3 segments in most images, except for the 6^th and 8^th images for which we recovered only 2. According to the simulations described above, those numbers are likely to reflect the true number of segments used by the participants on average in the aggregated data.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Human Segmentation of Natural Images.

From left to right: the original images, the corresponding segmentation maps, and the five corresponding probabilistic maps. Maps were reconstructed with regularization (λ = 5).

https://doi.org/10.1371/journal.pcbi.1011483.g006

Interestingly, our approach also revealed that regions of high perceptual uncertainty can be captured in the probabilistic maps, even when those regions do not account for a segment in the deterministic segmentation maps. For instance, in the fifth probabilistic map in image 8, the dry grass on the top is sometimes grouped separately from the ground, but most often the two are grouped together in the segment corresponding to the second probabilistic map. Regions of high perceptual uncertainty are also evident in other images, such as in image 7, where the branches are only partially occluding the background sky, so the pixels around those are sometimes grouped together with the bottom branches and sometimes with the background sky. One caveat is that here we reconstructed the maps from the aggregate data across participants (see section Materials and methods, Experimental Participants), therefore these observations may reflect variability across individuals, and we did not assess per-participant uncertainty. In the next section, we examine more closely how our approach can be used to study the uncertainty of perceptual segmentation at both individual and aggregate levels.

Measured uncertainty in human participants correlates with image uncertainty

To demonstrate the use of our method to study human visual segmentation, we conducted a pilot study online in which we manipulated the segmentation-related uncertainty in artificial images (see Appendix B in S1 Text). Note that our goal here is to reconstruct and analyze the probabilistic maps of each individual participant, not those reconstructed from aggregate data as in the previous section. We analyzed data from 15 participants in the low-uncertainty and in the high-uncertainty conditions. To not bias our analysis towards reconstructing smooth probabilistic maps we have not used regularization i.e. λ = 0. We observe that the segmentation maps (Fig 7, second column) are very similar between conditions, except for a few, noisier pixels in the high-uncertainty condition. However, we find that the measured uncertainty of the inferred probabilistic maps (i.e. the total entropy of the maps; Fig 7, numbers in the third column) is significantly larger for images with higher segmentation-related uncertainty (Cohen’s d = 1.003, Welch’s t-test with t = 2.746 and p = 0.0109). Furthermore, the entropy maps reveal a spatial structure that suggests the measured variability does not simply reflect noise: when uncertainty is low, human-uncertainty is localized around the edge between textures, whereas when image uncertainty is high, human uncertainty is more uniformly spread across the entire image (see also individual entropy maps in Appendix E in S1 Text). These results highlight the importance of measuring the variability and uncertainty of human segmentation, and they are consistent with the hypothesis that perceptual processes underlying segmentation include a correct representation of uncertainty [37, 59]. As we show in the next section, these measurements of variability also allow us to compare models of perceptual uncertainty and reveal the image features that participants use to perform segmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Variability in human segmentation reflects image uncertainty.

From left to right: tested images, segmentation maps, probabilistic maps of the left region and entropy maps corresponding to the reconstructed probabilistic maps i.e. p_i[1] log (p_i[1]) + p_i[2] log (p_i[2]) (average entropy ± 3 standard errors is indicated by the text in white). Top: low uncertainty case (texture orientation distributions are weakly overlapping). Bottom: high uncertainty case (texture orientation distributions are strongly overlapping). In all panels, the red line represents the ground truth boundary between the two segments (shown only for visualization purposes, not in the real experiments). Maps are reconstructed without regularization (λ = 0).

https://doi.org/10.1371/journal.pcbi.1011483.g007

Fitting parametric models to infer the image features used for segmentation

We have shown in section Parametric models that the hypothesis of the existence of underlying probabilistic segmentation maps can be strengthened by the additional assumption that they are parametric probabilistic maps, which depend on some features of the image (Eqs (9) and (10)). In other words, with this approach it is possible to use the measured data to estimate the parameters of any hypothesized relation between features of the image and the probability that each pixel belongs to any segment, i.e. the parameters of a segmentation model or algorithm. The motivation for fitting such parametric models is twofold: (i) it will allow quantitative model comparison and hypothesis testing of perceptual segmentation theories and, (ii) it offers the opportunity of finding models that are more data-efficient than the non-parametric model.

Numerical simulation.

We first validated the parametric approach in Fig 8. We generated images whose features are the color values (a 3–dimensional vector i.e. D = 3) of each pixel, and these color features are sampled from a generative model with different parameters for each segment (see Appendix B in S1 Text for details). We use a high resolution (N = 48) in this simulation to provide more samples when training the model and therefore identify the clusters more accurately. We then generated N_b = 10 blocks of simulated data, and applied our inference algorithms.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Validation of the parametric approach.

Reconstruction using a parametric model for the class probabilities (Eq (10)). Reconstruction was achieved minimizing the SE with regularization (λ = 1) Left: probabilistic maps and segmentation maps. Right: features displayed as an image and as 3d points in the RGB cube with the planes separating each pair of segments.

https://doi.org/10.1371/journal.pcbi.1011483.g008

The parametric model correctly recovers the probabilistic maps up to some noise matching the sampling noise of the image color features (in Fig 8, compare the features in the top–right and the reconstructed probabilistic maps in the bottom–left). Importantly, the parametric model also properly characterizes the features associated to each segment by a single 3–dimensional vector (see the bottom–right scatter-plot in Fig 8).

Human participants.

Having validated the parametric approach, we further illustrate its power by applying it to our human data. To this purpose, we use a reparametrized version of the model defined by Eqs 9 and 10 with K = 2. Specifically, for k ∈ {1, 2}, where the inverse is taken component wise. Such a paramatrization allows to interpret as the average feature energy. Because the textures used in the experiment are generated as superpositions of wavelets (Appendix B in S1 Text), we defined for each pixel i the feature as the vector of average wavelet energy, with D = 36 orientation bands. The average is calculated over all wavelet scales and pixels in a small square, partitioning the stimulus in a grid of size N (matching the experimental grid ). As K = 2, Eq (10) can be simplified revealing that only the vector difference is relevant for the fitting. Again, to not bias our analysis towards reconstructing smooth probabilistic maps we have not used regularization i.e. λ = 0. Yet, note that the parametric model at use provides another type of regularization (see section Materials and methods). Results are shown in Fig 9 where we compare the fitted vector of differential variances to the ground truth (i.e. corresponding to the energy of the image in each of the 36 orientation bands). Qualitatively, the participants correctly attributed more weight to the relevant orientations around 90 degrees, although we also observed a small bias away from 90 degrees compared to the ground truth (compare orange lines i.e. the true distribution in the textures, versus blue lines from the fitted parameters). This bias could reflect that this is where the orientation energy distributions of the two textures are least overlapping. In addition, the widths of the bumps are larger for the high uncertainty condition than for the low uncertainty condition, consistent with the ground truth. This indicates that participants integrated information over a broader range of orientations when image segmentation uncertainty was larger, further supporting the hypothesis that uncertainty plays an important role in perceptual segmentation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Uncertainty modulates the perceptual mapping between features and segments.

Left: tested images (same images and data as Fig 7). Right: differential variance (or weight vector, see main text) best relating oriented wavelet features to human responses. Top: low uncertainty case (texture orientation distributions are weakly overlapping). Bottom: high uncertainty case (texture orientation distributions are strongly overlapping). Maps are reconstructed without regularization (λ = 0).

https://doi.org/10.1371/journal.pcbi.1011483.g009