Liquid-Crystal Display (LCD) of achromatic, mean-modulated flicker in clinical assessment and experimental studies of visual systems

Achromatic, mean-modulated flicker—wherein luminance increments and decrements of equal magnitude are applied, over time, to a test field—is commonly used in both clinical assessment of vision and experimental studies of visual systems. However, presenting flicker on computer-controlled displays is problematic; displays typically introduce luminance artifacts at high flicker frequency or contrast, potentially interfering with the validity of findings. Here, we present a battery of tests used to weigh the relative merits of two displays for presenting achromatic, mean-modulated flicker. These tests revealed marked differences between a new high-performance liquid-crystal display (LCD; EIZO ColorEdge CG247X) and a new consumer-grade LCD (Dell U2415b), despite displays’ vendor-supplied specifications being almost identical. We measured displayed luminance using a spot meter and a linearized photodiode. We derived several measures, including spatial uniformity, the effect of viewing angle, response times, Fourier amplitude spectra, and cycle-averaged luminance. We presented paired luminance pulses to quantify the displays’ nonlinear dynamics. The CG247X showed relatively good spatial uniformity (e.g., at moderate luminance, standard deviation 2.8% versus U2415b’s 5.3%). Fourier transformation of nominally static test patches revealed spectra free of artifacts, with the exception of a frame response. The CG247X’s rise and fall times depended on both the luminance from which, and to which, it responded, as is to be generally expected from LCDs. Despite this nonlinear behaviour, we were able to define a contrast and frequency range wherein the CG247X appeared largely artifact-free; the relationship between nominal luminance and displayed luminance was accurately modelled using a causal, linear time-invariant system. This range included contrasts up to 80%, and flicker frequencies up to 30 Hz. This battery of tests should prove useful to others conducting clinical assessment of vision and experimental studies of visual systems.


DISPLAY OF MEAN-MODULATED FLICKER
light transmitted by the backlight unit to the observer; voltage applied between the electrode pair in turn applies torque to molecules, which in turn alters the molecules' orientation. A key observation is that these changes in orientation are sluggish relative to changes in voltage.
Furthermore, changes in orientation are asymmetric: changes involving an increase in torque on molecules have different temporal dynamics to those involving a torque decrease [25]. These asymmetric dynamics are further complicated by digital response time compensation mechanisms, a feature of most modern LCDs designed to enhance video by accelerating step transitions between certain luminance levels [25,26,27,24]. These LCD mechanisms confer many desirable behaviours to displays, including the independence of neighbouring pixels [28,29]. However, it follows from these mechanisms that, in general, LCDs cannot be used to present achromatic, mean-modulated flicker because, when response time depends on both source and destination luminance, time-averaged luminance depends on flicker frequency and contrast.
Here we present a battery of tests used to weigh the relative merits of two displays. We are specifically interested in the presentation of achromatic, mean-modulated flicker, as exemplified by the studies cited above [2,3,4,5,6,7,8,9]. We used these tests to reveal differences between a new high-performance LCD (EIZO ColorEdge CG247X) and a consumer-grade LCD (Dell U2415b); these differences were marked, despite the displays' vendor-supplied specifications being almost identical. The CG247X's rise and fall times depended on both the luminance from which it responded as well as the luminance to which it responded, and, overall, were slow by comparison to the consumer-grade LCD. However, despite this nonlinear and sluggish behaviour, our tests revealed a contrast and frequency range wherein the CG247X's display of p. 6 DISPLAY OF MEAN-MODULATED FLICKER mean-modulated flicker was without artifacts, that is, the relationship between nominal and displayed luminance was linear. p. 7

Methods
Displays and Visual stimuli. We measured luminance on two LCDs: EIZO ColorEdge CG247X, and Dell U2415b. The former is a high-performance display; the latter is a consumer-grade display. Both displays were purchased in 2018, have received limited use, and have almost identical vendor-issued specifications: panel type (in-plane switching), size (61.1 cm), native resolution (1920-by-1200 pixels at 60 Hz), pixel pitch (0.270-by-0.270 mm), viewing angle (178 deg horizontal and vertical), response time (gray-to-gray between 8 and 10 ms), and light-emitting diode (LED) backlighting. We used the displays' factory default settings with one exception: the "brightness" of the U2415b was reduced to 35% to make the maximum displayed luminance of the two devices approximately equal (120 cd/m 2 ). For both displays, we used spatial resolution of 1920-by-1200 pixels and temporal resolution of 60 Hz. Luminance was rendered on displays using Psychtoolbox [30,31,32] (version 3.0.16) and MATLAB (version 9.5.0.944444, R2018b); Mathworks, Natick, Massachusetts , USA). We used a standard desktop computer running Linux (Ubuntu version 18.04) installed with an NVIDIA graphics card (QUADRO P620; Santa Clara, California , United States ). We always allowed displays at least one hour to warm up prior to testing. Our preliminary measurements revealed no warm-up transient on either display; we found negligible difference between displayed luminance of a static test patch at one minute versus thirty minutes after power-on. 100%. We then fit a model of displayed luminance: where is the nominal luminance, and and are free parameters. We verified the fitted x a b model, presenting luminances , , … and ensuring the displayed luminances formed a zero-intercept straight line (e.g., see Figure 1). Photodiode device and Electrical recordings. We used a device comprising a PN photodiode (BPW21R; Vishay Intertechnology, Malvern, Pennsylvania , United States ) designed for applications requiring high-precision and linearity, and a custom electronic circuit ( Figure S1A).
The full-width at half-sensitivity of this photodiode is 105 deg, which, at a distance of 1 cm from the display, encompasses 96.3 pixels. We verified the linearity of the device by measuring the luminance of 11 static test patches (luminances = 0, 10, 20 … 100%; see Display linearization ) with both the photodiode device and the LS-110 spot meter ( Figure S1B). We also tested the photodiode device using an LED at a fixed distance of 1 cm. We used a function generator (InfiniiVision DSO-X 2002A, firmware version: 02.50.2019022736; Keysight Technologies, Colorado Springs, CO, USA) to apply square voltage waveforms (50% duty cycle) to the LED at frequencies from 1 to 32 Hz (at one-octave increments). At these frequencies, the photodiode p. 9 DISPLAY OF MEAN-MODULATED FLICKER device's response to the LED was square with no measurable change in response amplitude with frequency. We also applied a wider range of frequencies from 1 Hz to 1 MHz (at half-decade increments). The photodiode device's response was attenuated at frequencies greater than 100 kHz ( Figure S1C). Photodiode device output was visualized online using our digital oscilloscope. We also used the oscilloscope to acquire electrical recordings, and to derive response times, cycle-averaged luminance, and r.m.s. luminance. This oscilloscope enables high sampling rates (up to 2 GHz); we used different sampling rates in different experiments ranging from 10 kHz to 100 kHz.
Spatial and temporal uniformity. To quantify the spatial and temporal uniformity of the displays, we presented static test patches (384-by-384 pixels) at nine positions ( Figure 2, inset). At each position, we presented low-, medium-, and high-luminance patches (luminance = 0, 50, and 100%, respectively), measuring luminance with the LS-110 spot meter which we held by hand fronto-parallel to the patch. Furthermore, at position 5, we presented static test patches at 11 different luminances (0, 10, 20 … 100%), measuring 10 one-second recordings of luminance with the photodiode device. The photodiode was positioned 1 cm from the display, fronto-parallel, using a laser-cut acrylic arm mounted on the display housing. We computed the Fourier transforms of these photodiode device recordings using MATLAB (version 9.3.0.713579, R2017b).
Effect of viewing angle. To estimate the effect of viewing angle on luminance, we presented large (1920-by-1200 pixels), static test patches, measuring luminance near the center of the display with the spot meter. In these experiments, the spot meter was mounted on a tripod at 1 m. We placed the display on a turntable; to vary the angle of incidence of the spot meter's p. 10 DISPLAY OF MEAN-MODULATED FLICKER   optical axis with the display, we rotated the turntable. Thus, we measured effective luminance   for azimuth = -60, -40, -20, -10, 0, 10, 20, 40, 60 deg (negative angles are clockwise rotations of   the turntable when seen from above). Additionally, we swiveled the display from "landscape" to "portrait" configuration, and repeated our measurements, thus varying elevation from -60 to 60 deg. Rotating the turntable (as opposed to rotating the spot meter) enabled precise control over a wide range of azimuth and elevation. However, this approach assumes that the display is spatially uniform, i.e., that spatial variability is negligibly small with respect to the effect of viewing angle. We modeled luminance as a function of azimuth, , using a circular von Mises (θ) l function, i.e., where is a modified Bessel function of the first kind, order zero. We optimized the free () I 0 parameters, and , by minimizing the sum of the squared error between the model and data.
α κ We modeled luminance as a function of elevation in the same fashion.
Response times. We measured response times, that is, the time taken to transition between luminances L1 and L2. L1 and L2 took values 0, 25, 50, 75, or 100%. Stimuli were square patches (384-by-384 pixels) at position 5 ( Figure 2, inset). Stimuli were sequenced as follows: black (luminance = 0%) preceded stimulus L1; L1 was presented for 10 frames (166.67 ms); L2 was presented for 10 frames immediately following L1; black followed L2. We used the standard definition of rise (fall) time between luminance L1 and L2 as the duration of the transition from p. 11 DISPLAY OF MEAN-MODULATED FLICKER 10 to 90% (90 to 10%). For example, the rise time between luminance 10% and 50% was defined as the time taken to transition from 14% to 46%.
Achromatic, mean-modulated flicker. We presented square-wave, achromatic mean-modulated flicker at position 5 ( Figure 2, inset). The time-averaged mean luminance of flicker was nominally 50%. We computed contrast (the temporal analogue of Michelson contrast) of the nominal waveform using the equation: where L max is maximum luminance and L min is minimum luminance. We used five flicker contrasts ranging from 20 to 100%, and flicker frequencies ranging from 0.94 Hz (flicker period = 64 frames) to 30 Hz (flicker period = 2 frames).
Paired biphasic luminance pulses. We presented single, biphasic luminance pulses (positive/negative; see Figure 8A) at position 5 ( Figure 2, inset). The mean luminance of these pulses was 50%, and their peak-to-peak amplitude was 50%. The duration of each phase was 1 frame period (16.67 ms); the duration of each pulse was 2 frames. The first of these pulses was delayed by a fixed period relative to a stable trigger, T t . The second through fifth of these pulses were identical to the first pulse, but delayed by 1 through 4 frames relative to the first pulse, respectively. We also presented paired, biphasic pulses. The first pulse of all pairs was delayed by T t . We parametrically varied the offset between pulses comprising the pair, T = 0, 1, 2, 3, and

Results
Spatial uniformity of displayed luminance can vary widely between different makes and models of LCD, the major determinant of uniformity being the backlight scheme [33] (some older LCDs allowed VGA input and relied on built-in analog-to-digital conversion, also a potential source of noise). Two commonplace schemes are, first, direct backlighting, wherein a spatial array of light-emitting diodes (LEDs) and a diffuser screen sit behind the liquid crystal panel, and, second, edge illumination, wherein light emitted by a linear array of diodes at one of the display's edges is spatially distributed via lightguide. We quantified the spatial uniformity of the CG247X by presenting low-, medium-, and high-luminance static test patches at nine display positions ( Figure 2, inset) and using the LS-110 spot meter to measure the luminance of each patch. At each luminance tested, we calculated the grand average over all display positions, and divisively normalized measurements by that average. As illustrated in Figure 2, at mediumand high-luminance, the CG247X showed greater spatial uniformity than our consumer-grade LCD (Dell U2415b): for the CG247X, spatial variation was 5.1% at medium and 3.5% at high luminance, whereas for the U2415b, variation was 8.1% at medium and 8.5% at high luminance. The uniformity of the two displays was comparable at low luminance (CG247X, 27% versus U2415b, 17%). Prior to normalization, there were, as expected, marked differences between low-, medium-, and high-luminance measurements. For example, at display position 5 ( Figure 2, inset) on the CG247X, low-luminance measurements ranged from 0.07 to 0.10 cd/m 2 , medium-luminance measurements ranged from 57.70 to 57.93 cd/m 2 , and high-luminance measurements ranged from 113.9 to 114.2 cd/m 2 (Table 1). We also quantified spatial surround effects; using a tripod at 1 m, we measured displayed luminance at position 5 comparing large (1920-by-1200 pixels) and small (384-by-384 pixels) 100%-luminance patches. For CG247X, p. 14 DISPLAY OF MEAN-MODULATED FLICKER the mean of 10 large-patch measurements was 0.56 cd/m 2 greater than that of 10 small-patch measurements (two-sample t-test, p < 0.01), i.e., an increase of 0.50%. For the U2415b, the increase was 0.71 cd/m 2 , i.e., 0.67% (two-sample t-test, p < 0.01).  Table 1 . Spatial uniformity of static test patches at low-, medium-, and high-luminance.
Unit of measurement is cd/m 2 .
[ Table 1 here] In-plane switching (IPS) LCDs, like our CG247X and U2415b, enable larger viewing angles than older LCD technology (e.g., twisted-nematic displays) [23]. To do so, IPS displays interdigitate electrodes (see Introduction ) on a single substrate interposed between the backlight and the LC material [23].  [35]). We quantified the temporal uniformity of the CG247X by presenting (nominally) static test patches at display position 5 (Figure 2, inset) and using the linearized photodiode device to measure displayed luminance over time. At each of 11 luminances (0, 10, 20 … 100%) we made 10 one-second recordings, averaging the Fourier amplitude spectra of those 10 recordings. Figure   4 shows the average spectrum at each luminance. The spectra of the CG247X revealed a frame response comprising a 60 Hz component as well as harmonic components at integer multiples of 60 Hz. The response at 60 Hz varied non-monotonically in amplitude with the luminance of the static test patch, peaking at a luminance of 50%. However, the CG247X appeared free of backlight modulations. This absence of modulations free us of the consequences of said modulations (often desynchronized with the frame refresh signal) on increment/decrement transitions between luminances (see Fig. 5 in [24]). The spectra of our consumer-grade LCD also revealed a frame response, as well as 1.2 kHz flicker, likely associated with the back light.
This latter temporal nonuniformity increased linearly with the luminance of the static test patch.

DISPLAY OF MEAN-MODULATED FLICKER
For each display, we verified that the frame response was optical and not related to any radiated electromagnetic noise: We used the oscilloscope to visualize the Fourier amplitude spectrum online. We then interposed opaque cardboard between the photodiode and display which caused the disappearance of the frame response. For the U2415b, we similarly verified that the 1.2 kHz response was optical. RTC mechanisms speed luminance transitions by transiently altering the voltage applied to the liquid crystal associated with individual pixels (e.g., Figure 1 in [27]; Figure 5 in [24]). We measured the CG247X's response times by presenting luminance steps --both increments and decrements --to the linearized photodiode device.
Step source and destination took values 0, 25, 50, 75, or 100%. As illustrated in Figure 5, response times varied as a function of both luminance step source and destination. For example, stepping from 0% luminance to 25% luminance took 24.5 ms, stepping from 75% to 100% took 12.9 ms, and stepping from 25% to 0% took 8.1 ms. All of these steps are the same height, but response times differ markedly.
Overall, the response times of our consumer-grade LCD were less than the CG247X response times. However, as we will illustrate below, faster is not better; although RTC mechanisms reduced the response times of our consumer-grade LCD, they contaminated displayed luminance with overshoot and undershoot artifacts which are problematic for many applications in clinical and experimental vision research, including the presentation of mean-modulated flicker. RTC mechanisms lower "black-white-black" and "grey-to-grey" response times, which are used to promote displays to the gaming community and other consumer markets. These rise times (upward triangles) decreased with increasing destination luminance.
The gray box labelled "25%" shows mean response times of transitions from source luminance = 25% to destination luminances = 0, 50, 75, and 100%. Graphical conventions are as in A. Overall, U2415b response times were less than CG247X response times.
At the outset of this study, we made preliminary measurements similar to those illustrated in Figure 5. We noticed that rise and fall times straddling 50% luminance were approximately equal (e.g., rise time from 25% to 75% = 16.3 ms; fall time from 75% to 25% = 17.1 ms) which led us to wonder whether the CG247X could be used to display achromatic, mean-modulated flicker without the introduction of unworkable artifacts. To better determine the CG247X's potential suitability for presenting mean-modulated flicker, and its susceptibility, or otherwise, to overshoot and undershoot artifacts typical of LCDs implementing RTC mechanisms, we presented mean-modulated flicker on both the CG247X and our consumer-grade display, using the linearized photodiode device to measure luminance over time. We used a flicker period of 20 frames (333.3 ms), and contrast ranging from 20 to 100%. As illustrated in Figure 6, the consumer-grade display's luminance traces revealed overshoot and undershoot artifacts symptomatic of RTC. The CG247X's luminance traces, however, appeared free of RTC artifacts.
We used these traces to estimate response times specific to mean-modulated flicker, illustrated in Figure 7. Overall, CG247X rise and fall times were greater than those of our consumer-grade LCD. However, with the exception of 100% contrast, CG247X rise and fall times were approximately equal, indicating its potential suitability for presenting mean-modulated flicker.
p. 20  To further determine whether the CG247X could be used to display achromatic, mean-modulated flicker without the introduction of unworkable artifacts, we presented flicker at frequencies ranging from 0.94 to 30 Hz and contrasts ranging from 20 to 100%. We used recorded traces (similar to those in Figure 6) to derive cycle-averaged luminance. In Figure 8,  Taken together, Figure 8, and the traces used to derive the measures plotted there, indicated a simple relationship between nominal and displayed luminance on the CG247X, namely, that the latter was, simply, a low-pass-filtered version of the former. To test this hypothesis, we modeled the function transferring nominal luminance to displayed luminance as a causal, exponential decay (Methods). We optimized the single free parameter in this model, the time constant of the exponential decay ( ), by minimizing the sum of the squared error between the model-derived τ cycle-averaged mean luminance and cycle-averaged r.m.s. luminance, and those derived from the photodiode traces. For the CG247X, the fit is illustrated in Figure 8 (blue). There, the fitted parameter, , was 6.6 ms. To assess the fit to cycle-averaged luminance, we computed the τ root-mean-square error (RMSE) separately at each flicker contrast. For the CG247X, the RMSE was negligibly small for contrasts from 20 to 80% (ranging from 6.0e-4 to 6.3e-3 normalized units For the CG247X, R 2 was high, ranging from 0.9965 to 0.9999. As expected, the same calculation for the U2415b was consistent with a poor fit; at its worst, R 2 = 0.03.

DISPLAY OF MEAN-MODULATED FLICKER
To quantify the nonlinearities associated with high-contrast, mean-modulated flicker, and to quantify temporal dependence between frames, we used a paired-pulse paradigm [36,37]. We presented paired biphasic luminance pulses at position 5 ( Figure 2, inset), systematically varying the inter-pulse interval, T (Methods). We used the measured responses to individual pulses to predict paired-pulse responses, and to model the display's nonlinearities we subtracted each paired-pulse response from its prediction. Figure 9 shows the nonlinear behaviour of the CG247X and, for comparison, that of our consumer-grade LCD. In our CG247X, a nonlinear mechanism appeared to speed the transition between white and black (100% and 0% luminance, respectively; leftmost upper panel in Figure 9B). When paired pulses were separated by 16.67 ms or more (the three rightmost upper panels in Figure 9B where predicted and displayed luminance are approximately equal), the CG247X behaved linearly, that is, we saw no evidence of temporal dependence between frames. In our consumer-grade LCD, a nonlinear mechanism appeared to attenuate the transition to white (100% luminance; leftmost lower panel in Figure 9B). This attenuation reconciles with Figure 6 (lower), which shows marked overshoot at moderate contrast (e.g., 60% contrast, middlemost panel of Figure 6), but a near absence of overshoot at high-contrast (rightmost panel of Figure 6). Compared to the CG247X, the U2415b's nonlinearities were large in magnitude and long-lasting. Paired pulses separated by as much as 33.33 ms (the third lower panel in Figure 9B, where predicted and p. 24 DISPLAY OF MEAN-MODULATED FLICKER displayed luminance are unequal) evoked nonlinear behaviour in the U2415b, that is, we saw clear evidence of temporal dependence between frames.

Discussion
We have presented a battery of tests used to weigh the relative merits of two displays. We are specifically interested in the presentation of achromatic, mean-modulated flicker, as exemplified by the studies cited in the Introduction [2,3,4,5,6,7,8,9]. We recorded luminance waveforms from two LCDs, and derived several measures, including spatial uniformity, response times, Fourier amplitude spectra, cycle-averaged luminance, and root-mean-squared luminance. We presented paired luminance pulses to quantify the displays' nonlinear dynamics at high-contrast and high frequency. We find the EIZO ColorEdge CG247X is suitable for displaying mean-modulated flicker within a range of contrasts (<= 80%) and frequencies (<= 30 Hz). Unlike CRT displays, relatively little has been written in the clinical and experimental vision literature about LCDs. Here, we review detailed studies of LCD behaviour, discussing their findings with respect to ours.

Relationship to Previous Studies
Wang & Nikolić [28] tested a Samsung 2233RZ LCD, comparing its performance to a consumer-grade LCD, as well as a CRT display. They estimated spatial uniformity using measurement procedures similar to ours, reporting spatial variability of approximately 15% at high-luminance. Spatial variability of their CRT display (again, at high-luminance) was greater than 20%. Our CG247X, which varied by only 3.5% at high-luminance (Fig. 2), was more high-luminance. These results, taken together with LCDs' ability to render fine horizontal, oblique, and vertical gratings with little contrast attenuation [28,29], confirm that overall the spatial uniformity of LCDs is usually superior to that of CRT displays.
Using high-luminance pulses, Wang & Nikolić [28] measured the reliability of their displays, reporting errors of less than 0.04%. Although their results are difficult to interpret because "error" was not clearly defined, they are likely to be comparable to ours: For flicker contrast at 100%, and flicker frequency at 0.94 Hz, we used 10 measurements of cycle-averaged mean-luminance, and 10 measurements of cycle-averaged r.m.s. luminance (Fig. 8 contrast) of their displays by fitting exponential functions to rise and fall step responses separately. For their 2233RZ, they found fall time to be greater than rise time by a factor of 1.15.
The most straightforward comparison between their results and ours is as follows: at 100% contrast, our CG247X's rise time was greater than its fall time by a factor of 1.72 (Fig. 7); therefore, the CG247X is not suitable for presenting mean-modulated flicker at 100% contrast.
Matsumoto and colleagues [38] noted that approximately equal rise and fall times are key LCD behaviour for presenting flickering stimuli. At 80% contrast, we found a much smaller discrepancy between fall time and rise time: the former was greater than the latter by a factor of only (an LCD specifically marketed to vision researchers and clinicians) which appeared immediately after white-to-black transitions. None of our recordings on either of our displays revealed these trailing blips. However, our paired luminance pulses revealed nonlinearities in both our displays ( Figure 9). These nonlinear mechanisms speeded transitions between extremes of luminance (CG247X), or attenuated overshoot and imparted temporal dependence between frames (U2415b).
Our spectral analysis (Fig. 4) of displayed luminance of (nominally) static test patches on both our CG247X and our consumer-grade display revealed a frame response [24,23,35], that is, luminance modulation associated with display refresh rate (here, 60 frames per second). For both displays, the peak-to-peak amplitude of the fundamental component (60 Hz) of this response was small --on the order of 0.1 cd/m 2 . However, this amplitude reflects our photodiode's spatial aperture, which was large relative to the size of single pixels; because frame inversion schemes typically have fine spatial structure on the scale of individual pixels [35], the frame response of single pixels on either of our displays was likely much larger than our measurement. Our spectral analysis confirms and extends measurements by Elze Fig. 8C; RMSE = 0.03). In the model of Figure 8, and the above-described example, the derived measure was cycle-averaged r.m.s. luminance. However, a similar approach could be used to fix some other derived measure of displayed luminance (e.g., cycle-averaged mean absolute luminance). For the U2415b, this straightforward approach p. 31

DISPLAY OF MEAN-MODULATED FLICKER
to adjusting for artifacts is impeded by the fact that, for that display, no simple model relates nominal and displayed luminance.

Concluding Remarks
The above-described studies of LCD behaviour, taken together with our test results, capture a trend: for the intents and purposes of vision research, LCD performance continues to improve.
Our tests revealed marked differences between two displays, despite these displays' vendor-supplied specifications being almost identical. In other words, these specifications were entirely useless in weighing the relative merits of our displays for presenting achromatic, mean-modulated flicker. One of the displays tested --the EIZO ColorEdge CG247X --appears largely free of artifacts when presenting flicker within a restricted range of contrasts and frequencies. We hope the present battery of tests proves useful to others developing rigorous approaches to clinical assessment of vision and experimental studies of visual systems; we intend to use it when introducing new displays to our experimental set-ups.