1.1 Motivation

The motivation for our work is two-fold.

Firstly, in spite of a large body of studies on spectrum sensing, several representatives of the research community agree there is no standard way of designing and comparing signal detection methods. A variety of different evaluation approaches are being used in the literature. The results from different studies are thus hard, if not impossible to objectively compare. This situation is calling for a common methodology for designing and evaluating signal detection methods, similarly to what we can find in many other research areas related to wireless networks.

Secondly, we noticed that many analyses based only on theory and simulations [10–13] fail to account for the effects of practical receiver designs on signal detection performance. By introducing a methodology that also covers practical validation, we hope to encourage experimentation in this field that will eventually result in a more solid understanding of signal detection methods.

1.2 Contributions

The main contribution and a novelty of this paper is a unified methodology, which is based on commonly used yet thus far not harmonised and generalised steps. This methodology can be used to resolve the existing tussles around various signal detection methods by means of uniform objective quantitative evaluation and comparison. We focus solely on blind signal detection, which we define as a form of signal reception where we are only interested in the fact that a transmission exists and not in the information it carries. The proposed methodology is comprised of seven steps and can be applied to evaluate methods in simulation or practical experiments. Some steps can be seen as examples of good practice leading to reproducibility of results [14]. We show that these steps are even more important when it comes to fair comparison of methods that differ significantly in their implementation.

Another major contribution of this work is a comprehensive experimental evaluation of analog and digital energy detection, covariance-based and eigenvalue-based detection and cyclostationary detection. We measure minimal detectable signal power for signals with two typical analog and digital modulations, sensitivity of detection methods to noise power changes and their computational complexity. We use an experimental setup that covers typical capabilities from embedded to high-end software defined radio devices. We also experimentally evaluate the effect of filter compensation on covariance-based detection. In order to support reproducibility and cross-comparison of existing and new spectrum sensing methods, we openly publish the source code of our implementations of signal detectors and signal models.

The rest of the paper is organized as follows. Section 2 discusses how existing literature on methodology and spectrum sensing relates to our work. Section 3 describes the proposed methodology with the subsequent sections describing individual steps in our specific evaluation. Section 4 lists the selected detection methods using a common form. Section 5 describes the signal and noise model waveforms. Our implementation of detection methods and waveforms is described in Section 6. The experimental setup is outlined in Section 7. The evaluation procedures and results are presented in Sections 8 and 9 respectively. Finally, Section 10 concludes the paper with a summary of our main findings.

2.1 On methodology

In the wireless network literature authors have proposed various methodologies for developing new models and technologies. The authors of [17] propose a design methodology that enables developing more efficient and dependable receivers using cross-layer design. In [18], the author proposes a design methodology for highly integrated low-power receivers for wireless communications, while in [19] the authors propose a design methodology for using TCP over wireless links that have a link level error control mechanism.

The authors of [20] propose a methodology for optimizing wireless sensor networks dynamically using Markov Decision Processes. In [21] a multi-purpose testing methodology for cellular network components is introduced and demonstrated by measuring the data rate under different experimental conditions. In [22] authors propose a methodology for spectrum occupancy surveys, mostly focusing on energy detection, with the aim of producing directly comparable quantitative results.

All these works have a different scope than the one proposed in this paper. Our work is the first of its kind by proposing a methodology for uniform, scientifically objective evaluation of the large number of existing signal detection methods that promotes reproducible repetitions and re-evaluations, thus also representing a novelty in the field.

The closest related work to ours is [10], where several blind detection methods are reduced to a common analytic form to aid their comparison. The work in [10] represents only the very first steps required to objectively study the performance of existing methods. While their work identifies detection methods, presents a common formalization and signal models, it does not address the implementation, simulation and/or experimental and evaluation procedures. Compared to [10], our work not only proposes a generic methodology covering all necessary steps but it also demonstrates the validity of the methodology on the largest number of signal detection methods evaluated against each other thus far.

2.2 On spectrum sensing

Spectrum sensing has several definitions, perhaps the most representative being the “ability of a device to listen for other nearby spectrum users to determine whether it is possible to transmit” [23] and, more broadly, as “a task of obtaining awareness about the spectrum usage and operational electromagnetic environment” [24].

Various aspects of spectrum sensing have been widely covered in recent literature, both in the form of surveys, papers focusing on individual methods and papers focusing on TVWS use. A vast majority of existing work, however, is based on theoretical models and simulations.

In [15], a comprehensive overview of the broader field is given, covering a variety of topics across multiple disciplines, including signal detection, cooperative sensing, machine learning, as well as security issues. Only a qualitative comparison of signal detection methods is given. Authors identify spectrum sensing hardware requirements as a future challenge and also note that published experiments to date mostly use energy detection. In our work we address this challenge by evaluating methods beyond energy detection. We also investigate requirements in respect to sampling rate and number of samples.

In [16], signal detection methods are classified based on the required knowledge of the source signal and noise power. The paper gives a thorough overview of the theoretical background, however it presents directly comparable simulation results only for some of the presented methods. The authors conclude that more tests of practical realizations of signal detectors are required. Our work can be seen as a contribution towards this goal. The experiments we performed contradict their finding that cyclostationary detectors require higher sampling rates and number of samples than other detectors.

In [10], blind signal detection methods are reduced to a common form. Probability of detection and computational complexity of the methods is compared in numerical simulation using a common set of parameters. Covariance-based detection is found to be very computationally complex, which is in contrast with our observations. Unlike in our work, cyclostationary detection is considered non-blind.

In [13], MATLAB simulations of energy detector with various thresholding methods, cyclostationary and matched filter detection are tested for probability of detection versus signal-to-noise ratio under AWGN and Rayleigh fading. Unlike in our work, covariance- and eigenvalue-based detection is not evaluated.

A review of spectrum sensing for applications in vehicular networks is available in [8]. Authors identify TVWS as a possible way of providing extra spectrum when dedicated channels get congested. They consider energy detection, matched filtering and cyclostationary detection in a qualitative comparison.

In general, qualitative comparisons based on unrelated evaluations of individual methods are problematic and can lead to erroneous conclusions. Results of such evaluations are rarely directly comparable due to plethora of different signals, parameters and environments used in the evaluations. For instance in [25], energy detection is experimentally evaluated using a wireless sensor network, including measurements of minimal detectable signal power. A sine wave and a 4 MHz QPSK signal were used. In [26], a 4 MHz QPSK signal with 64 MHz sampling rate is used to experimentally evaluate a cyclostationary detector. In [27], a different cyclostationary detector is evaluated with a simulated wireless microphone signal using 2048 samples and 12 MHz sampling rate. In [11], a simulated wireless microphone signal with 6 MHz sampling rate and an experimentally recorded digital TV waveform is used to compare covariance-based detectors and energy detection. In [28], a BPSK signal is used in a simulation to evaluate eigenvalue-based detectors.

Individual evaluations would also benefit from a methodology, such as the one proposed in this paper. In [10], authors discuss the often missing quantitative comparisons in publications of novel methods. Clearly defined steps can aid researchers in performing such comparisons and documenting their experiments to help with the reproducibility of published results. For instance, evaluation in [26] does not document the parameter N′ of the FAM method used for cyclostationary detection and the number of trials used to obtain probability estimates. In [29], a cyclostationary detector based on compressive sensing is experimentally evaluated, however signal models used in the evaluation are not presented and confidence estimates of the results are not given.

Step 1: Identify detection methods and form of evaluation

As a first step, one has to identify the target signal detection methods to be compared. For each method this includes finding the description, the formalization including any parameters, possibly a reference implementation and the performance evaluation carried out by the original author. The identified methods can then be evaluated against each other in a simulation environment, in an experimental lab test or both.

In this paper, we selected nine representative methods from four categories: energy detection, three covariance-based detection methods, four eigenvalue-based detection methods and one cyclostationary detector. We evaluated these methods in a lab test. We consider a scenario in which a TVWS device attempts to detect the presence of analog and digital legacy devices before transmitting.

Step 2: Common formalization

In this step, the selected detection methods from Step 1 have to be brought to the same form. The goal is to document how a method establishes a relationship between the channel observation and the binary channel occupancy decision. The notations in the formulas have to be the same across all methods. At this point also parameters common to all methods are identified.

We represented all nine methods selected in the previous step as a scalar test statistic, which is a function of a vector of signal samples. We use a static threshold for the binary decision. We identified commonly used parameters such as time of sampling, target probability of false alarm, etc. This definition is detailed in Section 4. An example of alternative formalizations would be to have multiple features instead of a scalar test statistic or to use a different thresholding method.

Step 3: Signal and noise model

In the third step, signal models that will be used to represent vacant and occupied channels have to be selected. The detection is performed using various levels of signal and noise power, which also have to be selected. This selection is affected by the assumptions about signal and noise in the methods. Some methods focus on detection of specific types of waveforms while others have more generic assumptions about signals being detected. The choice of models is also affected by the form of evaluation identified. In experimental evaluation, practical limitations (e.g. receiver noise) must be taken into account. Note that this step is relatively independent of the choice of method formalization in Step 2.

In this study we use a IEEE wireless microphone signal test vector, a IEEE 802.15.4 direct sequence spread spectrum BPSK waveform and Gaussian noise, as discussed in section 5. Common alternative signal models seen in the literature are BPSK [10, 13, 28] and QPSK [25, 26] modulated carriers using various settings and an unmodulated carrier [28].

Step 4: Implementation

Next, the implementation details should be determined. This is useful for reproducibility, allowing other researchers to accurately and unambiguously repeat the results of the evaluation. In the case where software implementation is published as well, it also allows for code re-use. Furthermore, code additionally clarifies the signal detection methods used in case of any ambiguity in their mathematical description.

Implementation aspects for instance include the choice of receiver and signal generation hardware used in the lab test and description of any hardware signal processing components. Similarly for software components, detailed information on frameworks, libraries, languages and compilers is essential for rigorous evaluation. For the evaluation of computational complexity, methods should share as many software implementation details as possible for the results to be comparable.

In our evaluation, we implemented the nine selected methods in software, using a common software-defined radio (SDR) digital front-end as a receiver. Additionally, we implemented energy detection also with an analog energy detector. All software was written in an interpreted language due to its simplicity and speed of development. Signals were generated using a laboratory vector signal generator. The detailed description is provided in Section 6.

Step 5: Experimental setup

In this step, the complete experimental setup should be documented. In case of a lab test, this includes the physical setup and any extra equipment used that was not already described in the previous step. Device settings that affect the outcome of the evaluation should be documented as well. Values for any parameters specific to detection methods used should be determined, as well as values for parameters common to all methods.

We use a table-top setup with components connected via a coaxial cable. We perform our evaluation in the UHF frequency band and select sampling rates and sensing times consistent with our chosen TVWS device scenario. We chose to evaluate covariance- and eigenvalue-based detectors with and without filter compensation and with 4 values for the smoothing factor. We evaluate cyclostationary detection with 2 values for N′. Detailed description is in section 7.

Step 6: Evaluation procedures

The sixth step of our methodology requires documentation of evaluation procedures. This step should describe how the results of the evaluation were obtained using the setup described in the previous step.

We evaluate minimal detectable signal at a fixed probability of false alarm and minimal required probability of detection, noise level sensitivity and computational complexity. Detailed descriptions are in section 8.

Step 7: Evaluation results

The final step of our methodology discusses the evaluation results. This step is relevant for checking the correctness and completeness of the results and to validate these against theoretical findings. The results should be presented with estimated confidence intervals that present uncertainties in measurements and statistical analysis. Our results are presented in section 9.

4.1 Energy detection

All radio transmitters by definition emit energy into the electromagnetic field. Energy detection (ED) is the simplest spectrum sensing method. As such it is often used as a baseline when evaluating other spectrum sensing methods. Energy detection is the optimal non-coherent detection algorithm [30].

The test statistic γ for energy detectors is based on the definition of signal energy, which is related to the physical concept of energy. For a continuous signal, using the common notation from this paper, the formalization is: (5) For sampled-time real signals, the equation transforms into: (6) The largest disadvantage of energy detection is the fact that the test statistic is as sensitive to noise power as to the signal power. In practice the noise power varies. Internal receiver noise changes with production tolerances, temperature and other environmental factors. Environment noise changes with time and location. This leads to a SNR wall [31], where a detector is incapable of reliably detecting a signal, regardless of the choice of N_s, when signal power is small compared to noise power.

4.2 Covariance-based detection

Covariance methods attempt to solve the problem of unknown noise power by deriving the test statistic from sample covariance instead of variance (i.e. power). Covariance methods work on two basic assumptions: firstly, noise samples are independent and hence uncorrelated, and secondly, samples of realistic information-carrying signals are correlated to some degree [11].

Sample covariance estimates σ_l are calculated based on the vector of observed signal samples: (7) where L is a parameter called a smoothing factor, which determines the minimum time window within which the signal must exhibit correlation.

The test statistic is derived from a Toeplitz matrix of covariance estimates R. (8) In general, non-diagonal elements of R will be near zero in case of a vacant channel while diagonal elements are related to signal power and are not affected by correlations in the signal. Covariance methods calculate a scalar test statistic based on this observation: (9) where T₁ is related to non-diagonal elements of R and T₂ to diagonal elements. We tested several variations on this method with different definitions of these two variables. They are listed in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Covariance-based test statistics.

https://doi.org/10.1371/journal.pone.0199550.t001

While covariance-based methods are robust against changes in power of Gaussian noise, realistic signals also include noise samples that are not independent. This so-called noise coloring can be due to narrow-band filtering in the receiver analog front-end [11] or non-white natural and man-made external noise [33]. The sensitivity of covariance-based methods in practical scenarios is hence limited by the noise-coloring uncertainty.

Noise coloring due to time-invariant filtering can be compensated by using a transformed covariance matrix R′ when calculating test statistics [11]: (10) where Q can be pre-calculated for a known receiver from the covariance-matrix of noise-only samples R_n using the matrix square root: (11)

4.3 Eigenvalue-based detection

Eigenvalue-based detection methods are closely related to covariance-based detection. These methods attempt to trade off some of the simplicity for better signal detection performance [34]. Here, the test statistic is based on eigenvalues λ_l of the covariance matrix R, ordered by their absolute value: (12) Eigenvalue calculation requires on the order of L³ operations, however since L is typically small compared to N_s, the calculation of covariance estimates in (7) dominates computing time [34]. Several variations of Eigenvalue-based test statistics are listed in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Eigenvalue-based test statistics.

https://doi.org/10.1371/journal.pone.0199550.t002

Noise coloring compensation method in (10) can be applied to eigenvalue-based detection as well.

4.4 Cyclostationary detection

Cyclostationary detectors exploit the fact that practical information carrying signals often have a repeating structure over some length of time [35]. Such cycles can arise from the constant bit or symbol rate in digital transmissions or from constant carrier tones in analog transmissions. This is in contrast to noise, which is assumed to be a stationary signal.

In this paper, we based the cyclostationary detection on the commonly used spectral correlation function S_x(α, f): (13) (14) where α is the cyclic frequency and R_x is the cyclic correlation function.

It can be seen that S_x(α, f) will be zero for α ≠ 0 for stationary noise. On the other hand, a cyclostationary signal will have S_x(α, f) non-zero for specific values of α and f.

We approximate S_x(α, f) for its complete range of α ∈ [−f_s, f_s] and . This makes our detector sensitive to a wide range of cyclostationary signals. This approach overcomes a known shortcoming with detectors that assume a known α where sampling rate uncertainties affect the detector sensitivity [26].

We use the FFT accumulation method (FAM) [36] to approximate the spectral correlation function. FAM is based on sets of conventional Fourier transforms. An example plot of S_x(α, f) approximation showing characteristic diamond shapes is shown in Fig 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Spectral correlation function for a frequency modulated carrier (19), approximated using the FAM method from USRP measurements (N′ = 256).

https://doi.org/10.1371/journal.pone.0199550.g003

The FAM method takes several parameters. N′ is the size of the first FFT and determines the frequency resolution Δf, L is the decimation factor (also called smoothing factor and usually set to ) and is the size of the FFTs in the second set and determines the cyclic frequency resolution Δα. (15) (16) In our experiment we use the following test statistic that has been previously applied to wireless microphone detection [27]: (17) In respect to computational complexity, the dominating part of the FAM method is the calculation of the second set of Fourier transforms. This step requires N′² complex P-point transforms which in turn require 2N′² P log₂ P floating point multiplications [36].

5.1 Microphone signal

The IEEE wireless microphone test vectors are designed to emulate transmissions from legacy analog UHF wireless microphones, also called programme making and special events (PMSE) devices. In contrast to transmissions from real microphones that vary with audio input levels, these vectors were defined to provide deterministic waveforms that can be reliably reproduced in simulations and in laboratory spectrum sensing experiments. They can be reproduced using commonly available equipment. These test vectors are often used in the literature for signal detection evaluations [12, 27].

In [38], 6 different wireless microphone test vectors are defined. The document recommends that spectrum sensing schemes in the context of the IEEE 802.22 standard are evaluated using all 6 vectors. However due to limited time and a lack of a Rayleigh fading simulator we chose to perform our evaluation using only the “Outdoor, LOS, soft speaker” vector (same vector was used in [27]). Since our task was to perform a scientific evaluation and not evaluate a specific engineering solution for IEEE 802.22 networks, we considered one vector sufficient. Implementations for the other microphone test vectors are available in our source code repository (see Section 10).

The selected microphone test vector is based on a constant-amplitude, frequency modulated carrier using a single tone as a modulating signal. In an analog form it can be written as: (18) where A is the signal amplitude, f_c is the carrier frequency, f_m is the frequency of the modulating tone and f_dev is the frequency deviation. The soft speaker vector defines the following values (see Section 3, paragraph 2 in [38]): (19)

5.2 BPSK signal

The direct sequence spread spectrum BPSK waveform is one of the physical layers used by IEEE 802.15.4 low-rate wireless personal area networks in the sub 1-GHz spectrum. Our model follows the specification given in Section 11.2 of [38]. A sequence of binary symbols to be transmitted is first converted to a binary chip sequence using a static symbol-to-chip mapping table (see Table 78 in [38]). Each symbol is converted into a sequence of 15 chips.

Individual chips are transmitted as a raised cosine pulse with the roll-off factor of 1. The single pulse signal p(t) can be written as: (20) (21) where T_c is the chip period. The complete test vector can be written as a sum of pulses with pulse polarity depending on the chip value: (22) where A is the signal amplitude, f_c is the carrier frequency, N_c is number of chips in the test vector and c_n ∈ [0, 1] is the binary value of the n-th chip in the vector. For our evaluation we chose the chip rate specified for operation in the 868 MHz band: (23)

5.3 Noise

In our evaluation, two noise signals were added to the selected test signal: (24) u_int is the unavoidable noise generated by the equipment used in the experiment: internal noise of the receiver and the signal generator. The characteristics of this noise are defined by the construction of devices used in the experiment. We considered it colored and non-Gaussian.

u_ext is the external noise generated by the signal generator. This noise was zero-mean Gaussian with i.i.d. samples: (25) where is the selected noise power.

We selected Gaussian noise because all detection methods in our evaluation, except for energy detection, aim to be immune to changes in Gaussian noise power. By selecting this type of noise, we wanted to focus on this specific property of the methods, rather than on their robustness to more realistic sources of external noise. The latter could be a focus of a separate study. Furthermore, it should be noted that while the external noise model u_ext was Gaussian, the actual noise produced in our experiment was not Gaussian due to the practical limitations of our equipment, such as the limited signal generator bandwidth and waveform length.

6.1 Receiver hardware

In our evaluation we used two radio receivers: An Ettus Research USRP N200 device and an in-house developed SNE-ISMTV-UHF receiver. These devices provided raw data that was further processed in software.

Ettus Research USRP N200 device is a software-defined radio front-end [39]. The USRP series of devices commonly appears in the literature due to relatively low cost and high performance. It was chosen as a typical representative of a receiver used in a software-defined radio architecture.

USRP device in our evaluation was equipped with an SBX daughterboard and could record the baseband signal using 14-bit complex (I/Q) samples. It has an adjustable receive bandwidth and a sampling frequency f_sUSRP up to 50 Msample/s [39]. We used the GNU Radio framework to control the device from a personal computer [40].

In our experimental evaluation we used the USRP only as a digital receiver to record vectors of N_s unprocessed baseband signal samples. These vectors were then passed to our software implementation of (digital) energy detection, covariance, eigenvalue and cyclostationary methods as described in Section 6.3.

SNE-ISMTV-UHF receiver is a VHF/UHF band receiver with an analog energy detector [41] based on the VESNA sensor node platform [42]. The receiver has a configurable channel filter bandwidth and contains an analog logarithmic power detector [43]. The output of the power detector is sampled by a 12-bit ADC with a fixed sampling rate f_sISMTV = 147kHz. VESNA spectrum sensor application was used to record the data from the sensor [44].

Of the detection methods listed in Section 4, only energy detection can be practically implemented in the analog domain. Analog energy detectors are common in practical spectrum sensing experiments due to their simplicity, low power usage and low cost of implementation. In our experimental evaluation we used the SNE-ISMTV-UHF receiver to implement analog energy detection.

Since the analog detector sampling rate was fixed with this hardware, we implemented different sensing times t_s required by our evaluation by summing a number of consecutive detector samples in software. Summation was done in linear scale. Hence, to arrive to the test statistic γ approximating (5), we used the following expression: (26) where N_s is the number of detector samples, K₁ and K₂ are detector scaling constants and y_n is the n-th detector sample. N_s was calculated from the detector sampling rate: (27) Scaling constants were chosen such that the value of γ was within the floating point range used in the calculation.

6.2 Signal and noise generation

All signals in our experiment were generated using the Rohde&Schwarz SMBV100A vector signal generator [45].

The IEEE wireless microphone signal test vector was implemented using the analog modulation block. The block was set for frequency modulation. Modulating signal was internal sine wave source set to parameters given in (19).

The IEEE 802.15.4 direct sequence spread spectrum BPSK waveform and the Gaussian noise signal were implemented using the arbitrary waveform (ARB) block of the signal generator. The block was set to sampling frequency f_sDAC. A vector of N_sDAC I/Q samples was uploaded to the signal generator before the start of the experiment. The signal generator operated by continuously repeating the uploaded vector.

The I/Q vectors were generated in software on a personal computer. For the BPSK waveform, the symbol sequence entering the modulator was generated pseudo-randomly with a uniform distribution. Since the symbol sequence was random, the differential encoder specified in IEEE 802.15.4 was not implemented. Signal amplitude was full DAC range [−1, +1]. For the Gaussian noise signal, I/Q samples were similarly generated using a pseudo-random generator with a normal sample distribution. Standard deviation of digital samples was fixed at 0.19 to prevent excessive clipping. In both cases, the power of the actual signals was set by adjusting signal generator’s analog output level.

For noise signal, f_sDAC was selected to be sufficiently above the sampling rate of receivers so that noise samples appeared uncorrelated despite the finite bandwidth of the signal source. Similarly, N_sDAC was chosen to be large enough compared to the smoothing factor used in covariance- and eigenvalue-based detection: (28) where t_pDAC is the period of the signal generated by ARB, Δt is the time lag corresponding to the smoothing factor L.

6.3 Implementation of the methods

All software used in this evaluation was developed in the Python programming language (version 2.7). This choice was motivated by our knowledge of the language, a large set of available libraries for numeric calculation and good integration with both GNU Radio and VESNA libraries.

The software developed for this evaluation includes measurement automation, calculation of test statistics and data post-processing.

Measurements were fully automated. The experiment was controlled by a Python script running on a laptop PC. It recorded raw data from radio receivers to laptop’s hard drive and also controlled the signal generator according to the steps discussed in Section 8. The only manual intervention required was re-connecting the coaxial cable connection when changing the receiver.

The test statistics for energy detection, covariance, eigenvalue and cyclostationary detection discussed in Section 4 were also calculated from raw data using Python. We used NumPy and SciPy packages for fast vector operations and common mathematical functions [46].

When evaluating computational complexity of test statistic calculation, the Python interpreter was running on an Intel Xeon E5520 processor with 2.27 GHz CPU clock under GNU/Linux operating system. These benchmarks were using CPython 2.7.3 interpreter with NumPy 1.6.1 module.

Postprocessing, such as deriving metrics presented in Section 9, was performed using iPython notebooks.

The source code of Python scripts, as well as raw measurement data and iPython notebooks with calculations used in the evaluation are available on-line (https://github.com/avian2/spectrum-sensing-methods) enabling reproducibility and code re-use.

7.1 Device settings

The central frequency for the signal generator and the USRP receiver was set to f_c = 864MHz. This frequency corresponds to the channel reserved for unlicensed use of wireless microphones in Europe. For measurements done with SNE-ISMTV-UHF, f_c = 850MHz was used, which was the closest frequency supported by this device.

We chose three settings for the devices, shown in Table 3, representing different classes of sensing devices. Settings A and B represent capabilities of an embedded sensing device. The required sample buffer length and sample rate can be implemented with microcontrollers with integrated A/D converters currently on the market [47]. Even though a simpler radio front-end than the USRP would most likely be employed in such a device, measurements using these two settings give an estimate of the performance of sensing methods achievable on low-cost embedded systems. Lastly, setting C was chosen to test performance that could be expected from a high-end device. Therefore a higher sampling rate and a longer receive buffer have been used.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Device settings.

https://doi.org/10.1371/journal.pone.0199550.t003

SNE-ISMTV-UHF device was configured to use a 1.7 MHz analog channel filter. To obtain energy detection results from SNE-ISMTV-UHF that would be comparable to USRP measurements, we adjusted the number of detector samples so that the total sensing time was equal to that of USRP measurements. (29) The Rohde&Schwarz SMBV100A vector signal generator was configured to generate the signal according to the model from Section 5. The following ARB settings were used for BPSK waveform generation: (30) The following ARB settings were used for noise generation: (31)

8.1 Comparing minimal detectable signal

In our first experiment, we compared the minimal signal power that was detectable using a given hardware receiver, sensing method and test signal.

With the signal generator turned off we first obtained N_p = 1000 samples of the test statistic γ for each method. Then, using the method illustrated in Fig 2, we estimated the threshold γ₀ for the chosen probability of false alarm P_fa = 0.1.

With the signal generator turned on, set to output one of our test signals and output power P_g, we then obtained another set of N_p samples of the test statistic γ for each method. By comparing these samples to the chosen threshold, we estimated the probability of detection P_d for input power P_in. This procedure was repeated for each test signal.

We used the maximum likelihood estimation to estimate P_d from the N_p detection trials. To verify that the number of trials was sufficient for an accurate estimate, we also calculated the binomial confidence interval for P_d using the Clopper-Pearson method [48]. We chose this method because it gives accurate results when the estimated probability is close to zero or one (which was the case in our experiments) and because its implementation was readily available to us. In this verification we only considered the binomial detection process and assumed that the uncertainty of γ₀ determination did not significantly affect the confidence interval of the final result. We also did not account for uncertainties in devices, such as errors in calibration, temperature changes, uncertainties in connector attenuations, and so on. We assumed these effects on our results were minimal.

Input power P_in was calculated as: (32) where A is the measured attenuation between the signal generator output and receiver input. We recorded P_d for P_g between -100 dBm and -70 dBm. We considered a signal reliably detected if P_d > 0.9.

8.2 Comparing noise level sensitivity

To compare the sensitivity of the methods to differences in noise level, we used the signal generator to inject additional noise into the antenna connector of USRP receiver (SNE-ISMTV-UHF was not used in this part of the experiment).

With the signal generator turned on, set to Gaussian noise and output power P_g, we obtained a set of N_p samples of the test statistic γ for each method. By comparing γ samples to the threshold from the previous experiment, we estimated the probability of false alarm P_fa for change in noise power ΔP_noise. (33) where P_n is receiver’s internal noise power which was estimated using energy detection with the signal generator turned off.

We recorded P_fa for P_g between -110 dBm and -80 dBm. We considered that the noise floor change affected the detection when P_fa ≥ 0.2. This is twice the initially set value for P_fa used for the estimation of the threshold γ₀.

8.3 Effect of noise compensation

To evaluate the effect of noise compensation on the performance of covariance-based methods, we repeated the minimal detectable signal and noise level sensitivity experiments with the compensation method applied. We applied the time-invariant filter compensation in (10) on the covariance matrix before the calculation of the test statistics γ. Noise-only covariance matrix R_n was obtained from N_s samples, obtained when the signal generator was turned off.

8.4 Comparing computational complexity

To measure the γ function execution time, we used the timeit module from the Python standard library. As an input we used N_s = 25ksamples of Gaussian noise samples. For each function we did 10 campaigns of N_p = 1000 repetitions. We took the time from the fastest campaign as the relevant result and discarded the rest. This minimized the effect of other processes interfering with the measurement [49].

9.1 Comparing minimal detectable signal

Fig 6 through 11 show the lowest detectable power P_in for each sensing method. Figs 6–8 show detection of the microphone signal when using the settings A, B and C respectively. Figs 9–11 show detection of the BPSK signal when using the settings A, B and C respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Lowest detectable power P_in of the microphone signal for various methods with sensing time t = 25.0ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Lowest detectable power P_in of the microphone signal for various methods with sensing time t = 12.5ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Lowest detectable power P_in of the microphone signal for various methods with sensing time t = 10.0ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g008

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Lowest detectable power P_in of the BPSK signal for various methods with sensing time t = 25.0ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Lowest detectable power P_in of the BPSK signal for various methods with sensing time t = 12.5ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g010

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Lowest detectable power P_in of the BPSK signal for various methods with sensing time t = 10.0ms.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g011

The results of the experiment show that energy detector was able to detect weaker signals than other methods under all tested conditions. This is in contrast to simulations in [10–12], where the energy detector with known noise floor had worse performance than covariance detection. Similarly, simulations in [13] showed that a cyclostationary detector had significantly better performance than energy detection with a static threshold. This suggests that the noise in our setup did not consist solely of uncorrelated white thermal noise, which is the noise model used in these simulations. Noise sources, like the undesired coupling of signals in the circuit, could account for correlation and periodicity in noise samples.

Device settings have a significant effect on some methods. For instance, the MAC method is among the best performing methods for detecting the microphone signal with setting A and worst with setting C. On the other hand, EME and the SCF method for N′ = 128 produced consistently good results across all settings. The results for the CAV method with setting C also show that increasing the smoothing factor L does not always lead to better performance.

Our results show that most methods perform worse when detecting the BPSK signal compared to the microphone signal. The most affected were the covariance- and eigenvalue-based methods with lower sampling rates (settings A and B). With setting A, CAV, CFN and MET methods were not able to detect the signal with the chosen probability P_d even with the highest signal generator output power used in the experiment. We can attribute at least part of this effect to the fact that the BPSK signal occupies a significantly wider bandwidth than the microphone signal. This is consistent with theoretical analysis in [11]: Covariance- and eigenvalue-based methods rely on sample correlations for detection. These correlations appear when the bandwidth of the detected signal is significantly narrower than that of the receiver.

Cyclostationary detection performed well in comparison to other methods even with the lowest sampling rate, which is in contrast with findings in [16]. The SCF method also exhibited lower sensitivity with the BPSK signal than with the microphone signal, although the effect was less significant than with the covariance-based methods.

As expected, the results for energy detection methods were almost identical for both test signals. The performance of energy detection with USRP varies slightly between different settings. This can be attributed to the fact that total thermal noise power varies with the channel bandwidth. Therefore the energy detector’s signal-to-noise ratio, and with it its minimal detectable power, is inversely proportional to the sampling frequency. Similarly, minimal detectable power is also inversely proportional to the number of samples. These two factors match the seen variation to within 0.5 dB.

From the results it can also be seen that the analog implementation of energy detection consistently outperformed the digital implementation. It should be noted that SNE-ISMTV-UHF with a fixed channel bandwidth of 1.7 MHz had an advantage over USRP for settings B and C, which had broader channels and therefore also higher thermal noise power. The simpler SNE-ISMTV-UHF receiver also likely has a lower noise figure than the USRP.

9.2 Comparing noise level sensitivity

The results of this experiment are shown in Table 4. The energy detector only tolerated minimal noise power changes without a significant increase in P_fa, which is consistent with theory. The EME method, which also uses signal energy in its test statistic, suffered from a similar problem as the energy detector. All other methods under evaluation (SCF, CAV, CFN, MAC, MME, AGM and MET) tolerated a larger than 40 dB increase in noise power without a significant effect on P_fa.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Sensitivity to changes in noise floor for various methods.

https://doi.org/10.1371/journal.pone.0199550.t004

9.3 Effect of noise compensation

The results of the minimal detectable signal test with noise compensation enabled are shown in Figs 12 and 13 for the microphone and BPSK signals respectively. It can be seen that noise compensation decreases the minimal detectable signal power for covariance- and eigenvalue-based methods by 7.3 to 14.1 dB, depending on the test statistic. The only exception was the EME method, which was not significantly effected by the noise compensation. This result supports our earlier conclusion that the relatively low performance of covariance methods without noise compensation is due to correlations in the USRP receiver noise samples.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Lowest detectable power P_in of the microphone signal when noise compensation is used with covariance-based methods.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g012

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13. Lowest detectable power P_in of the BPSK signal when noise compensation is used with covariance-based methods.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g013

The results of the noise level sensitivity test are shown in Fig 14. Comparison of these results with those in Table 4 indicates that with noise compensation most covariance methods become extremely sensitive to noise level. The most insensitive was the MET detector which tolerated only 2.5 dB increase in noise level. This is consistent with [33] which predicts that static pre-whitening filters will not be effective.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Sensitivity to changes in noise floor for covariance-based methods with noise compensation.

Error bars show binomial confidence intervals for α = 10⁻⁶.

https://doi.org/10.1371/journal.pone.0199550.g014

This outcome can be explained by the fact that noise compensation matrix Q was pre-calculated from samples with signal generator turned off and hence only took into account the colored noise generated internally by the USRP receiver. External noise applied in this experiment was white Gaussian noise. When added to internal USRP noise, Gaussian noise produced different correlations in signal samples than those compensated for. Hence from the standpoint of the compensated covariance detectors, adding external Gaussian noise had the same effect as adding signal with correlated samples.

9.4 Comparing computational complexity

Benchmark results are shown in Fig 15. Time required to calculate γ is compared to minimal detectable power results from Section 9.1 in Fig 16. For comparison, we have also approximated the number of required floating point multiplications for each detector, which is a common way of specifying algorithm complexity. The equations used for approximations are listed in Table 5. Floating point multiplication counts for specific settings used in our experiment are compared graphically in Fig 17.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Processing time required for calculation of test statistic γ.

https://doi.org/10.1371/journal.pone.0199550.g015

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Comparison of processing time versus minimal detectable power.

https://doi.org/10.1371/journal.pone.0199550.g016

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Approx. number of floating point multiplications.

https://doi.org/10.1371/journal.pone.0199550.t005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 17. Floating point multiplications required for calculation of test statistic γ.

Numbers in parenthesis are relative to energy detection. Approximations for other covariance-based detectors are equal to CAV.

https://doi.org/10.1371/journal.pone.0199550.g017

From the figures it can be seen that measurements do in fact follow the general trends predicted by theory. For example, the value of L significantly affects the processing time required for covariance-based detectors. There is also very little difference in the time required for test statistics that involve calculation of eigenvalues, confirming the assumption that eigenvalue calculation time is negligible. We can also see that cyclostationary detection is many orders of magnitude slower than energy detection or covariance-based detection. This is in contrast with results in [10] where covariance-based methods were found to have very high complexity.

However, ratios between energy detection and other methods of detection do not fit well. That can be explained by the fact that processing time of a script in a high-level programming language like Python is affected by many complicated factors, not only the number of multiplications. For example, choosing different vectorization methods in NumPy can significantly affect the processing time due to interpreter overhead and processor cache effects.

The Python interpreter has a significant overhead compared to native code. Even though the NumPy module contains optimized code for matrix operations, we believe these results are not an absolute measure of how optimized native code implementations would perform. Hence they are only useful for a relative comparison, i.e. to compare the methods against each other. Therefore we have normalized the results to the execution time of the energy detector.