Skip to main content

Head-to-head comparison of the accuracy of saliva and nasal rapid antigen SARS-CoV-2 self-testing: cross-sectional study

Abstract

Background

The diagnostic accuracy of unsupervised self-testing with rapid antigen diagnostic tests (Ag-RDTs) is mostly unknown. We studied the diagnostic accuracy of a self-performed SARS-CoV-2 saliva and nasal Ag-RDT in the general population.

Methods

This large cross-sectional study consecutively included unselected individuals aged \(\ge\) 16 years presenting for SARS-CoV-2 testing at three public health service test sites. Participants underwent molecular test sampling and received two self-tests (the Hangzhou AllTest Biotech saliva self-test and the SD Biosensor nasal self-test by Roche Diagnostics) to perform themselves at home. Diagnostic accuracy of both self-tests was assessed with molecular testing as reference.

Results

Out of 2819 participants, 6.5% had a positive molecular test. Overall sensitivities were 46.7% (39.3–54.2%) for the saliva Ag-RDT and 68.9% (61.6–75.6%) for the nasal Ag-RDT. With a viral load cut-off (≥ 5.2 log10 SARS-CoV-2 E-gene copies/mL) as a proxy of infectiousness, these sensitivities increased to 54.9% (46.4–63.3%) and 83.9% (76.9–89.5%), respectively. For the nasal Ag-RDT, sensitivities were 78.5% (71.1–84.8%) and 22.6% (9.6–41.1%) in those symptomatic and asymptomatic at the time of sampling, which increased to 90.4% (83.8–94.9%) and 38.9% (17.3–64.3%) after applying the viral load cut-off. In those with and without prior SARS-CoV-2 infection, sensitivities were 36.8% (16.3–61.6%) and 72.7% (65.1–79.4%). Specificities were > 99% and > 99%, positive predictive values > 70% and > 90%, and negative predictive values > 95% and > 95%, for the saliva and nasal Ag-RDT, respectively, in most analyses. Most participants considered the self-performing and result interpretation (very) easy for both self-tests.

Conclusions

The Hangzhou AllTest Biotech saliva self Ag-RDT is not reliable for SARS-CoV-2 detection, overall, and in all studied subgroups. The SD Biosensor nasal self Ag-RDT had high sensitivity in individuals with symptoms and in those without prior SARS-CoV-2 infection but low sensitivity in asymptomatic individuals and those with a prior SARS-CoV-2 infection which warrants further investigation.

Peer Review reports

Background

A molecular test, mainly real-time reverse-transcriptase polymerase chain reaction (RT-PCR), is considered the reference test for SARS-CoV-2 infection detection [1]. However, molecular tests may take up to 24 h to deliver a result. Tested individuals are asked to quarantine until they receive a result, which has personal and societal consequences. SARS-CoV-2 rapid antigen diagnostic tests (Ag-RDTs) have shown promising diagnostic accuracies [2,3,4,5,6]. These Ag-RDTs require no or minimal equipment, provide a result within 15–30 min and can be performed in a range of settings. While Ag-RDTs were initially introduced for use by trained staff at test sites, they can now also be bought in various outlets for self-testing. Such self-testing, without supervision of a trained professional, may potentially lower the threshold to testing and would allow individuals to obtain a test result quickly and at their own convenience. This in turn could support the early detection of infectious cases and reduce community transmission [7].

Previous studies of self-performed nasal Ag-RDTs showed sensitivities of close to 80% for the Becton Dickinson nasal Ag-RDT in a mixed population of asymptomatic and symptomatic individuals and 82.5% for the SD Biosensor nasal Ag-RDT in symptomatic individuals [8, 9]. Studies employing the SD Biosensor and Abbott Ag-RDTs have shown that supervised nasal self-sampling might be a reliable alternative to nasopharyngeal sampling by a trained professional [10, 11]. However, the sample sizes of these studies were modest, and the nasal self-sampling was supervised instead of self-performed in the home setting. The diagnostic accuracy evidence of self-performed saliva Ag-RDTs is scarce. A recent study found poor performance of four different (unspecified) self-collected saliva Ag-RDTs in a mixed population of symptomatic and asymptomatic individuals, with sensitivities varying from 3.6 to 32.8%, increasing to 5.3% and 41.0% when a “cell culture viability” cut-off was used [12]. A recent saliva Ag-RDT study employing self-sampling supervised by a trained professional and testing by that trained professional in a mixed population of symptomatic and asymptomatic individuals found a sensitivity of 66%, which increased to 89% when a cycle threshold (Ct) < 30 cut-off was used [13].

We conducted a large-scale prospective cross-sectional diagnostic accuracy study in the Netherlands of a self-performed saliva and a self-performed nasal Ag-RDT, using a molecular test as the reference standard for each, by head-to-head comparison. We included individuals presenting for routine SARS-CoV-2 testing at Dutch public health service test sites regardless of their reason for testing, vaccination status, and symptomatology at the time of sampling. A secondary aim was to evaluate user experiences and preferences for both self-performed Ag-RDTs.

Methods

The study is reported according to the STARD 2015 guidelines: an updated list of essential items for reporting diagnostic accuracy studies [14].

Study design and population

This large prospective cross-sectional diagnostic test accuracy study was embedded within the Dutch public testing infrastructure. Public testing in the Netherlands, by default molecular testing, is free-of-charge but only available for government-approved test indications. At the time of the study (9 to 26 September 2021), testing indications included having symptoms of a potential SARS-CoV-2 infection; having been identified as a close contact of a SARS-CoV-2 index case via traditional contact-tracing or the contact-tracing app regardless of symptomatology at the time of notification; having tested positive on a nasal self-test performed outside of this study; or having returned from a country on the government’s list of high risk countries [15]. Participants were recruited consecutively at three Dutch public health service COVID-19 test sites, located in West-Brabant (Roosendaal), Central- and Northeast Brabant (Tilburg), and Rotterdam-Rijnmond (Zuidland). Individuals were considered eligible if they were aged 16 years or older and willing and able to sign a digital informed consent in Dutch.

The study was conducted when the SARS-CoV-2 prevalence was 8.2% with the Delta variant as the dominant variant in the Netherlands (99.9%) during the entire study period [16,17,18].

Inclusion procedure

Individuals who attended one of the participating test sites for a routine molecular SARS-CoV-2 test were asked by the test site staff whether they were willing to participate. If interested, they received a participant information letter, the saliva and the nasal Ag-RDT together with an instruction manual, and an email with a study participation link to access study documentation. Next, trained test site staff took a swab for routine molecular testing (see below). Participants were asked to provide informed consent electronically via the participation link after arriving home, to subsequently perform both self-tests as soon as possible but within 3 h (the saliva test first, followed by the nasal test), and to complete a short online baseline questionnaire. This included questions on demographics; presence, type, and onset of COVID-19-related symptoms; indication for testing; vaccination status including type of vaccine and vaccination date(s); the results of the two Ag-RDTs that they had just performed; and their user experiences with and opinions about both Ag-RDTs (Additional file: material 1). Participants whose online questionnaire was not completed within 3 h of their test site visit were contacted by a call center with the request to perform both self-tests and complete the questionnaire as soon as possible.

Ten days after their test site visit, participants received an email asking them to complete an online follow-up questionnaire. The follow-up questionnaire included questions on COVID-19-related symptoms and SARS-CoV-2 testing during follow-up (Additional file: material 2) to capture any infections that may have been missed by the baseline molecular test.

Specimen collection and testing

Molecular reference test sampling was performed by trained test site staff. While molecular testing was always used as the reference standard, the three test sites used slightly different sampling methods and the three affiliated centralized laboratories used slightly different molecular testing methods (Additional file: material 3). Briefly, the Roosendaal site combined oropharyngeal-nasal sampling with RT-PCR testing on a Roche cobas 8800 platform. The Tilburg site combined oropharyngeal and nasopharyngeal sampling with the Abbott Alinity M SARS-CoV-2 assay or in-house RT-PCR [19]; samples that tested positive by RT-PCR in Tilburg were subsequently tested on the Roche cobas 8800 platform in Roosendaal to obtain Ct values for viral load calculation. The Zuidland site combined oropharyngeal and nasopharyngeal sampling with RT-PCR on a Roche cobas 6800 platform.

Participants performed the saliva and nasal Ag-RDTs themselves in their own homes (i.e., unsupervised) according to the instructions in the manual that they received at the test site; those instructions were identical to those provided by the manufacturers but translated into Dutch. The saliva Ag-RDT that we used was the COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co. Ltd. and is a so-called spitting test: participants had to spit in a funnel connected to a tube. The nasal Ag-RDT that we used was the SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing, distributed by Roche Diagnostics (Additional file: material 3). Both tests are CE-marked, and the Hangzhou AllTest Biotech was the only saliva self-test with CE-marking at the time of study conception. Participants interpreted their Ag-RDT test results visually in accordance with the instructions, and this interpretation was always done before they had received their molecular test result. Vice versa, the Ag-RDTs results were not available to those in the study team assessing the molecular test results. Participants received their molecular test results from the public health services test site that they attended conform routine practice to direct any further management (such as quarantine advice, if applicable).

Outcomes and statistical analyses

The primary outcomes were the diagnostic accuracies (sensitivity, specificity, positive and negative predictive values with corresponding exact 95% confidence intervals [CI] [20]) of each self-test, with molecular testing as the reference standard. We performed a complete cases analysis because the number of individuals without molecular test or Ag-RDT results was very low (n = 147 (5.0%) for the saliva Ag-RDT, and n = 131 (4.4%) for the nasal Ag-RDT); Fig. 1).

Fig. 1
figure 1

Flow of study participants. Saliva = COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co., Ltd., Nasal = SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing by Roche Diagnostics

Secondary outcomes were diagnostic accuracies stratified by presence of symptoms at the time of sampling (yes or no), COVID-19 vaccination status (vaccinated with at least one dose yes or no), having had a prior SARS-CoV-2 infection (yes or no), gender (female or male), and age (≥ 16 to ≤ 40 or > 40 to ≤ 65 or > 65 years). Additional secondary outcomes included all of the above but after using a viral load cut-off as a proxy of infectiousness (≥ 5.2 log10 SARS-CoV-2 E-gene copies/mL), which was the viral load cut-off above which 95% of people with a positive molecular test had a positive virus culture in a recent study by our group [3]. In addition, we assessed the user experiences and preferences for both self-performed Ag-RDTs and tested for overall differences across Ag-RDTs using a McNemar-Bowker test and for differences across categories using a McNemar test.

Finally, using the follow-up questionnaire, we determined whether participants who received a negative molecular test result at baseline had tested positive in the subsequent 10 days by either molecular test or Ag-RDT.

Sample size calculation

Previous nasal Ag-RDTs performance studies in symptomatic individuals found sensitivities around 85% when performed by trained staff [2] and around 80% when used as a self-test [8]. A recent accuracy study in The Netherlands that quantified the accuracy of saliva Ag-RDT performed by trained professionals found an overall sensitivity of 66% and around 89% when using a Ct < 30 cut-off [13]. We therefore based our sample size calculation on an expected sensitivity of 80% for each self-performed Ag-RDT, with a margin of error of 7%, type I error of 5%, and power of 80%. Hence, we aimed for approximately 140 positive molecular reference tests per Ag-RDT evaluation. When planning the study (June 2021), we anticipated a SARS-CoV-2 prevalence (based on molecular test positivity rates) in our target population of around 5%, and closely monitored molecular test positivity rates over time to prolong recruitment as needed.

Results

Between 9 and 26 September 2021, 2950 individuals participated in the study (Fig. 1). An Ag-RDT result with matching molecular reference test result were available for 2803 saliva Ag-RDT users (95.0%) and 2819 nasal Ag-RDT users (95.5%).

The demographic characteristics of study participants are presented in Table 1 and Additional file: Table S1. The mean age was 41 years (standard deviation 15.5) and 61% was female. The majority was vaccinated (85% once; 74% twice), had not had a previous SARS-CoV-2 infection (87%), and was symptomatic at the time of sampling (83%). These characteristics were comparable across test sites, although participants presenting at the Zuidland location were less often vaccinated than those presenting in Roosendaal and Tilburg (76.8% vs. 85.2% vs. 87.5%).

Table 1 Baseline characteristics of the study population, stratified by type of Ag-RDT

Table 2 and Fig. 2 show the results of the primary analysis and the secondary stratified analyses. Additional file: Fig. S1 presents them after the application of a viral load cut-off as a proxy for infectiousness. Additional file: Tables S1 and S2 show 2 × 2 tables for both Ag-RDTs. The main findings are presented in the text below.

Table 2 Diagnostic accuracy parameters for the saliva and nasal Ag-RDTs. Values are percentages (95% confidence interval) unless stated otherwise
Fig. 2
figure 2

Sensitivities with 95% confidence intervals of the Ag-RDT-molecular reference standard test comparisons stratified according to symptomatology, COVID-19 vaccination status, previous infection status, sex, and age. The vertical line indicates the sensitivity of the Ag-RDT in the overall study population, and the number of positive molecular tests out of the total or subgroup between parentheses

Self-performed saliva Ag-RDT

Overall test accuracy

SARS-CoV-2 molecular test positivity was 6.5% (182/2803) and overall sensitivity was 46.7% (85/182; 95% CI 39.3–54.2%; Table 2, Fig. 2). Among those with a positive molecular test result, the percentage of participants with a viral load above the cut-off as a proxy for infectiousness was 78.0% (142/182). Using this viral load cut-off, the overall sensitivity was 54.9% (78/142; 46.4–63.3%; Table 2, Fig. 2). Specificities were around 99% in both analyses; positive and negative predictive values are presented in Table 2.

Stratified analyses

The sensitivity was 51.0% (77/151; 42.7–59.2%) in participants who were symptomatic at the time of testing and 25.8% (8/31; 11.9–44.6%) in participants who were asymptomatic (Table 2, Fig. 2). After application of a viral load cut-off as a proxy for infectiousness, these sensitivities were 57.3% (71/124; 48.1–66.1%) and 38.9% (7/18; 17.3–64.3%), respectively (Additional file: Fig. S1). The sensitivity was 20.0% (4/20; (5.7–43.7%) in participants who had had a previous SARS-CoV-2 infection and 50.0% (81/162; 42.1–57.9%) in participants who had never had a prior infection (Table 2, Fig. 2). After applying the viral load cut-off, these sensitivities were 50.0% (4/8; 15.7–84.3%) and 55.2% (74/134; 46.4–63.8%), respectively (Additional file: Fig. S1). In males the sensitivity was higher than in females, 60.3% (44/73; 48.1–71.5%) versus 38.0% (41/108; 28.8–47.8%), and increased slightly by increasing age (Table 2, Fig. 2). We found no evidence of a differential impact on diagnostic accuracy by COVID-19 vaccination status. The sensitivities after application of a viral load cut-off as a proxy for infectiousness are presented in Additional file: Fig. S1. Specificities were > 99% and positive predictive values > 70% and negative predictive values > 9 5% in most analyses (Table 2).

Self-performed nasal Ag-RDT

Overall test accuracy

SARS-CoV-2 molecular test positivity was 6.4% (180/2819) and overall sensitivity was 68.9% (124/180; 61.6–75.6%; Table 2, Fig. 2). Among those with a positive molecular test result, the percentage of participants with a viral load above the cut-off as a proxy for infectiousness was 79.4% (143/180). Using this viral load cut-off, the overall sensitivity was 83.9% (120/143; 76.9–89.5%; Table 2, Fig. 2). Specificities were 99.5% in both analyses; positive and negative predictive values are presented in Table 2.

Stratified analyses

The sensitivity was 78.5% (117/149; 71.1–84.8%) in participants who were symptomatic at the time of testing and 22.6% (7/31; 9.6–41.1%) in participants who were asymptomatic (Table 2, Fig. 2). After application of a viral load cut-off as a proxy for infectiousness, these sensitivities were 90.4% (113/125; 83.8–94.9%) and 38.9% (7/18; 17.3–64.3%), respectively (Additional file: Fig. S1). On average, the viral load was lower in asymptomatic than in symptomatic individuals with a positive molecular reference test (Additional file: Fig. S2). The sensitivity was 36.8% (7/19; 16.3–61.6%) in participants who had had a previous SARS-CoV-2 infection and 72.7% (117/161; 65.1–79.4%) in participants who had never had a prior infection. After applying the viral load cut-off, these sensitivities were 100.0% (7/7; 59.0–100.0%) and 83.1% (113/126; 75.7–89.0%), respectively (Additional file: Fig. S1). We found no evidence of a differential impact on diagnostic accuracy by COVID-19 vaccination status, sex, and age (Fig. 2, Additional file: Table S3). Specificities were > 99% and positive predictive values > 90% and negative predictive values > 95% in most analyses (Table 2).

Diagnostic test accuracy results for both Ag-RDTs were the same when the analysis populations were limited to the 2746 participants for whom all three test results were available (data not shown).

User experiences and preferences

Participants indicated that taking a sample was easier for the self-performed saliva Ag-RDT than for the self-performed nasal Ag-RDT (p < 0.0001 by McNemar-Bowker test), with a larger proportion of participants indicating it was easy or very easy for the saliva Ag-RDT than for the nasal Ag-RDT (80.4% vs. 66.8% (p < 0.001 by McNemar test). Most participants reported that reading the test result was easy or very easy for both tests (92.2% vs. 92.4%; Table 3; p = 0.82 by McNemar test). Distributions of change scores for both usability components are presented in Additional file: Fig. S3. After performing both Ag-RDTs, but before receiving the molecular test result, 55.6% of the participants reported to prefer the saliva over the nasal Ag-RDT, 9.3% reported to prefer the nasal Ag-RDT, and 32.8% indicated to have no preference.

Table 3 Usability of the saliva and nasal Ag-RDTs

Follow-up

Follow-up information was available for 72% of participants (Table S3), of whom 1994 participants had a negative molecular test result at baseline. Of the latter group, 249/1994 (12.5%) were re-tested within 10 days, and 7/249 (2.8%) tested positive, wherein we did not know whether it was a new infection, or the initial molecular test was false negative.

Discussion

This largest diagnostic accuracy evaluation of two unsupervised self-performed Ag-RDTs to date showed a low overall sensitivity (46.7%) of the Hangzhou AllTest Biotech saliva self-test. Applying a viral load cut-off as a proxy for infectiousness did not improve the overall sensitivity meaningfully (54.9%) nor in any of the studied subgroups, with all sensitivities remaining far below the WHO standard of 80% [21].

The study showed better performance of the SD Biosensor nasal self-test with an overall sensitivity of 68.9%, increasing to 83.9% when the viral load cut-off was applied. The sensitivities were much higher in the 2319 symptomatic participants than in the 487 asymptomatic participants (78.5% and 22.6%, respectively) and in the 2417 individuals who never had COVID-19 in the past compared to the 372 individuals who did (72.7% and 36.8%, respectively), reaching sensitivities (with sufficient precision) above the WHO-recommended 80% in symptomatic individuals and in individuals without a previous infection after applying the viral load cut-off. The sensitivities in asymptomatic individuals and in individuals who had had COVID-19 in the past have wider 95% confidence intervals and should therefore be interpreted with caution. We recommend additional research in those groups. The diagnostic accuracy of this nasal self-test did not differ by COVID-19 vaccination status, sex, and age.

Discussion of the saliva self-test results

We identified two previous studies in the scientific literature; the sensitivities observed in our study were in between those found in the two studies. A Czech study evaluated four saliva Ag-RDTs and found sensitivities of 15% for a saliva test that required spitting in a cup, and 3.6%, 25.5%, and 32.8% for saliva tests requiring sucking on a sponge, in comparison with RT-PCR [12]. These sensitivities improved slightly after using a “cell culture viability” cut-off but remained well below 50%. The sampling was done by the participants themselves but supervised by trained personnel. Samples sizes were modest, ranging from 98 to 407 participants per evaluated test. A recent Dutch study of the SD Biosensor saliva test with 789 participants found a sensitivity of 66.1%, increasing to 88.6% when Ct < 30 and to 96.7% when viral culturability was used as cut-offs [13]. In the Dutch study, saliva was collected by letting nasal and cough discharge drool into a collection device and was supervised by trained test site staff. Sensitivity was lower (60%) in asymptomatic participants but only 10 asymptomatic participants tested RT-PCR positive. We tested the analytical performance of the Hangzhou and SD Biosensor lateral flow test devices on calibrated samples and found that both test devices performed (equally) well (Supplement 3). We therefore hypothesize that the widely ranging sensitivity results for saliva Ag-RDTs may be due to high variability in saliva sampling methods (spitting vs. sucking vs. drooling) and/or high variability in the quantity and quality of sample self-obtained by different individuals. Furthermore, saliva specimens may on average contain lower SARS-CoV-2 viral loads than upper respiratory tract samples. Studies have shown that saliva viral loads are usually sufficiently high for detection by molecular methods [22,23,24], but they may not be sufficiently high for detection by self-performed Ag-RDTs.

We saw trends of reduced diagnostic accuracy in persons without symptoms or with previous SARS-CoV-2 infection. These trends were like the trends that we observed for the nasal self-test and are discussed below. We also saw trends by gender and age, with sensitivities for men and for persons aged over 65 reaching around 60%, which is still well below the WHO-recommended 80% [21]. The saliva Ag-RDT evaluation studies to date did not stratify by gender and age [12, 13], and the nasal Ag-RDT studies, including the nasal self-test that we evaluated in this study, did not show these trends [3, 6]. We recommend that future saliva Ag-RDT evaluations stratify by gender and age to investigate this further.

Discussion of the nasal self-test results

The diagnostic performance of Ag-RDTs combined with nasopharyngeal sampling done by trained personnel or by individuals themselves have been evaluated extensively by us and others [2,3,4,5,6, 8, 9, 12]. The above-mentioned Czech study also evaluated an Ag-RDT in combination with anterior nasal sampling done by trained personnel [12]. These studies found good performance (70–80%) with nasopharyngeal sampling but lower performance with anterior nasal sampling (45–55%) and also lower performance in asymptomatic individuals (50–60%), regardless of self or professional sampling. Our study showed that nasal self-sampling with the SD Biosensor Ag-RDT provided good sensitivity, which equaled the sensitivity of Ag-RDTs found in other studies in which the nasal sampling was done by a trained professional [10, 11], but only for individuals who have symptoms at the time of testing. We found a very low sensitivity of this self-performed nasal Ag-RDT of only 23% in asymptomatic individuals, which is much lower than the sensitivities found in our previous studies using nasopharyngeal or oropharyngeal combined with nasal sampling done by trained personnel [3, 6]. This difference in performance persisted after applying a viral load cut-off. It is currently unclear why the sensitivities of the nasal Ag-RDT self-test differed depending on the presence of symptoms, even after applying the viral load cut-off. We tested the analytical performance of the SD Biosensor lateral flow test device on calibrated samples and found that the test device itself performed well (Supplement 3). We hypothesize that the difference in sensitivity may be explained by the difference in viral load distributions in asymptomatic and symptomatic individuals. In addition, it may be more difficult for asymptomatic individuals (i.e., with a dry nose) to retrieve sufficient nasal fluid by self-swabbing. The former hypothesis is supported by the fact that the sensitivity of the saliva self-test was also lower in asymptomatic than symptomatic individuals, but both hypotheses might play a role.

We also found a low sensitivity (36.8%) of the nasal self-test in individuals who had had COVID-19 in the past. These results should be interpreted with caution due to the small group sizes: only 19 participants with a positive molecular test reported having had COVID-19 (16 of whom were symptomatic at the time of testing), and only seven of them had a viral load above the viral load cut-off (six of whom were symptomatic). However, similar trends were observed for the saliva self-test in this study and for the SD Biosensor Ag-RDT conducted by trained staff in a previous study [6]. In that study, sensitivities were 54.5% for oropharyngeal-nasal sampling and 68.4% for nasopharyngeal sampling in individuals with prior infection, and 75.8% and 75.0%, respectively, in individuals without prior infection. The low sensitivity in individuals with a prior infection may be explained by lower viral loads in this group (in the current study, 12/19 participants with a prior infection were below the viral load cut-off compared to 113/161 in those without a prior infection), with some of them potentially carrying viral RNA in the absence of a productive infection (i.e., no viral antigen production). Another explanation might be that individuals who have had COVID-19 have circulating anti-nuclear capsid (N) protein antibodies [25]. These anti-N antibodies might bind to the N protein that is produced during the new infection, hampering the binding of monoclonal antibodies against the N-protein in the test device. It should be noted that we found a smaller reduced sensitivity of the BD Veritor Ag-RDT conducted by trained staff (oropharyngeal-nasal sampling) in individuals with and without a prior infection (64.6% versus 70.1%), so this effect may be test device-specific [6].

Strengths and limitations of this study

Strengths of this study include the large overall sample size covering multiple test sites nationwide, the collection of samples for the reference test and two Ag-RDTs in the same individuals within a few hours allowing for a head-to-head comparison of the two self-tests, the fact that sampling was done by the participants themselves without any supervision conform the real-world context of self-testing, that the index test was blinded for the reference test result and vice versa, and the use of a proxy for infectiousness. Furthermore, the follow-up information showed that very few infections were missed by the molecular reference tests.

Our study also has some limitations. First, the reference standards that we used were molecular tests, but platforms and test kits used differed among the centralized laboratories. However, the diagnostic accuracies of all molecular tests used are similarly high [26, 27], and we therefore believe that this has not influenced our findings significantly. In addition, Ct values used to calculate viral loads were determined by different yet comparable platforms (Additional file: material 3). Second, we used the viral load cut-off above which 95% of people with a positive RT-PCR test result had a positive virus culture as a proxy of infectiousness. Although this cut-off is not fully evidence based [3], it is a best estimate based on current knowledge and less arbitrary than using Ct cut-offs of 25 or 30 as is often done [28, 29]. In the current study, we relied on infectiousness viral load cut-offs that were determined in our previous study in a mainly unvaccinated population (the proportion of vaccinated individuals in the present study reached 85% at the end of the study) and when a different SARS-CoV-2 variant was dominant [3]. Whether this would have impacted the applied viral load cut-offs is unknown, but vaccination itself did not influence any of the test sensitivities. Third, our sample size calculation was based on the primary analysis and the diagnostic accuracy parameters are less precise for the secondary stratified analyses. Fourth, participants were not blinded to the results of the saliva Ag-RDT when interpreting the result of the nasal Ag-RDT, which could have potentially biased the test outcome assessment of the nasal Ag-RDT. We do, however, believe that the impact of this limitation is small considering that the interpretation of the test results was considered (very) easy by > 95% of participants, and the performance of each Ag-RDT was substantially different and higher for the last performed nasal Ag-RDT. If outcome assessment was biased, the diagnostic performance of the self-tests would likely have been more similar. Fifth, we had some, though very limited, missing index test data (5%). We did not perform multiple imputation techniques because the group with missing data was very similar to the group with complete data, suggesting that data was missing completely at random.

Policy implications

Ag-RDTs for self-use are widely available in the Netherlands. Until recently, the recommendation was to use them when asymptomatic prior to having contacts (such as going to school, events, or work) and visit a public health test site for molecular testing when symptomatic. Individuals whose self-test was positive are (still) asked to visit a public health test site for confirmatory testing. The SD Biosensor nasal self-test that we evaluated in this study is one of the self-tests that is commercially available in the Netherlands, although we do not know its market share. Our results indicate that the SD Biosensor nasal self-test sensitivity among individuals with mild symptoms is similar to Ag-RDT sensitivities found in studies where it was applied to professionally obtain upper respiratory tract samples. Based on those results, the Dutch Outbreak Management Team (OMT) that advises the Ministry of Health, Welfare, and Sports regarding COVID-19 policy recommended expanding nasal self-testing to individuals with mild symptoms. The OMT stressed that self-tests are not advised in vulnerable persons, in individuals meeting vulnerable persons, and in case of more severe symptoms, and that a negative self-test result is not sufficient for ending quarantine for contacts of a confirmed case. However, individuals testing negative by nasal self-testing would be allowed to go to work (if not working with vulnerable persons) or school despite their mild symptoms, preferably using mouth-nose masks and testing again a day later in case the initial test result was negative. Individuals testing positive by nasal self-testing would have to self-isolate and visit a public health test site for confirmatory molecular testing, to keep track of virus spreading and to allow for contact-tracing. All these nuances require careful communication, including on the implications of false-negative test results.

The very low sensitivity of the SD Biosensor nasal self-test in asymptomatic individuals is worrisome, even though the a-priori probability of being infected is lower in asymptomatic than symptomatic individuals. In addition, the potentially reduced sensitivity of the SD Biosensor nasal self-test in individuals who have had COVID-19 in the past is also worrisome. This is especially important because an increasing proportion of the population will have had COVID-19. We call for additional research in these two specific subgroups. We also recommend that persons who tested negative by a self-test continue to adhere to the general preventive measures such as physical distancing, wearing mouth-nose masks, and washing hands.

The SD Biosensor nasal self-test is only one of the commercially available self-tests. We recommend that all available self-tests are evaluated urgently by independent researchers, also addressing the relevant subgroups. Finally, in high-risk situations, such as testing of vulnerable people in care facilities, severely ill patients, or healthcare workers, we recommend molecular testing at all times, which is already in line with current policy.

Conclusions

The Hangzhou AllTest Biotech saliva self Ag-RDT is not reliable for SARS-CoV-2 detection, overall, and in all studied subgroups. The SD Biosensor nasal self Ag-RDT had high sensitivity in individuals with COVID-19 like symptoms and in those without a prior SARS-CoV-2 infection but a very low sensitivity in asymptomatic individuals and in those with a prior SARS-CoV-2 infection warrants further investigation.

Availability of data and materials

Individual participant data collected during the study will be available, after deidentification of all participants. Data will be available to researchers who provide a methodologically sound proposal to achieve the aims in the approved proposal. Proposals should be directed to the corresponding author to gain access to the data. Data requestors will need to sign a data sharing agreement. The study protocol is available upon request by contacting Karel Moons at k.g.m.moons@umcutrecht.nl.

Abbreviations

Ag-RDTs:

Rapid lateral flow antigen diagnostic tests

CI:

Confidence interval

COVID-19:

Coronavirus disease 2019

IQR:

Interquartile range

Nasal Ag-RDT:

SD Biosensor SARS-CoV-2 Rapid Antigen Test Nasal for self-testing, distributed by Roche Diagnostics

RT-PCR:

Real-time reverse-transcriptase polymerase chain reaction

SARS-CoV-2:

Severe acute respiratory syndrome coronavirus-2

Saliva Ag-RDT:

COVID-19 Antigen Rapid Test (Oral Fluid) for Self-testing by Hangzhou AllTest Biotech Co. Ltd.

STARD 2015:

Standards for Reporting of Diagnostic Accuracy Studies 2015

References

  1. WHO. Recommendations for national SARS-CoV-2 testing strategies and diagnostic capacities: interim guidance 25 June 2021. 2021. p. 1–16.

    Google Scholar 

  2. RIVM Centrum Infectieziektebestrijding. Status validatie SARS-CoV-2 antigeen sneltesten, 10 Mar 2021 [Dutch]. 2021.

    Google Scholar 

  3. Schuit E, Veldhuijzen IK, Venekamp RP, van den Bijllaardt W, Pas SD, Lodder EB, Molenkamp R, GeurtsvanKessel CH, Velzing J, Huisman RC, et al. Diagnostic accuracy of rapid antigen tests in asymptomatic and presymptomatic close contacts of individuals with confirmed SARS-CoV-2 infection: cross sectional study. BMJ. 2021;374:n1676.

    Article  Google Scholar 

  4. Brummer LE, Katzenschlager S, Gaeddert M, Erdmann C, Schmitz S, Bota M, Grilli M, Larmann J, Weigand MA, Pollock NR, et al. Accuracy of novel antigen rapid diagnostics for SARS-CoV-2: A living systematic review and meta-analysis. PLoS Med. 2021;18(8):e1003735.

    Article  CAS  Google Scholar 

  5. Scheiblauer H, Filomena A, Nitsche A, Puyskens A, Corman VM, Drosten C, Zwirglmaier K, Lange C, Emmerich P, Muller M, et al. Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021. Euro Surveill. 2021;26(44):2100441.

    Article  CAS  Google Scholar 

  6. Venekamp RP, Veldhuijzen IK, Moons KGM, van den Bijllaardt W, Pas SD, Lodder EB, et al. Detection of SARS-CoV-2 infection in the general population by three prevailing rapid antigen tests: cross-sectional diagnostic accuracy study. BMC Med. 2022;20(1):97.

    Article  CAS  Google Scholar 

  7. European Centre for Disease Prevention and Control (ECDC): Considerations on the use of self-tests for COVID-19 in the EU/EEA. In: ECDC Technical Report. 2021. https://www.ecdc.europa.eu/sites/default/files/documents/Considerations-use-of-self-tests-for-COVID-19-in-the-EU-EEA-17-March2021-erratum.pdf.

  8. Stohr J, Zwart VF, Goderski G, Meijer A, Nagel-Imming CRS, Kluytmans-van den Bergh MFQ, et al. Self-testing for the detection of SARS-CoV-2 infection with rapid antigen tests for people with suspected COVID-19 in the community. Clin Microbiol Infect. 2022;28(5):695–700.

    Article  CAS  Google Scholar 

  9. Lindner AK, Nikolai O, Rohardt C, Kausch F, Wintel M, Gertler M, Burock S, Horig M, Bernhard J, Tobian F, et al. Diagnostic accuracy and feasibility of patient self-testing with a SARS-CoV-2 antigen-detecting rapid test. J Clin Virol. 2021;141:104874.

    Article  CAS  Google Scholar 

  10. Klein JAF, Kruger LJ, Tobian F, Gaeddert M, Lainati F, Schnitzler P, Lindner AK, Nikolai O, Knorr B, Welker A, et al. Head-to-head performance comparison of self-collected nasal versus professional-collected nasopharyngeal swab for a WHO-listed SARS-CoV-2 antigen-detecting rapid diagnostic test. Med Microbiol Immunol. 2021;210(4):181–6.

    Article  CAS  Google Scholar 

  11. Lindner AK, Nikolai O, Kausch F, Wintel M, Hommes F, Gertler M, Kruger LJ, Gaeddert M, Tobian F, Lainati F, et al. Head-to-head comparison of SARS-CoV-2 antigen-detecting rapid test with self-collected nasal swab versus professional-collected nasopharyngeal swab. Eur Respir J. 2021;57(4):2003961.

    Article  CAS  Google Scholar 

  12. Homza M, Zelena H, Janosek J, Tomaskova H, Jezo E, Kloudova A, Mrazek J, Murinova V, Madar R. Performance of seven SARS-CoV-2 self-tests based on saliva, anterior nasal and nasopharyngeal swabs corrected for infectiousness in real-life conditions: a cross-sectional test accuracy study. Diagnostics (Basel). 2021;11(9):1567.

    Article  CAS  Google Scholar 

  13. Igloi Z, Velzing J, Huisman R, Geurtsvankessel C, Comvalius A, IJpelaar J, et al. Clinical evaluation of the SD Biosensor SARS-CoV-2 saliva antigen rapid test with symptomatic and asymptomatic, non-hospitalized patients. PLoS One. 2021;16(12):e0260894.

    Article  CAS  Google Scholar 

  14. Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, Lijmer JG, Moher D, Rennie D, de Vet HC, et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527.

    Article  Google Scholar 

  15. Combined indicator: 14-day notification rate, testing rate and test positivity, updated 16 September 2021, weeks 35–36 [https://www.ecdc.europa.eu/en/publications-data/combined-indicator-week-36-2021]

  16. Variants of the corona virus [https://coronadashboard.rijksoverheid.nl/landelijk/varianten]

  17. Variants of the corona virus SARS-CoV-2 [Dutch] [https://www.rivm.nl/coronavirus-covid-19/virus/varianten]

  18. Epidemiologische situatie van SARS-CoV-2 in Nederland [Dutch] [https://www.rivm.nl/sites/default/files/2021-09/COVID-19_WebSite_rapport_wekelijks_20210928_1146_final.pdf]

  19. Kluytmans-van den Bergh MFQ, Buiting AGM, Pas SD, Bentvelsen RG, van den Bijllaardt W, van Oudheusden AJG, van Rijen MML, Verweij JJ, Koopmans MPG, Kluytmans J. Prevalence and clinical presentation of health care workers with symptoms of coronavirus disease 2019 in 2 Dutch hospitals during an early phase of the pandemic. JAMA Netw Open. 2020;3(5):e209673.

    Article  Google Scholar 

  20. Collet D. Modelling binary data. Boca Raton: Chapman & Hall/CRC; 1999.

    Google Scholar 

  21. WHO. Antigen-detection in the diagnosis of SARS-CoV-2 infection using rapid immunoassays. 2020. p9. [https://apps.who.int/iris/bitstream/handle/10665/334253/WHO-2019-nCoV-Antigen_Detection-2020.1-eng.pdf]

  22. Tsang NNY, So HC, Ng KY, Cowling BJ, Leung GM, Ip DKM. Diagnostic performance of different sampling approaches for SARS-CoV-2 RT-PCR testing: a systematic review and meta-analysis. Lancet Infect Dis. 2021;21(9):1233–45.

    Article  CAS  Google Scholar 

  23. Uddin MKM, Shirin T, Hossain ME, Alam AN, Ami JQ, Hasan R, Miah M, Shaly NJ, Ahmed S, Rahman SMM, et al. Diagnostic performance of self-collected saliva versus nasopharyngeal swab for the molecular detection of SARS-CoV-2 in the clinical setting. Microbiol Spectr. 2021;9:e0046821.

    Article  Google Scholar 

  24. Bastos ML, Perlman-Arrow S, Menzies D, Campbell JR. The sensitivity and costs of testing for SARS-CoV-2 infection with saliva versus nasopharyngeal swabs : a systematic review and meta-analysis. Ann Intern Med. 2021;174(4):501–10.

    Article  Google Scholar 

  25. Allen N, Brady M, Carrion Martin AI, Domegan L, Walsh C, Doherty L, Riain UN, Bergin C, Fleming C, Conlon N. Serological markers of SARS-CoV-2 infection; anti-nucleocapsid antibody positivity may not be the ideal marker of natural infection in vaccinated individuals. J Infect. 2021;83(4):e9–10.

    Article  CAS  Google Scholar 

  26. Eberly AR, Germer JJ, Boerger AC, Fernholz EC, Binnicker MJ, Yao JD. Performance of three polymerase chain reaction-based assays for detection of SARS-CoV-2 in different upper respiratory tract specimens. Diagn Microbiol Infect Dis. 2021;101(2):115441.

    Article  CAS  Google Scholar 

  27. EQA of laboratories performing SARS-CoV-2 diagnostics for the Dutch, May 2021 [Dutch] [https://www.rivm.nl/sites/default/files/2021-08/EQA%20of%20Laboratories%20Performing%20SARS-CoV-2%20Diagnostics%20for%20the%20Dutch%20Population%20round%203_May%202021_0.pdf]

  28. Igloi Z, Velzing J, van Beek J, van de Vijver D, Aron G, Ensing R, Benschop K, Han W, Boelsums T, Koopmans M, et al. Clinical evaluation of Roche SD Biosensor Rapid antigen test for SARS-CoV-2 in municipal health service testing site, the Netherlands. Emerg Infect Dis. 2021;27(5):1323–9.

    Article  CAS  Google Scholar 

  29. van Kampen JJA, van de Vijver D, Fraaij PLA, Haagmans BL, Lamers MM, Okba N, van den Akker JPC, Endeman H, Gommers D, Cornelissen JJ, et al. Duration and key determinants of infectious virus shedding in hospitalized patients with coronavirus disease-2019 (COVID-19). Nat Commun. 2021;12(1):267.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We thank the participants and study staff at the participating public health service test sites, participating laboratories, the University Medical Center Utrecht, and RIVM for their contributions to the study. A special thanks to Esther Stiefelhagen, Renske Beekes, Sophie Neeleman, Roel Ensing, Wendy Mouthaan, Lieke Brouwer, and Timo Boelsums. Written permission was obtained from all seven of them to list their names. ES, RB, SN, RE, WH, LB, and TB did not receive any compensation for their contributions.

Funding

This study was funded by the Dutch Ministry of Health, Welfare, and Sport. The funder had no role in the study design; collection, analysis, and interpretation of data; writing of the report and decision to submit the paper for publication.

Author information

Authors and Affiliations

Authors

Contributions

KGMM initiated the study. ES, RPV, IKV, WvdB, SDP, EL, MH, RM, ZI, CW, IV, CRSN-I, WGHH, JAJWK, SvdH, JHHMvdW, and KGMM designed the study. ES, RPV, IKV, CRSN-I, and KGMM coordinated the study. WvdB, SDP, JS, RM, and ZI were responsible for laboratory analyses and data processing. ES, RPV, and IKV verified the underlying data. ES performed the statistical analysis and verified the underlying data in close collaboration with RPV and KGMM. ES, RPV, JHHMvdW, and KGMM drafted the first version of the manuscript. All authors critically read the manuscript and provided feedback. All authors read and approved the final manuscript, and approved its submission. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Corresponding author

Correspondence to Karel G. M. Moons.

Ethics declarations

Ethics approval and consent to participate

Not required because the study was judged by the METC Utrecht to be outside the scope of the Dutch Medical Research Involving Human Subjects Act (protocol No 21–470 /C). All participants signed an informed consent form before any study procedure.

Consent for publication

Not applicable.

Competing interests

All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Dutch Ministry of Health, Welfare, and Sport for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Material 1.

Short questionnaire after performing both self-tests at home. Material 2. Short questionnaire provided 10 days after test site visit. Material 3. Specimen collection, SARS-CoV-2 diagnostic testing, and SARS-CoV-2 virus culture procedures. Table S1. 2x2 tables for primary and secondary analyses of the molecular reference test – saliva Ag-RDT comparison. Table S2. 2x2 tables for primary and secondary analyses of the molecular reference test – nasal Ag-RDT comparison. Table S3. Follow-up information of participants that were tested again within 10 days after the initial test. Fig. S1. Sensitivities with 95% confidence intervals of the Ag-RDT-molecular reference standard test comparisons at viral load cut-off for infectiousness. Fig. S2. Distribution of viral loads in participants with a positive molecular reference test, stratified by symptomatology at time of sampling. Fig. S3. Distribution of change scores in difficulty of taking a sample and difficulty of reading the test result when comparing difficulty ratings of the saliva Ag-RDT and the nasal Ag-RDT.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schuit, E., Venekamp, R.P., Veldhuijzen, I.K. et al. Head-to-head comparison of the accuracy of saliva and nasal rapid antigen SARS-CoV-2 self-testing: cross-sectional study. BMC Med 20, 406 (2022). https://doi.org/10.1186/s12916-022-02603-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12916-022-02603-x

Keywords

  • SARS-CoV-2
  • Rapid antigen detection test
  • Antigen test
  • COVID-19
  • Saliva test
  • Nasal test
  • Diagnostic test accuracy
  • Cross-sectional Study