This largest diagnostic accuracy evaluation of two unsupervised self-performed Ag-RDTs to date showed a low overall sensitivity (46.7%) of the Hangzhou AllTest Biotech saliva self-test. Applying a viral load cut-off as a proxy for infectiousness did not improve the overall sensitivity meaningfully (54.9%) nor in any of the studied subgroups, with all sensitivities remaining far below the WHO standard of 80% [21].
The study showed better performance of the SD Biosensor nasal self-test with an overall sensitivity of 68.9%, increasing to 83.9% when the viral load cut-off was applied. The sensitivities were much higher in the 2319 symptomatic participants than in the 487 asymptomatic participants (78.5% and 22.6%, respectively) and in the 2417 individuals who never had COVID-19 in the past compared to the 372 individuals who did (72.7% and 36.8%, respectively), reaching sensitivities (with sufficient precision) above the WHO-recommended 80% in symptomatic individuals and in individuals without a previous infection after applying the viral load cut-off. The sensitivities in asymptomatic individuals and in individuals who had had COVID-19 in the past have wider 95% confidence intervals and should therefore be interpreted with caution. We recommend additional research in those groups. The diagnostic accuracy of this nasal self-test did not differ by COVID-19 vaccination status, sex, and age.
Discussion of the saliva self-test results
We identified two previous studies in the scientific literature; the sensitivities observed in our study were in between those found in the two studies. A Czech study evaluated four saliva Ag-RDTs and found sensitivities of 15% for a saliva test that required spitting in a cup, and 3.6%, 25.5%, and 32.8% for saliva tests requiring sucking on a sponge, in comparison with RT-PCR [12]. These sensitivities improved slightly after using a “cell culture viability” cut-off but remained well below 50%. The sampling was done by the participants themselves but supervised by trained personnel. Samples sizes were modest, ranging from 98 to 407 participants per evaluated test. A recent Dutch study of the SD Biosensor saliva test with 789 participants found a sensitivity of 66.1%, increasing to 88.6% when Ct < 30 and to 96.7% when viral culturability was used as cut-offs [13]. In the Dutch study, saliva was collected by letting nasal and cough discharge drool into a collection device and was supervised by trained test site staff. Sensitivity was lower (60%) in asymptomatic participants but only 10 asymptomatic participants tested RT-PCR positive. We tested the analytical performance of the Hangzhou and SD Biosensor lateral flow test devices on calibrated samples and found that both test devices performed (equally) well (Supplement 3). We therefore hypothesize that the widely ranging sensitivity results for saliva Ag-RDTs may be due to high variability in saliva sampling methods (spitting vs. sucking vs. drooling) and/or high variability in the quantity and quality of sample self-obtained by different individuals. Furthermore, saliva specimens may on average contain lower SARS-CoV-2 viral loads than upper respiratory tract samples. Studies have shown that saliva viral loads are usually sufficiently high for detection by molecular methods [22,23,24], but they may not be sufficiently high for detection by self-performed Ag-RDTs.
We saw trends of reduced diagnostic accuracy in persons without symptoms or with previous SARS-CoV-2 infection. These trends were like the trends that we observed for the nasal self-test and are discussed below. We also saw trends by gender and age, with sensitivities for men and for persons aged over 65 reaching around 60%, which is still well below the WHO-recommended 80% [21]. The saliva Ag-RDT evaluation studies to date did not stratify by gender and age [12, 13], and the nasal Ag-RDT studies, including the nasal self-test that we evaluated in this study, did not show these trends [3, 6]. We recommend that future saliva Ag-RDT evaluations stratify by gender and age to investigate this further.
Discussion of the nasal self-test results
The diagnostic performance of Ag-RDTs combined with nasopharyngeal sampling done by trained personnel or by individuals themselves have been evaluated extensively by us and others [2,3,4,5,6, 8, 9, 12]. The above-mentioned Czech study also evaluated an Ag-RDT in combination with anterior nasal sampling done by trained personnel [12]. These studies found good performance (70–80%) with nasopharyngeal sampling but lower performance with anterior nasal sampling (45–55%) and also lower performance in asymptomatic individuals (50–60%), regardless of self or professional sampling. Our study showed that nasal self-sampling with the SD Biosensor Ag-RDT provided good sensitivity, which equaled the sensitivity of Ag-RDTs found in other studies in which the nasal sampling was done by a trained professional [10, 11], but only for individuals who have symptoms at the time of testing. We found a very low sensitivity of this self-performed nasal Ag-RDT of only 23% in asymptomatic individuals, which is much lower than the sensitivities found in our previous studies using nasopharyngeal or oropharyngeal combined with nasal sampling done by trained personnel [3, 6]. This difference in performance persisted after applying a viral load cut-off. It is currently unclear why the sensitivities of the nasal Ag-RDT self-test differed depending on the presence of symptoms, even after applying the viral load cut-off. We tested the analytical performance of the SD Biosensor lateral flow test device on calibrated samples and found that the test device itself performed well (Supplement 3). We hypothesize that the difference in sensitivity may be explained by the difference in viral load distributions in asymptomatic and symptomatic individuals. In addition, it may be more difficult for asymptomatic individuals (i.e., with a dry nose) to retrieve sufficient nasal fluid by self-swabbing. The former hypothesis is supported by the fact that the sensitivity of the saliva self-test was also lower in asymptomatic than symptomatic individuals, but both hypotheses might play a role.
We also found a low sensitivity (36.8%) of the nasal self-test in individuals who had had COVID-19 in the past. These results should be interpreted with caution due to the small group sizes: only 19 participants with a positive molecular test reported having had COVID-19 (16 of whom were symptomatic at the time of testing), and only seven of them had a viral load above the viral load cut-off (six of whom were symptomatic). However, similar trends were observed for the saliva self-test in this study and for the SD Biosensor Ag-RDT conducted by trained staff in a previous study [6]. In that study, sensitivities were 54.5% for oropharyngeal-nasal sampling and 68.4% for nasopharyngeal sampling in individuals with prior infection, and 75.8% and 75.0%, respectively, in individuals without prior infection. The low sensitivity in individuals with a prior infection may be explained by lower viral loads in this group (in the current study, 12/19 participants with a prior infection were below the viral load cut-off compared to 113/161 in those without a prior infection), with some of them potentially carrying viral RNA in the absence of a productive infection (i.e., no viral antigen production). Another explanation might be that individuals who have had COVID-19 have circulating anti-nuclear capsid (N) protein antibodies [25]. These anti-N antibodies might bind to the N protein that is produced during the new infection, hampering the binding of monoclonal antibodies against the N-protein in the test device. It should be noted that we found a smaller reduced sensitivity of the BD Veritor Ag-RDT conducted by trained staff (oropharyngeal-nasal sampling) in individuals with and without a prior infection (64.6% versus 70.1%), so this effect may be test device-specific [6].
Strengths and limitations of this study
Strengths of this study include the large overall sample size covering multiple test sites nationwide, the collection of samples for the reference test and two Ag-RDTs in the same individuals within a few hours allowing for a head-to-head comparison of the two self-tests, the fact that sampling was done by the participants themselves without any supervision conform the real-world context of self-testing, that the index test was blinded for the reference test result and vice versa, and the use of a proxy for infectiousness. Furthermore, the follow-up information showed that very few infections were missed by the molecular reference tests.
Our study also has some limitations. First, the reference standards that we used were molecular tests, but platforms and test kits used differed among the centralized laboratories. However, the diagnostic accuracies of all molecular tests used are similarly high [26, 27], and we therefore believe that this has not influenced our findings significantly. In addition, Ct values used to calculate viral loads were determined by different yet comparable platforms (Additional file: material 3). Second, we used the viral load cut-off above which 95% of people with a positive RT-PCR test result had a positive virus culture as a proxy of infectiousness. Although this cut-off is not fully evidence based [3], it is a best estimate based on current knowledge and less arbitrary than using Ct cut-offs of 25 or 30 as is often done [28, 29]. In the current study, we relied on infectiousness viral load cut-offs that were determined in our previous study in a mainly unvaccinated population (the proportion of vaccinated individuals in the present study reached 85% at the end of the study) and when a different SARS-CoV-2 variant was dominant [3]. Whether this would have impacted the applied viral load cut-offs is unknown, but vaccination itself did not influence any of the test sensitivities. Third, our sample size calculation was based on the primary analysis and the diagnostic accuracy parameters are less precise for the secondary stratified analyses. Fourth, participants were not blinded to the results of the saliva Ag-RDT when interpreting the result of the nasal Ag-RDT, which could have potentially biased the test outcome assessment of the nasal Ag-RDT. We do, however, believe that the impact of this limitation is small considering that the interpretation of the test results was considered (very) easy by > 95% of participants, and the performance of each Ag-RDT was substantially different and higher for the last performed nasal Ag-RDT. If outcome assessment was biased, the diagnostic performance of the self-tests would likely have been more similar. Fifth, we had some, though very limited, missing index test data (5%). We did not perform multiple imputation techniques because the group with missing data was very similar to the group with complete data, suggesting that data was missing completely at random.
Policy implications
Ag-RDTs for self-use are widely available in the Netherlands. Until recently, the recommendation was to use them when asymptomatic prior to having contacts (such as going to school, events, or work) and visit a public health test site for molecular testing when symptomatic. Individuals whose self-test was positive are (still) asked to visit a public health test site for confirmatory testing. The SD Biosensor nasal self-test that we evaluated in this study is one of the self-tests that is commercially available in the Netherlands, although we do not know its market share. Our results indicate that the SD Biosensor nasal self-test sensitivity among individuals with mild symptoms is similar to Ag-RDT sensitivities found in studies where it was applied to professionally obtain upper respiratory tract samples. Based on those results, the Dutch Outbreak Management Team (OMT) that advises the Ministry of Health, Welfare, and Sports regarding COVID-19 policy recommended expanding nasal self-testing to individuals with mild symptoms. The OMT stressed that self-tests are not advised in vulnerable persons, in individuals meeting vulnerable persons, and in case of more severe symptoms, and that a negative self-test result is not sufficient for ending quarantine for contacts of a confirmed case. However, individuals testing negative by nasal self-testing would be allowed to go to work (if not working with vulnerable persons) or school despite their mild symptoms, preferably using mouth-nose masks and testing again a day later in case the initial test result was negative. Individuals testing positive by nasal self-testing would have to self-isolate and visit a public health test site for confirmatory molecular testing, to keep track of virus spreading and to allow for contact-tracing. All these nuances require careful communication, including on the implications of false-negative test results.
The very low sensitivity of the SD Biosensor nasal self-test in asymptomatic individuals is worrisome, even though the a-priori probability of being infected is lower in asymptomatic than symptomatic individuals. In addition, the potentially reduced sensitivity of the SD Biosensor nasal self-test in individuals who have had COVID-19 in the past is also worrisome. This is especially important because an increasing proportion of the population will have had COVID-19. We call for additional research in these two specific subgroups. We also recommend that persons who tested negative by a self-test continue to adhere to the general preventive measures such as physical distancing, wearing mouth-nose masks, and washing hands.
The SD Biosensor nasal self-test is only one of the commercially available self-tests. We recommend that all available self-tests are evaluated urgently by independent researchers, also addressing the relevant subgroups. Finally, in high-risk situations, such as testing of vulnerable people in care facilities, severely ill patients, or healthcare workers, we recommend molecular testing at all times, which is already in line with current policy.