Evaluation of an Automated Digital Scoring System of Dental Plaque

Good oral health is associated with maintaining optimal general health.¹ The normal oral flora of healthy individuals includes gram-positive organisms and dental pathogens; good oral hygiene, including brushing and flossing, helps to keep bacteria within the oral cavity under control. In contrast, lack of proper oral hygiene practices leads to an increase in microbial flora within the oral cavity and subsequent dental and periodontal diseases. Abundance of these microbial flora leads to accumulation of dental plaque, a biofilm that provides a microhabitat for organisms with opportunity for adherence either to the tooth surface or to other microorganisms. Organisms in dental plaque ferment carbohydrates within the oral cavity leading to dental caries, contribute to inflammation of the gingiva and underlying tissues, and have a potential for pathogenicity.^2-4 Involvement of underlying tissues within the oral cavity further deteriorate oral health and allow the microbes to enter the blood stream.⁵ This can contribute to debilitating general health including endocarditis,^6,7 and has also been associated with a variety of systemic diseases including but not limited to atherosclerosis and vascular disease,^8-10 poor glycemic control in diabetes,¹¹ preterm birth,¹² and dementia.¹³

Critically ill, hospitalized patients are susceptible to hospital-acquired infections which may be related to poor oral health.^14,15 Oral health in the intensive care unit (ICU) may be compromised by ICU equipment, medical conditions or treatments, and the patient's inability to attend to their own self-care.¹⁶ Within 48 hours of hospital admission, the composition of oral flora in the critically ill adult can change to predominantly gram-negative and virulent gram-positive organisms, including the potential ventilator associated pneumonia (VAP) pathogens Staphylococcus aureus, Streptococcus pneumoniae, Acinetobacter baumanii, Haemophilus influenzae and Pseudomonas aeruginosa.^16-18 Moreover, the risk of developing pneumonia is 6-20 fold higher in ventilated as compared to non-ventilated ICU patients,^16,19 with the highest risk occurring early in the course of hospitalization.^20,21

Mechanically ventilated hospitalized patients are dependent upon ICU nurses to perform their oral hygiene due to their critical illness.¹⁴ The placement of endotracheal and other tubes through the oral cavity, puts these patients at high risk for aspiration and subsequent VAP. Oral care administered by ICU nurses in ventilated patients includes effective elimination of evolving gram-negative organisms and prevention of overall bacterial growth in the mouth.^22-25 ICU nurses also provide a variety of oral interventions designed to address patients' comfort, rather than solely focusing on oral hygiene and dental plaque removal.^16,26-28 Oral care intervention research in the ICU has been challenged due to the complicated nature of measuring dental plaque in critically ill, mechanically ventilated subjects, which has prompted this particular investigation of a valid and reliable asynchronous automated plaque scoring system.

Measurement of dental plaque is frequently used as an indicator of overall oral health. Quantification of dental plaque is critically important in clinical practice, as well as in research studies. Commonly used visual scoring systems of dental plaque include: University of Mississippi Oral Hygiene Index (UM-OHI),²⁹ Oral Hygiene Index (OHI),³⁰ Simplified Oral Hygiene Index (OHI-S),³¹ Turesky Plaque Index,³² and Silness and Loe Index.³³ While all of these instruments are relatively straightforward to use in terms of assessing oral health, limitations on evaluating plaque levels exist, including variable number of teeth that are assessed, the relatively crude coding of binary values of 0 (No) and 1 (Yes) for tooth segments, the presence versus absence of plaque, the subjective nature of the process, the lack of reliability within and between assessors, personnel burden, and differing plaque level scoring properties.

The purpose of this study was to compare the psychometric properties of an automated versus a more traditional manual plaque scoring method (UM-OHI) to gain insight into whether an automated method may be warranted for both research and clinical practice purposes.

Methods

Data used in this cross-sectional analysis are from a subgroup of critically ill, mechanically ventilated patients (n=79) enrolled in a prospective, randomized controlled trial designed to determine the optimal frequency (once, twice, or three times daily) of ICU nurse delivered tooth brushing. Informed consent was obtained from the subject's legally authorized representative, in accordance with the University of South Florida IRB approval process (Pro 00016479). The first research aim of the parent clinical trial, was to evaluate the clinical equivalence (non-inferiority) of three tooth brushing frequencies on oral health (dental plaque and mucosal inflammation). All tooth brushing interventions were delivered by study personnel using a standardized tooth brushing protocol, performed with a compact head adult tooth brush and Biotene^® (GSK; Philadelphia, PA) fluoride toothpaste.³⁴ Subjects received scheduled toothbrushing interventions for the first 7 days of intubation, or until extubation within a 7-day period. All subjects received standard clinical care for mechanically ventilated patients, as per clinical and agency guidelines.³⁵ The trial was registered at Clinical Trials.gov (NCT02289131); a detailed description of the parent study protocol has been previously published.³⁶

Dental plaque assessment

Digital images of all teeth (buccal and lingual) were obtained with the use of an intraoral camera (Soprocare, Acteon Inc.; Mount Laurel, NJ). For this study, dental plaque levels were assessed from the digital images in two ways. The first method utilized the well-established UMOHI instrument (as specified in the parent study protocol) and was completed by the same dental hygienist. The dental hygienist evaluator was experienced in evaluating oral health of critically ill mechanically ventilated patients. Concurrently, a software code was used to obtain and score images of every tooth. Both methods used video and photographs taken of the subject's entire oral cavity while in the ICU. Assessment was blinded by random group assignment. No disclosing dyes were used in either scoring system; the intraoral camera included fluorescence and chromatic amplification to highlight dental plaque.

Manual plaque assessment scoring procedure

The UM-OHI had been selected as the manual, visual plaque scoring method for the parent study because every available tooth in the mouth is scored as compared to certain representative teeth as in other plaque scoring systems. The dental hygienist evaluator reviewed digital photos of the oral cavity, divided into 12 regions: left and right posterior teeth and anterior teeth in each arch, further subdivided into buccal and lingual surfaces.²⁹ Each individual digital photo was divided into five sections for the buccal and lingual surfaces and included the mesial, distal, and middle sections which were further subdivided horizontally into gingival, middle, and occlusal sections. Each section, a total of 10 per tooth, was scored for presence or absence of plaque. If a section was determined to have plaque present, it was scored as 1; if no plaque was present, the section received a value of 0. Each tooth was scored from 0 (no plaque in any section) to 10 (plaque in every section). The mean plaque score for the subject was calculated by dividing the total score by number of teeth.

Automated plaque assessment scoring procedure

A patented, algorithm-based, automated scoring system was used to quantify extent of dental plaque. The automated system used video and photographic images, selected based on overall clarity, from digital images taken with an intraoral camera of all teeth (buccal and lingual surfaces).³⁷ Specifically, all images used in the assessment had a 640*480 resolution, and were cropped to leave only the tooth in the picture. The final cropped image had at least 10,000 pixels in resolution and was imported into a computer software program which scored the value of a specific pixel to produce a three-dimension point (x, y, z) that uniquely defined the color of the pixel. The software used two digits for each color dimension and each digit used a hexadecimal system to count the numbers. There were 256 possible values to score each color dimension. Since plaque typically presents as yellow in color, the automated scoring system was developed and used to judge whether each pixel should be classified as yellow (plaque) or not (no plaque).

To identify the right combination of the three colors leading to the determined yellow color, the color dimension was divided into four categories: (0,64), (64,128), (128,192), (192,255). These four categories were chosen with the rationale of being an acceptable balance between accuracy and computational difficulty and there were 4*4*4 = 64 categories in total. Next, the middle point of each range was chosen, namely 32, 96, 160, 224, and used the color of that specific combination to represent the color for that category. For example, for the category (0,64) in red, (0,64) in blue and (0,64) in green, the color point, 32 in red, 32 in blue, and 32 in green was used to represent the color for that category. After scoring all 64 categories, several common properties shared by categories were identified as yellow. These properties included: value of the red dimension must be between 0.75 to 2.5 times the value of the green dimension; the value of both the green and red dimensions must be at least 1.2 times of the value of the blue dimension; and the value of the red dimension must be at least 60. Therefore, if values of the pixel met these requirements, they were classified as yellow (plaque), otherwise they were classified as non-yellow (normal). From the binary results of each pixel, the percentage of yellow (dental plaque) was calculated by the number of yellow pixels divided by the total number of pixels in the picture. Selected examples of this coding system are depicted in Table I.

Statistical analysis

Demographic and clinical characteristics of the patient sample were described using means and standard deviation (SD) for continuous variables and percentages for categorical variables. The distribution of dental plaque scores along with their interquartile range and the 5th and 95th percentiles from both scoring methods were plotted side-by-side and mean differences were compared using paired t tests. Analyses were also stratified by period of assessment (during intervention vs. post-extubation), number of pictures used for automated scoring (<15 vs. >15), and number of pixels used for automated scoring (<150,000 vs. >150,000). Pearson correlation coefficients were calculated between manual and automated dental plaque scores, including stratified analyses. Furthermore, the intra-class coefficient (ICC) was computed between both methods using their respective transformed z-score values. Finally, a Bland-Altman plot was constructed to examine the manner (direction and magnitude) in which dental plaque scores differed between the manual and automated scoring methods. A 2-sided p-value of <0.05 was used to define statistical significance for all analyses and 95% confidence intervals for the mean difference were reported. The sample size (n=79) was based on the goal of having a minimum of 50 subjects with non-missing data for reliable estimation of correlation coefficients and confidence intervals. Methods and results are presented using the strengthening the reporting of observational studies in epidemiology (STROBE) guidelines for reporting observational studies.

Results

The patient sample consisted of critically ill, intubated ICU patients (n=79), with a mean age of 57.3 (SD=16.5), Sample population demographics are shown in Table II. The most prevalent admission diagnosis was respiratory failure (55.7%), and sepsis was present in 24.1% of the sample. The mean number of decayed/missing/filled teeth (DMFT score) was 13.7 (SD= 9.4) and was distributed as follows: decayed: (2.2 + 3.0); missing (7.4 + 7.6); filled (4.0 + 4.5). Patient mortality during the ICU stay was 38%, and the mean Acute Physiology and Chronic Health Evaluation (APACHE-IV) score³⁸ was 70.6 + 20.3.

Calculation of automated digital plaque scores

The number of pictures used in the automated digital plaque scoring (based on clarity and number of teeth assessed) was highly variable and ranged from 2 to 58, with a mean of 13.5 (SD= 8.9) and a median of 13 (IQR=10). The total number of pixels used in the automated digital plaque scoring ranged from 10,272 to 586,433 with mean of 139,778 (SD=93,229) and median value of 130,130 (IQR= 105,801). The Spearman rank correlation between the number of pictures and pixels evaluated was 0.96, indicating a near perfect correlation, with a mean of 10,391 + 2,159 pixels evaluated per tooth.

Comparison of plaque scoring methods

Plaque assessment distribution by scoring method (manual vs. automated) and stratified by subgroups are shown in Figure 1. As depicted, dental plaque scores calculated by the manual method were remarkably higher (mean = 67.3 + 18.7) than those calculated by the automated scoring method (mean = 23.7 + 15.2). The mean difference between manual and automated dental plaque scores was 43.6 (95% confidence interval: 40.2 - 46.9, p<0.0001). In stratified analyses, the disparity between manual and automated dental plaque scores was most evident for the subjects (n=32) whose automated assessment involved >15 pictures (mean difference = 47.8 95% confidence interval: 44.1 - 51.6, p<0.0001), and the subjects (n=32) whose automated assessment involved 150,000 or more pixels (mean difference = 48.3, 95% confidence interval: 44.6-52.1, p<0.0001). Nonetheless, in all subgroups examined, mean manual dental plaque scores were significantly higher (p<0.0001) than mean automated dental plaque scores.

Despite the automated dental plaque scores being systematically substantially lower than manual dental plaque scores, the Pearson correlation coefficient between the two methods was relatively strong (r = 0.62, p < 0.0001); shown in the upper plot in Figure 2. In stratified analyses, the highest correlations between the manual and automated dental plaque scoring methods were observed for subjects (n= 34) with measurements obtained postintubation (r = 0.75, p < 0.0001), and those (n = 32) whose automated assessment involved >15 pictures (r = 0.68, p < 0.0001); illustrated in the middle and lower plots of Figure 2. Similarly, the automated scoring method showed good reliability with the manual method, as shown by an ICC value of 0.63.

In the Bland-Altman plot (Figure 3), with both manual and automated dental plaque scores averaged on the x-axis, the substantial difference in scores between the two methods (y-axis) was consistently present (i.e. irrespective of magnitude of plaque burden). However, there was an indication that as plaque burden increased, the disparity between the 2 scoring methods increased. Only a very small percentage of subjects ( n=2, 2.5%) had a lower manual dental plaque score compared to the corresponding automated dental plaque score.

Discussion

In this study, distributions of estimated total plaque scores varied dramatically between using a manual method of plaque assessment (UM-OHI, mean score of 67.3 + 18.7) versus an automated system (patented algorithm-based system, mean score 23.7 + 15.2). Profound differences (mean difference of 43.6) in the distributions of plaque scores were observed across patient subgroup analyses; only a small number of subjects (n=2, 2.5%) had lower manual dental plaque scores compared to their corresponding automated dental plaque score. Despite the dramatic differences in scores, the Pearson correlation coefficient (0.62) and intra-class correlation coefficient (0.63), were quite strong between the two methods, indicating that both methods measured overall plaque burden, yet with substantially different numerical properties.

An obvious question that arises is "which method is more accurate?" Empirically, this question cannot be definitely answered with the data at hand. However, it can be postulated that the automated method is likely to be more accurate simply based on numerical properties of each method. Specifically, with the tooth "section-based" method used with the UM-OHI, even a small amount of plaque observed in a section results in a value of 1.0 for the entire section. Therefore, tooth sections with small versus large amounts of plaque receive the same value and cannot be differentiated with this particular scoring method. In contrast, the automated scoring method evaluated each pixel from each tooth for presence versus absence of plaque which conceptually, allows for fine gradation (on a continuous scale) between teeth with small, versus medium, versus large amounts of plaque. However, the automated method is only as accurate as the binary algorithmic-based determination of plaque (yellow color) versus absence of plaque. Correlation coefficients between the methods were highest under more favorable assessment conditions (post-extubation, with the availability of a larger number of clearer images) which again suggests the validity of both methods (i.e. both are measuring the same quantity of plaque burden).

The automated digital scoring system used in this study is not the only method previously proposed to objectively measure dental plaque. Bellamy et al. developed a digital plaque image analysis (DPIA) system designed to capture images of healthy subjects' teeth via an external digital computer-controlled camera, under white light without disclosing agents, as well as image processing and image analysis software that identifies color differences indicative of dental plaque.³⁹ The accuracy of the Bellamy et al. system versus the automated digital scoring system, utilizing algorithms from the red-green-blue color spectrum used in this study, cannot be directly compared. However, the DPIA systems used in dental research, designed for healthy subjects, are not appropriate for populations of critically ill, mechanically ventilated, patients. DPIA systems require a cooperative subject, sitting upright, and positioned in a cephalometric head restraint apparatus in order to obtain external images. As technology progresses, a major emphasis will be on systems that can capture images in a comprehensive, minimally invasive manner and across a range of clinical settings, and ideally with real time scoring feedback.

Considering both research and clinical practice relevance, there are strengths and limitations to both an established manual method of assessment (UM-OHI) and this algorithm-based automated scoring system. The UM-OHI method has been used for decades, and is well known and accepted by dental health professionals. However, the current analysis suggests that it likely overestimates the percentage of total plaque burden in a given individual, can be time-consuming to score, and has an inherent degree of subjectivity in interpretation of presence versus absence of plaque by the evaluator. Due to the ordinal and compressed (0-10 per tooth) nature of the scoring algorithm, it is insensitive to all but large changes in the amount of dental plaque present. The automated scoring system offers the advantages of being entirely objective and reproducible, and digital dental images result in archival raw data which can be retested and used to refine the scoring algorithms to enhance validity. New objective measures may add value as documented evidence to support diagnostic criteria for procedures to be approved for insurance coverage. However, the automated scoring system requires selection of appropriate images of acceptable visual quality, and images that are exclusive to the areas suitable for plaque assessment (e.g. teeth only and not gums). These conditions and the refinement of the color-coding algorithms to represent full assessment exclusively of plaque remain challenges for future use in both research and clinical settings. Results from this study are specific to mechanically ventilated patients and may not generalize directly to other clinical settings, including primary care practice.

Conclusion

Automated digital systems have been postulated to be more precise than conventional visual methods of assessing and scoring dental plaque. In this study, an automated digital scoring system resulted in much lower overall dental plaque scores as compared to those from the an established manual scoring system (UM-OHI). While the objective automated digital scoring system may be more precise than the manual or visual scoring of dental plaque, its use should be weighed against the added effort, cost, and expertise required to use the method. Further study is needed to determine whether an automated digital scoring system can be commercialized and is warranted for use outside of research settings.

Acknowledgments

The study was funded by the National Institute of Nursing Research (NINR) under award number: R01 NR007652. The authors would like to acknowledge study trial research assistants: Monika Endredi; Michael Harrison; Lydia Phan; Lauren Wright; Lanette Dumas; Allison Erlenbusch; Nicole Libell; Ana Gutierrez.

About the Authors

Cindy L. Munro, Ph.D., RN is Dean of the University of Miami School of Nursing and Health Studies, Miami, FL; Zhan Liang, Ph.D., RN is an assistant professor at the University of Miami School of Nursing and Health Studies, Miami, FL; Nnadozie Emechebe, MPH is a Ph.D. student in epidemiology at the University of South Florida, College of Public Health, Tampa, FL; Xusheng Chen, MS is a data manager and statistician at the University of Miami School of Nursing and Health Studies, Miami, FL; Paula L. Cairns, Ph.D is an assistant professor at the University of South Florida, College of Nursing, Tampa, FL; Priyashi Manani, MPH is a recent graduate in epidemiology at the University of South Florida, College of Public Health, Tampa, FL; Lucia Hamilton, BSN, RN is a research coordinator at the University of South Florida, College of Nursing, Tampa, FL; Gwendolyn Good, BSN, RN, RDH is a registered dental hygienist in Sarasota, FL; Kevin E. Kip, Ph.D is a distinguished health professor at the University of South Florida, College of Public Health, Tampa, FL.

Corresponding author: Kevin Kip, PhD; kkip@usf.edu

References

1. CDC. Oral health. [Internet]. Atlanta (GA): Centers for Disease Control; 2019 [accessed 2019 January 17]. Available from: https://www.cdc.gov/oralhealth/index.html.

2. Munro C, Michalek SM, Macrina FL. Cariogenicity of streptococcus mutans v403 glucosyltransferase and fructosyltransferase mutants constructed by allelic exchange. Infect Immun.1991; Jul;59(7):2316-23.

3. Poole DF, Newman HN. Dental plaque and oral health. Nature. Dec. 10;1971; 234(5328):329-31.

4. Stralfors A. Studies of the microbiology of caries; the buffer capacity of the dental plaques. J Dent Res. 1948; Oct;27(5):587-92.

5. Holt R, Roberts G, Scully C. ABC of oral health. Dental damage, sequelae, and prevention. BMJ. 2000; Jun. 24;320(7251):1717-19.

6. Cahill TJ, Prendergast BD. Infective endocarditis. Lancet. 2016 Feb. 27;397: 882-93.

7. Munro CL, Macrina FL. Sucrose-derived exopolysaccharides of streptococcus mutans v403 contribute to infectivity in endocarditis. Mol Microbiol. Apr 8;1993; 8(1):133-42.

8. Aarabi G, Thomalla G, Heydecke G, Seedorf U.Chronic oral infection: an emerging risk factor of cerebral small vessel disease. Oral Dis. 2018; Jun 7.

9. Carrizales-Sepúlveda EF, Ordaz-Farías A, Vera-Pineda R, Flores-Ramírez R. Periodontal disease, systemic inflammation and the risk of cardiovascular disease. Heart Lung Circ. 2018; Nov; 27(11):1327-34.

10. Pereira LC, Nascimento JCR, Rego JMC, et al. Apolipoprotein e, periodontal disease and the risk for atherosclerosis: a review. Arch Oral Biol. Feb; 2019; 98:204-12.

11. Simpson TC, Weldon JC, Worthington HV, et al. Treatment of periodontal disease for glycaemic control in people with diabetes mellitus. Cochrane Database Syst Rev. 2015; Nov. 6;(11):Cd004714.

12. Ren H, Du M. Role of maternal periodontitis in preterm birth. Front Immunol. 2017; Feb. 13;8:139.

13. Gil-Montoya JA, Sanchez-Lara I, Carnero-Pardo C, et al. Oral hygiene in the elderly with different degrees of cognitive impairment and dementia. J Am Geriatr Soc. 2017; Mar;65(3):642-7.

14. CDC. Healthcare-associated infections - ventilator-associated pneumonia (vap) [Internet]. Atlanta (GA): Centers for Disease Control; 2019. [accessed 2019 January 17]. Available from: "https://www.cdc.gov/hai/vap/vap.html.

15. Munro C. Oral care for acutely and critically ill patients. Crit Care Nurse. 2017; Jun;37(3):e19-e21.

16. Munro CL, Grap MJ. Oral health and care in the intensive care unit: state of the science. American journal of critical care. Am J Crit Care. 2004; Jan;13(1):25-33; discussion 34.

17. Abele-Horn M, Dauber A, Bauernfeind A, et al. Decrease in nosocomial pneumonia in ventilated patients by selective oropharyngeal decontamination (SOD). Intensive Care Med. 1997; Feb;23(2):187-95.

18. Johanson WG, Jr., Seidenfeld JJ, de los Santos R, et al. Prevention of nosocomial pneumonia using topical and parenteral antimicrobial agents. Am Rev Respir Dis. 1988; Feb;137(2):265-72.

19. Safdar N, Crnich CJ, Maki DG. The pathogenesis of ventilator-associated pneumonia: Its relevance to developing effective strategies for prevention. Respir Care. 2005;50(6):725-39.

20. American Thoracic Society. Guidelines for the management of adults with hospital-acquired, ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med. 2005;171:388-416.

21. Chastre J, Fagon JY. State of the art: ventilator-associated pneumonia. Am J Respir Crit Care Med. 2002;165(7):867-903.

22. de Melo Neto JP, Melo MS, dos Santos-Pereira SA, et al. Periodontal infections and community-acquired pneumonia: a case-control study. Eur J Clin Microbiol Infect Dis. 2013;32(1):27-32.

23. Türk G, Kocaçal Güler E, Eser I, Khorshid L. Oral care practices of intensive care nurses: a descriptive study. Int J Nurs Pract. 2012;18:347-53.

24. Jones DJ, Munro CL, Grap MJ. Natural history of dental plaque accumulation in mechanically ventilated adults: a descriptive correlational study. Intensive Crit Care Nurs. 2011;27(6):299-304.

25. Munro CL, Grap MJ, Elswick RK, Jr, et al. 3rd Oral health status and development of ventilator-associated pneumonia: a descriptive study. Am J Crit Care. 2006;15(5):453-60.

26. Feider LL, Mitchell P, Bridges E. Oral care practices for orally intubated critically ill adults. AmJ Crit Care. 2010; Mar;19(2):175-83.

27. Fitch JA, Munro CL, Glass CA, Pellegrini JM. Oral care in the adult intensive care unit. Am J Crit Care. 1999;Sep;8(5):314-8.

28. Grap MJ, Munro CL, Ashtiani B, Bryant S. Oral care interventions in critical care: Frequency and documentation. Am J Crit Care. 2003; Mar;12(2):113-8; discussion 119.

29. Silberman SL, Le Jeune RC, Serio FG, et al. A method for determining patient oral care skills: The university of mississippi oral hygiene index. J Periodontol. 1998;Oct;69(10):1176-80.

31. Greene JC. The oral hygiene index--development and uses. J Periodontol. 1967; Nov-Dec;38(6):Suppl:625-37.

31. Greene JC, Vermillion JR. The simplified oral hygiene index. J Am Dent Assoc. 1964; 68:7-13.

32. Turesky S, Gilmore ND, Glickman I. Reduced plaque formation by the chloromethyl analogue of victamine C. J Periodontol. 1970; Jan;41(1):41-3.

33. Loe H, Silness J. Periodontal disease in pregnancy II. Correlation between oral hygiene and periodontal condition. Acta Odontol Scand. 1964; 22:121-35.

34. Munro CL, Grap MJ, Jablonski R, Boyle A. Oral health measurement in nursing research: state of the science. Biol Res Nurs. 2006; Jul;8(1):35-42.

35. Munro CL, Grap MJ, Jones DJ, et al. Chlorhexidine, toothbrushing, and preventing ventilator-associated pneumonia in critically ill adults. Am J Crit Care. 2009;Sep;18(5):428-37; quiz 438.

36. Munro CL, Liang Z, Cairns P, et al. Optimal frequency of tooth brushing in mechanically ventilated adults: study protocol for a randomized controlled trial. Res Nurs Health. 2018; Dec;41(6):511-8.

37. Munro CL, Kip KE, Cairns PL, et al., Inventors. 2016; standardized oral health assessment and scoring using digital imaging. United States patent 15B142.

38. Zimmerman JE, Kramer AA, McNair DS, Malila M. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today's critically ill patients. Crit Care Med. 2006;May;34(5):1297-1310.

39. Bellamy PG, Jhaj R, Mussett AJ, et al. Comparison of a stabilized stannous fluoride/sodium hexametaphosphate dentifrice and a zinc citrate dentifrice on plaque formation measured by digital plaque imaging (DPIA) with white light illumination. J Clin Dent. 2008; 19(2):48-54.