Garces-Elias_et

INTRODUCTION

To conduct an epidemiological survey, examiners must undergo prior training to ensure consistency in estimating the magnitude of the condition they will assess. Additionally, measures should be taken to limit the influence of physical and psychological factors inherent to the examiner on their judgment during the application of the instrument. It is important to reduce measurement differences when multiple evaluators are involved in data collection for epidemiological studies. To achieve this, a well-structured process of training and calibration is essential. This ensures that all examiners can evaluate consistently, reducing variability among them. Calibration not only standardizes the work of evaluators when applying an epidemiological index, but also helps minimize errors. In the dental field, the World Health Organization (WHO) highlights the importance of using indices to gather information on oral health, including dental assessments (1-3).

Detection and assessment of dental caries rely on radiographic, electronic, and tactile-visual tools for accurate identification. However, from an epidemiological standpoint, the latter is particularly suitable when evaluating oral health conditions in the field. Therefore, instruments for assessing dental caries must meet specific requirements, including feasibility, reliability, precision, and prior validation (4). In this context, the DMFT index is an optimal tool for estimating dental caries experience over an individual's or population's lifetime. This is significant given the high global prevalence of this oral health condition, making it the most common disorder (5, 6). For the DMFT index, dental caries is described as a visible lesion in a fissure, pit, or smooth surface of the tooth, characterized by an evident cavity, weakened enamel, or a soft surface to the touch. Additionally, caries is considered in teeth with temporary fillings or sealants showing apparent deterioration. When the tooth's crown has been destroyed, leaving only the root, it is classified as crown-origin caries. To ensure an accurate diagnosis, the Community Periodontal Index (CPI) probe is recommended; in case of doubt, caries should not be recorded as present.

In recent years, studies have implemented calibration processes for both new and existing oral health assessment models (3, 7, 8). However, external factors may complicate the traditional development of these processes, underscoring the need for tools that facilitate their application, such as online approaches (9). Additionally, scientific evidence describing such management in dentistry is scarce. Thus, this study aims to develop a calibration process for diagnosing dental caries using the DMFT index among three groups of dentists.

MATERIALS AND METHODS

This cross-sectional study involved a population of 15 dentists with more than 5 years of professional experience who participated voluntarily: 5 general dentists working in the public sector of the Ministry of Health of Peru (MINSA), 5 students from the specialty of dental public health (SPE), and 5 students from the restorative and aesthetic dentistry (ORE) specialty, all referred by the postgraduate program of the Dental School of the Universidad Peruana Cayetano Heredia (UPCH) in Lima, Peru. These professional groups were chosen due to their close involvement in epidemiological studies of dental caries.

Once the examiner groups were identified, they were convened and divided into groups of 5 individuals each. They underwent separate calibration processes, first virtually via Zoom® and the following day, in-person in a classroom at UPCH. For the calibration process of dental caries experience, the DMFT index was chosen following WHO criteria (6): 1 for teeth with caries, 2 for teeth with restoration and caries, 3 for teeth with restoration but without caries, 4 for teeth extracted due to caries, 5 for teeth extracted for other reasons, 6 for teeth with pit and fissure sealants, 7 for abutments of bridges, 8 for unerupted teeth, and finally, 9 for non-recordable teeth.

Initially, the calibration process recommended by the WHO was adopted, considering the experience gained during the COVID-19 pandemic, which limited in-person activities and prioritized virtual ones. Three stages were proposed: theory, calibration using photographs, and calibration using natural teeth; supervised, and validated by the study's gold standard (GS), who was a specialist in restorative and aesthetic dentistry with over 20 years of professional experience.

The theory stage involved a 30-minute lecture via Zoom® to explain the DMFT index and its criteria and address participants' questions. The second stage, calibration using photographs, was conducted virtually. The GS selected 30 high-resolution photographs (minimum 1280x720p) of teeth from the Restorative and Aesthetic Dentistry Service of UPCH that met all DMFT index criteria. These photographs were coded and evaluated according to DMFT criteria by the GS. The pictures were randomly presented in two groups of 15 images, each using a Microsoft PowerPoint® presentation set to change automatically every 30 seconds. A first round of inter-examiner calibration was conducted to assess validity, with each examiner completing their responses on a Google Forms® questionnaire. Subsequently, the gold standard reviewed each case to clarify examiners' doubts. This process was repeated for a second round of photographs, where all examiners evaluated the same images.

The final stage involved natural permanent teeth donated by the Maxillofacial Surgery Service of UPCH. Due to logistical challenges and the risk of COVID-19 transmission, clinical evaluation on patients was excluded. Instead, the GS crafted natural teeth models separately and selected them based on DMFT diagnostic criteria. Teeth were mounted on plaster bases for manipulation, coding, and evaluation by the GS and examiners. Each group of 30 teeth was assessed twice by each examiner, with 30 seconds allotted per tooth. Responses were recorded using a Google Forms® questionnaire. Feedback from the GS was not provided during this stage, focusing solely on inter-examiner calibration. One hour after the final process, examiners reassessed the last group of teeth randomly ordered using the same procedure to conduct intra-examiner calibration for reliability. It is important to note that only results from examiners achieving a Kappa coefficient above 0.60 in their final process were considered for this report, following calibration process conventions to include only those who exceeded the minimum agreement threshold with the GS in inter-examiner validity calibration.

Data from all calibration processes using photographs and teeth were compiled into a single database using Microsoft Excel®, including records from both the GS and examiners. Validity and reliability values were subsequently calculated using the Kappa coefficient. These values were categorized according to WHO standards: poor agreement (<0.20), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), and almost perfect agreement (0.81–1.00), and described in absolute and relative frequencies. Statistical analysis was performed using STATA v. 17.0 with a confidence level of 95% and significance set at p<0.05.

This study was approved by the Ethics Committee of Universidad Peruana Cayetano Heredia on September 12, 2022.

RESULTS

During the inter-examiner calibration process using photographs, validity values varied with Kappa coefficients ranging between 0.55 (p<0.001) and 0.91 (p<0.001) in the first process and between 0.73 (<0.001) and 1.00 (<0.001) in the second process. Regarding the inter-examiner calibration process using teeth, Kappa values were observed ranging from 0.20 (p=0.090) to 0.81 (p<0.001) in the first process and between 0.31 (p=0.002) and 0.89 (p<0.001) in the second process. In the final intra-examiner calibration process, only 10 examiners who surpassed a Kappa value of 0.60 in their last validity process were included. In this case, the observed minimum reliability according to Kappa was 0.61 (p<0.001), while the maximum reached was 0.95 (p<0.001) (Table 1).

Additionally, Kappa values were categorized as follows: in the first validity process using photographs, 33.33% (n=5) demonstrated "substantial agreement," and 53.33% (n=8) achieved "almost perfect agreement"; in the second process, 46.67% (n=7) reported "substantial agreement," with the same percentage achieving "almost perfect agreement." Regarding tooth evaluation, no cases of "substantial agreement" were recorded in the first validity process (0.00%, n=0), and only 33.33% (n=5) reached "almost perfect agreement"; in the second process, "substantial agreement" increased to 20.00% (n=3), while "almost perfect agreement" rose to 46.67% (n=7). Finally, among the examiners considered for reliability analysis, representing 66.67% (n=10), 30.00% (n=3) demonstrated "substantial agreement," and 70.00% (n=7) achieved "almost perfect agreement" (Table 2).

DISCUSSION

Upon completion of the calibration process, dentists responsible for data collection demonstrated a consistent ability to identify and record oral conditions present in individuals (7). Within this process, two concepts stand out: validity and reliability. Whereas validity refers to the instrument's capacity to accurately predict or determine the condition as per its design, reliability considers the extent to which this measurement yields the same result for the same object/subject when taken at different times by the same or different observers (11).

In this research, approximately half of the participants in the calibration process achieved the "almost perfect agreement" category. This indicates that only these individuals successfully met the criteria to apply the CPO-D index. According to the World Health Organization manual, an evaluator is considered calibrated upon achieving an "almost perfect agreement" category post-process (1). However, other studies have established lower cutoff points (Kappa=0.69) to determine agreement among examiners (12, 13). This suggests that reducing the reference value could be considered feasible in contexts requiring, for example, greater human resource demand, as achieving a "perfect" category is often unattainable. It should be noted that these studies evaluated variations of the CPOD index, including the assessment of incipient lesions.

The process implementation consisted of 3 phases: theory, photographs, and natural teeth on models. The first phase included a 30-minute virtual lecture for the 15 evaluators to receive training, highlighting that the virtual nature of the classes facilitated everyone's participation. In line with this, the successful implementation of an online calibration process for grading epithelial dysplasia was observed in Malaysia, resulting in improved inter-observer agreement (14). In the second implementation phase, which was also virtual, a proper calibration was conducted, consisting of two rounds with clarification sessions by the GS between them. Conversely, a study aimed at calibrating 10 specialists in pediatric dentistry on diagnosing molar incisor hypomineralization employed visual and clinical resources. Video classes proved ineffective; however, their combination with photo-image discussion significantly improved diagnostic concordance. The inter-examiner reliability achieved was insufficient, attributed to the limited number of participants, constituting a significant study limitation (15). Alternatively, Biorn et al. evaluated the impact of online instructional videos for calibrating a group of dental hygiene lecturers. The results showed that this virtual alternative outperformed traditional in-person methods in key areas, such as calibration accuracy, reflecting a significant improvement in the instructors' knowledge and an increase in their confidence and sense of preparedness (16). Faculty members preferred virtual calibration due to its accessibility, effectiveness, and ease of repetition.

When comparing the success of clinical and virtual calibration of the Basic Oral Evaluation, Kallás et al. identified intra-examiner agreement levels above 80% and Kappa values indicating substantial agreement across all domains of the applied instrument. However, variability was observed in inter-examiner agreement, with some dimensions only reaching Kappa levels of fair agreement, while others achieved substantial or almost perfect agreement (17).

The final stage of this implementation utilized permanent natural teeth installed in plaster models, replacing clinical evaluation in patients. The GS selected these teeth according to CPOD Index criteria and evaluated them twice by each examiner, with 30 seconds allotted per piece. Unlike this work, most implementing DMF indices include a clinical calibration phase in patients. This phase is crucial as it combines multiple factors that complicate data collection, such as lighting and ergonomics. An example is a study targeting inexperienced dentists, comprising three four-hour training sessions conducted under conditions similar to outdoor environments. This was done to replicate as closely as possible the reality of epidemiological studies (18). Our protocol was executed during the COVID-19 pandemic, where in-person assessments were not feasible, and model teeth provided an alternative that facilitated continuity in calibration.

Regarding the development of dentistry during health emergencies, it should be mentioned that virtuality emerged as a tool that sustained its development. In dentistry, online education was not applicable because practical skills require laboratory and clinical exercises alongside theoretical knowledge. However, the coronavirus forced dental schools to use distance learning models with various digital tools. Educators pointed out issues arising from these changes, such as difficulties in teaching, practical skill training, the assessment process, etc. (19). Concerning data collection, dental researchers reported challenges in sustaining these activities during the pandemic year (20), which may have also influenced oral health instrument calibration processes.

Virtuality and technological tools enabled the success of this implementation, which continues to be used in the academic programs that initially participated. Furthermore, this experience suggests testing other oral health epidemiological indicators. However, this will pose challenges in implementation, especially in indices like community periodontal or dental fluorosis, where obtaining samples would be very complex. Consequently, simulation could prove to be an exciting alternative.

Online learning has gained significant importance after the COVID-19 pandemic, standing out for its accessibility, flexibility, and cost-effectiveness. This has established it as an essential tool in the post-COVID context. Technology has enabled the implementation of virtual environments, providing versatility and optimizing time management. Currently, a hybrid approach is being promoted, integrating digital experiences with in-person teaching, thus facilitating the continuity of education in both clinical and academic settings. Although technological tools and online learning have evolved to effectively complement education, they do not replace the direct experience offered by the clinical environment (21, 22).

Among the limitations of this study are the small sample size number of participants and dental samples. Additionally, this study lacked direct clinical evaluation in patients. Gathering more scientific evidence to justify modifying the cutoff point that defines whether an evaluator is calibrated is imperative. Establishing protocols and standards for obtaining clinical cases used in the second phase of the calibration process would be highly beneficial. The WHO criteria for epidemiological survey of dental caries stand out as one of the most widely used diagnostic criteria, with a long history and proven cost-effectiveness. Despite its limitations, this study could apply them appropriately, even in complex contexts such as a pandemic.

CONCLUSION

A dental caries experience calibration process was implemented using the CPOD Index in three stages: theory, calibration with photographs, and natural teeth. As a result, out of the initial 15 examiners, 7 (46.67%) achieved the "almost perfect agreement" category, while 3 (20.00%) achieved "substantial agreement." The use of photographs and natural teeth as calibration tools allows for effective adaptation even in challenging contexts, such as the COVID-19 pandemic, providing an alternative to maintain the standards of an epidemiological survey without compromising the validity and reliability in assessing oral health conditions. It is recommended to expand this calibration process to other epidemiological indices, as well as to increase the sample size and diversify clinical and geographical settings to obtain more representative results. Additionally, it is crucial to promote the development of digital platforms that enable continuous and asynchronous calibration, ensuring greater flexibility and accessibility for professionals.

AUTHOR CONTRIBUTION STATEMENT

Conceptualization and design: J.A.B., R.A.L.M. and J.M.L.

Literature review: M.C.G.E., A.P.F.C. and R.A.L.M.

Methodology and validation: R.A.L.M. and J.M.L.

Formal analysis: M.C.G.E., A.P.F.C. and R.A.L.M.

Investigation and data collection: M.C.G.E., A.P.F.C. and J.A.B.

Resources: J.A.B., R.A.L.M. and J.M.L.

Data analysis and interpretation: R.A.L.M. and J.M.L.

Writing-original draft preparation: M.C.G.E., A.P.F.C. and R.A.L.M.

Writing-review & editing: M.C.G.E., A.P.F.C. and R.A.L.M.

Supervision: J.A.B., R.A.L.M. and J.M.L.

Project Administration: R.A.L.M. and J.M.L.

REFERENCES

1. World Health Organization. Oral Health Surveys. Basic Methods. Geneva: World Health Organization; 2013.

2. Christian B., Amezdroz E., Calache H., Gussy M., Sore R., Waters E. Examiner calibration in caries detection for populations and settings where in vivo calibration is not practical. Community Dent Health. 2017; 34 (4): 248-53.

3. Vilella K.D., Assunção L.R., Junkes M.C., Menezes J.V., Fraiz F.C., Ferreira F. de M. Training and calibration of interviewers for oral health literacy using the BREALD-30 in epidemiological studies. Braz Oral Res. 2016; 30 (1): e90.

4. Frencken J. Caries Epidemiology and Its Challenges. Monogr Oral Sci. 2018; 27: 11-23.

5. Madera M., Bernabé E. Estimating the prevalence of untreated caries in permanent teeth from the DMF index and lifetime caries prevalence. J Public Health Dent. 2021; 81 (2): 143-9.

6. World Health Organization. Oral Health. World Health Organization; 2022. (Consultado el 24 de junio del 2022). Disponible en: https://www.who.int/news-room/fact-sheets/detail/oral-health

7. Rechmann P., Jue B., Santo W., Rechmann B.M.T., Featherstone J.D.B. Calibration of dentists for Caries Management by Risk Assessment Research in a Practice Based Research Network - CAMBRA PBRN. BMC Oral Health. 2018; 18 (1): 2.

8. Su C.W., Yen A.M., Lai H., Chen H.H., Chen S.L. Receiver Operating Characteristic Curve-Based Prediction Model for Periodontal Disease Updated With the Calibrated Community Periodontal Index. J Periodontol. 2017; 88 (12): 1348-55.

9. Su C.W., Yen A.F., Lai H., Lee Y., Chen H.H., Chen S.S. Effects of risk factors on periodontal disease defined by calibrated community periodontal index and loss of attachment scores. Oral Dis. 2017; 23 (7): 949-55.

10. Woo D.A. An innovative online approach to clinical faculty calibration [published online ahead of print, 2020 Jun 5]. J Dent Educ. 2020; 10.1002/jdd.12244.

11. Frencken J.E., Giacaman R.A., Leal S.C. An assessment of three contemporary dental caries epidemiological instruments: a critical review. British Dental Journal. 2020; 228 (1): 25-31.

12. Assaf A.V., Zanin L., Meneghim M. de C., Pereira A.C., Ambrosano G.M. Comparação entre medidas de reprodutibilidade para a calibração em levantamentos epidemiológicos da cárie dentária [Comparison of reproducibility measurements for calibration of dental caries epidemiological surveys]. Cad Saude Publica. 2006; 22 (9): 1901-7.

13. Assaf A.V., de Castro Meneghim M., Zanin L., Tengan C., Pereira A.C. Effect of different diagnostic thresholds on dental caries calibration - a 12 month evaluation. Community Dent Oral Epidemiol. 2006; 34 (3): 213-9.

14. Amarante B.C., Arima L.Y., Pinheiro E., Carvalho P., Michel-Crosato E., Bönecker M. Diagnosis training and calibration for epidemiological studies on primary and permanent teeth with hypomineralization. Eur Arch Paediatr Dent . 2022; 23: 169-77.

15. Vieira H.A.O., Araújo L.D.C., Carvalho F.K., Queiroz A.M., Paula-Silva F.W.G. Use of images and clinical experience to calibrate dental surgeons for the diagnosis of molar incisor hypomineralization. Pesqui Bras Odontopediatria Clín Integr. 2023; 23: e220040.

16. Biorn C.M., Rogo E.J., Williams R. Effectiveness of Online Faculty Calibration Activities. J Dent Hyg. 2023; 97 (5): 103-15.

17. Kallás M.S., Guardieiro B., Henrique E.A., Silva D.G., Takashi D.A., Marchini L. Telehealth in geriatric dentistry: A comparative analysis of concordance between virtual and in-person examinations for hospitalized older patients. Spec Care Dentist. 2024. 10.1111/scd.13036.

18. Nabarrette M., Santos P.R.D., Assaf A.V., Ambrosano G.M.B., Meneghim M.C, Vedovello S.A.S., et al. Longitudinal study for dental caries calibration of dentists unexperienced in epidemiological surveys. Braz Oral Res. 2023; 37: e023.

19. Antoniadou M., Rahiotis C., Kakaboura A. Sustainable Distance Online Educational Process for Dental Students during COVID-19 Pandemic. Int J Environ Res Public Health. 2022; 19 (15): 9470.

20. De Souza G.M., Tiwari T., Fox C.H., Miguez P.A., Letra A., Geisinger M.L., et al. Perception of COVID-19 pandemic restrictions on dental researchers. J Dent Educ. 2023; 87 (2): 170-81.

21. Shah S., Diwan S., Kohan L., Rosenblum D., Gharibo C., Soin A., Sulindro A., Nguyen Q., Provenzano D.A. The Technological Impact of COVID-19 on the Future of Education and Health Care Delivery. Pain Physician. 2020; 23 (4S): S367-S380.

22. Sahi P.K., Mishra D., Singh T. Medical Education Amid the COVID-19 Pandemic. Indian Pediatr. 2020; 57 (7): 652-57.