Tooth shape and sex estimation: a 3D geometric morphometric landmark-based comparative analysis of artificial neural networks, support vector machines, and Random Forest models

This study evaluated the performance of three artificial intelligence (AI) algorithms—support vector machine (SVM), artificial neural network (ANN), and Random Forest (RF)—in sex estimation using 3D geometric morphometric data derived from nine permanent tooth classes in 120 individuals (60 males, 60 females). Dental casts from 60 males and 60 females, aged 13–20 were digitized using a 3D scanner. Anatomic and geometric landmarks were identified on nine tooth types (maxillary/mandibular premolars and molars) via 3D Slicer software. Landmark coordinates underwent Procrustes superimposition and principal component analysis. Three AI models (ANN, SVM, RF) were trained on pre-processed landmark data, with performance evaluated using fivefold cross validation, accuracy, precision, recall, F1-score, and AUC. RF outperformed SVM and ANN across all tooth types, achieving the highest accuracy (97.95% for mandibular second premolars) and balanced precision/recall (0.85–1.0). SVM showed moderate performance (70–88% accuracy), while ANN had the lowest metrics (58–70% accuracy). Maxillary first molars (95.83% accuracy) and mandibular second premolars (97.95%) exhibited the highest sexual dimorphism. RF demonstrated minimal sex bias, whereas ANN struggled with female classification (recall: 0.33–0.88 vs. males: 0.36–1.0). Feature analysis highlighted mandibular premolars as most dimorphic, with RF leveraging complex spatial relationships between landmarks effectively. Random Forest emerged as the most robust model for sex estimation using 3D dental landmarks, likely due to its ability to handle tabular data and high-dimensional feature spaces. Traditional machine learning models (RF, SVM) outperformed ANN, suggesting data set structure and feature engineering influence AI efficacy. These findings underscore AI’s potential to enhance objectivity and accuracy in forensic odontology, particularly with geometric morphometric data. Future research should explore hybrid models combining AI strengths with traditional morphometrics for improved reliability.