DeepSeg-Net: a novel approach for automated segmentation of cytoplasm and nucleus in pap smear images for enhanced cervical cancer diagnosis

Accurate segmentation of the cytoplasm and nucleus in Pap smear images remains a challenging task for automated cervical cancer screening due to weak cell boundaries, overlapping structures, and staining variability. In this work, DeepSeg-Net, a novel deep learning-based segmentation framework, has been designed to robustly delineate cellular structures in cervical cytology images. The core novelty of DeepSeg-Net lies in its unified segmentation strategy that enhances feature representation while selectively focusing on diagnostically relevant regions, enabling reliable separation of cytoplasm and nucleus even under complex imaging conditions. By effectively suppressing background interference and strengthening multi-scale feature learning, the proposed approach addresses key limitations observed in existing segmentation methods. The performance of DeepSeg-Net is systematically evaluated against widely used U-Net-based architectures under identical experimental settings. The results demonstrate that the proposed method consistently outperforms conventional approaches in segmenting both cytoplasm and nucleus across cancerous and non-cancerous samples. Furthermore, the model exhibits strong generalisation capability when tested on an independent benchmark dataset with differing imaging characteristics. Overall, DeepSeg-Net offers a robust and effective segmentation backbone for Pap smear image analysis and has the potential to significantly support downstream cervical cancer screening and diagnostic workflows through accurate pixel-level cellular delineation.