Emergence of dermatophytosis pose alarming concerns due to its recurrence and difficulty in management. Diagnostic laboratories are beset with burgeoning challenge of screening multiple specimens for direct microscopy. This has necessitated automation of microscopic image analysis of clinical specimens to augment efficiency and ease in laboratory workflow. Such approaches may be used as point of care facility in the outpatient departments of dermatologists. We identified a robust deep transfer learning model by comparing four popular pre-trained CNN architectures namely EfficientNetB0, VGG16, ResNet50 and MobileNet. Less than 33% of the CNN layers were frozen and the remaining were enabled to learn new features from dermatophyte datasets of clinical origin. EfficientNetB0 outperformed all other models with an accuracy of 98.52%, AUC of 0.99 and F1 score of 0.98 with 97.6% sensitivity and 99.4% specificity. These results with unstained samples are comparable and even better than those from fluorescent stained studies reported earlier.