COVID-19 has been designated as a once-in-a-century pandemic, and its impact is still being felt severely in many countries, due to the extensive human and green casualties. While several vaccines are under various stage of development, effective screening procedures that help detect the disease at early stages in a non-invasive and resource-optimized manner are the need of the hour. X-ray imaging is fairly accessible in most healthcare institutions and can prove useful in diagnosing this respiratory disease. Although a chest X-ray scan is a viable method to detect the presence of this disease, the scans must be analyzed by trained experts accurately and quickly if large numbers of tests are to be processed. In this paper, a benchmarking study of different preprocessing techniques and state-of-the-art deep learning models is presented to provide comprehensive insights into both the objective and subjective evaluation of their performance. To analyze and prevent possible sources of bias, we preprocessed the dataset in two ways-first, we segmented the lungs alone, and secondly, we formed a bounding box around the lung and used only this area to train. Among the models chosen to benchmark, which were DenseNet201, EfficientNetB7, and VGG-16, DenseNet201 performed better for all three datasets.