Video anomaly detection has gained much attention in the computer vision community due to its wide applications in security. Specifically, the focus has been on feature extraction and the design of inference algorithms. The extraction of features to model the normality is challenging due to the scarcity of data and supervision. To this end, current computer vision technologies use reconstruction based methods that relied on auto-encoders to reconstruct normal events in an unsupervised manner. Higher reconstruction errors are often used to detect anomalies. However, the use of multiple auto-encoders to extract features (temporal and appearance) is redundant and expensive for videos. In this context, the present study proposes a novel feature extractor that uses a single CNN architecture to extract both temporal and appearance features. Also, this model is trained for classification tasks which are adapted as feature extractors in anomaly detection. The training of this model is easy and can be deployed efficiently due to its lightweight architecture. Further, the proposed model has been quantitatively evaluated on the UCSD ped 2 dataset and found to perform competitively with an AUC of 0.958.