Epilepsy is the most common neurological disorder that has affected 50 million worldwide population. Epilepsy is a chronic condition that is characterized as recurrent and unprovoked seizures. A seizure is defined as abnormal, excessive paroxysmal discharge of the cerebral neurons. In general, epilepsy evaluation, the high time resolution (~1–2 ms) electrophysiological data recorded using electroencephalography (EEG) is commonly used. The EEG data are visually inspected for epileptic seizure instances. However, the manual interpretation may lead to subjective error-causing misdiagnosis, and identifying the seizure instances in high temporal resolution data may be time-consuming. To mitigate these problems, the proposed study uses a hand-crafted deep learning model to classify the epileptic and non-epileptic EEG data. This study used EEG data acquired from two databases—University of Bonn and Physiobank Children’s Hospital Boston—Massachusetts Institute of Technology (CHB-MIT). We split the EEG data in 80:20 ratio for training and testing the model. The model was trained under three network specifications, which were designed based on the number of high-level features and low-level features. The result showed that the proposed study successfully classified epileptic and non-epileptic signals with the accuracy of 67% and 92% for University of Bonn data and CHB-MIT EEG data, respectively, for the network specification which had a high number of low-level features. We tested the model for optimal value of number of epochs (20) and learning rate (0.001). The study shows that the MIT-CHB data are suitable for classification as they have a good number of samples and balanced epileptic and non-epileptic signals.