Artificial Intelligence and its applications are evolving in all domains especially in the field of medical science. Voluminous amount of clinical data is available and most of the data remains unused. If it is used in effective way then it will be useful in diagnosing the human lives in earlier stages. An effective classification system will help the medical practitioners to provide appropriate diagnosis in earlier stage. Usually, medical data will have lot of features and if all the features are included in decision making it may lead to over fitting of the classification model and in turn accuracy will be affected. Also, if all the features are included in decision making then it will take more resource in building the classification model. Hence there is a need for an effective dimensionality reduction method which not only reduces the number of structures but also improves the classification accuracy. This article recommends a novel ensemble technique named Hybrid Linear Discriminant Analysis (HLDA) for reducing the dimensionality of the medical data. And Random Forest classifier algorithm has been used for building the classification system. For proving the effectiveness of the proposed method, it is compared with LDA, PLS, PCA dimensionality reduction algorithms. Three UCI benchmark datasets namely Breast cancer Wisconsin dataset, Cervical cancer dataset & Heart failure clinical records dataset have been used for the evaluation of the work. The new result proves that this proposed method outclasses the other methods in accuracy, specificity, and sensitivity.