Enhancing Financial Sentiment Analysis: Integrating the LoughranMcDonald Dictionary with BERT for Advanced Market Predictive Insights

One critical aspect of financial markets is understanding investor sentiment to facilitate effective decision-making. This study integrates traditional sentiment analysis methods, such as the Loughran-McDonald (LM) dictionary - designed for financial sentiment - with advanced Natural Language Processing (NLP) techniques using Bidirectional Encoder Representations from Transformers (BERT). The LM dictionary provides domain-specific word lists to label sentiment, whereas BERT enhances this by capturing nuanced meanings and semantic relationships in financial texts. It involves pre-processing Financial NewsHeadlines, applying the LM dictionary for sentiment scoring, and finetuning a pretrained BERT model to classify sentiment. A PyTorch dataset was created, tokenized using BERT, and processed through the model using techniques like dropout regularization and cross-entropy loss for optimization. The hybrid approach yields promising results: a classification accuracy of 97%, precision of 0.98, recall of 0.93, and an F1 score of 0.95, confirming its effectiveness in capturing sentiment polarity. In addition, comparisons between dictionary-labelled and pre-annotated datasets demonstrate the model's improved generalization ability. The results also show that our hybrid model outperformed various other existing models. This hybrid approach attempts to improve accuracy in capturing sentiment polarity by implementing methods to overcome imbalanced dataset, thereby facilitating a better understanding of sentiment in financial reports and facilitating informed decision-making. The integration of Named Entity Recognition (NER) with sentiment analysis based on sentiment polarity (positive, negative, or neutral) enables a more granular view of how specific companies are perceived in financial reports by highlighting the entities that are most affected by market sentiment.