Demystifying Market Dynamics: Utilising STM and Explainable AI for Stock Analysis

The financial sector faced data overload, ineffective data scrutiny, and challenges discerning market trends within extensive datasets. Structural Topic Modeling (STM) facilitates the extraction of hidden themes from textual stock data, enriching sentiment analysis, risk evaluation, and investment strategies offering indispensable perspectives on market behaviour. This helps investors detect trends early, find shifts in sentiment and explore systemic vulnerabilities, thus improving portfolio management and financial decision-making processes. The proposed methodology begins with gathering and preparing textual data from various origins for 6000 stocks, followed by the use of Topic Modeling to find underlying themes. The model is refined by integrating structural attributes using LIME for the better understanding of the topics. Comparison of the impacts of different topics or covariate levels offers a comprehension of market dynamics. Key findings from the project include the ability to assess the impact of various publishers on specific stocks and compare the influence of different topics or covariate levels.