Extensive Log Analysis Using Small Language Models in Minimal Resource Environments

The process of assessing, evaluating, interpreting and extricating the meaningful representation of system originated logs is known as log analysis. Log analysis is an essential technique to identify anomalies or abnormalities that occur in the system that may reveal the issues in the systems. Traditional deterministic techniques like rule based are relied on manual inspection. It faces the scalability issue for identifying huge unstructured logs and Natural Language Processing techniques play a significant role in log analysis. Large Language Model extracts meaningful insight from the logs, identify deviations from normal behavior by recognizing patterns and makes log analysis techniques automated efficiently. Large Language Models are crucial and effective when huge resources are available for the entire system. Problems occur in analyzing detailed logs for a resource constrained environment. The Small Language Model plays a critical role in performing log analysis competently with minimal resources and illustrates the insight of AI effects. This research intends to leverage log processing and anomalies detection along with Small Language Model, which have demonstrated advanced capabilities in natural language understanding and pattern recognition, to automatically process log data in realtime with minimal resources, and applies clustering to identify anomalies more effectively.