Hybrid TF-IDF and BERT Embedding Framework for Fake News Detection Using Supervised Machine Learning Models

Abstract

With the increase in the emergence of digital media and social network platforms, the production of fake news and misinformation has reached unprecedented levels. However, machine learning-based techniques for classifying fake news employ statistical methods, including TF-IDF for the representation of text which can capture text frequencies but not their semantics and context. This paper proposes a hybrid approach for feature extraction in which hybrid feature vectors are created by combining text vectors generated using the TF-IDF technique with sentence embeddings using the all-MiniLM-L6-v2 Sentence Transformer model, and the size of the hybrid feature vector is 5,384. The five classification models used in this paper are Logistic Regression, Random Forest, XGBoost, LightGBM, and MLP. Unlike previous approaches, where a single evaluation was done using a single train/test split, the hybrid representation approach is compared to the unimodal approaches (TF-IDF and BERT) using an ablation study, and results are reported as mean ± std. deviation on five splits (seed 42-46). In case of the ISOT Fake News Dataset, the multi-modal representation reached the highest average accuracy of 98.96% ± 0.23% (MLP classifier), with a slight improvement of around 0.3 percentage points relative to the most accurate uni-modal approach (TF-IDF alone: 98.68% ± 0.16%, LightGBM classifier) and a much larger one than the BERT alone model (97.03% ± 0.25%, MLP classifier). From the ablation study results, it becomes clear that the utilization of TF-IDF features is still the leading technique for the ISOT dataset, despite the marginal improvement that the semantic information provides for the model.