Security detection algorithm using CNN: Anomaly detection for API call sequence

• Published in: Journal of computational methods in sciences and engineering Vol. 25; no. 4; pp. 3239 - 3254
• Main Authors: Chang, Jie, Shi, Lipeng, Li, Zhefeng, Zuo, Xiaojun, Hou, Botao
• Format: Journal Article
• Language: English
• Published: London, England SAGE Publications 01.07.2025; Sage Publications Ltd
• Subjects: Accuracy; Algorithms; Anomalies; Application programming interface; Artificial neural networks; Clustering; Data processing; Feature extraction; Real time; Redundancy; Security; security detection algorithm; CNN; anomaly detection; API call sequence; feature extraction
• ISSN: 1472-7978; 1875-8983
• DOI: 10.1177/14727978251318813

This study proposes a security detection algorithm based on convolutional neural networks (CNNs) to enhance anomaly detection in API call sequences, addressing the challenges of capturing complex temporal relationships and nonlinear features in high-dimensional sparse API data. The proposed algorithm includes several preprocessing steps, such as deduplication to reduce redundancy, feature extraction using the TF-IDF (term frequency-inverse document frequency) algorithm, and logarithmic transformation to mitigate the impact of high-frequency APIs. An importance scoring mechanism is introduced to quantify the role of each API in anomaly detection. A customized TextCNN architecture is designed for API sequences, incorporating input layers, word embedding, multi-size convolution and pooling layers, attention layers, and fully connected layers. The attention layer is particularly applied to enhance the detection efficiency of evasion features. The model is trained using the Sigmoid activation function, CrossEntropyLoss loss function, and optimized via the Adam algorithm. The Softmax function is utilized to transform the feature vector into a probability distribution, with a threshold of 0.5 for anomaly detection. Clustering and auxiliary information are integrated to further improve classification accuracy and guide security strategy formulation. Experimental results demonstrate that the optimized TextCNN anomaly detection algorithm achieves an average accuracy of 95.88%, a recall rate of 91.23%, a false positive rate of 2.34%, and a false negative rate of 1.78%. These findings highlight the algorithm’s ability to enhance feature extraction accuracy, improve high-dimensional data processing, and provide an effective solution for real-time security monitoring, thus strengthening the security of the development environment.