Stealthy Domain Generation Algorithms

• Published in: IEEE transactions on information forensics and security Vol. 12; no. 6; pp. 1430 - 1443
• Main Authors: Yu Fu, Lu Yu, Hambolu, Oluwakemi, Ozcelik, Ilker, Husain, Benafsh, Jingxuan Sun, Sapra, Karan, Dan Du, Beasley, Christopher Tate, Brooks, Richard R.
• Format: Journal Article
• Language: English
• Published: New York IEEE 01.06.2017; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Algorithms; BotDigger; botnet; Computer simulation; Cybersecurity; DGA; DGA detection; Domain names; edit distance; Game theory; Grammars; Hidden Markov models; HMM; Indexes; intrusion detection; IP (Internet Protocol); Jaccard index; Kullback-Leibler distance; Malware; Markov chains; Measurement; PCFG; Personnel; Pleiades; Probabilistic logic; Probability distribution; Security; Security personnel
• ISSN: 1556-6013; 1556-6021
• DOI: 10.1109/TIFS.2017.2668361

Botnets are groups of compromised computers that botmasters (botherders) use to launch attacks over the Internet. To avoid detection, botnets use DNS fast flux to change the mapping between IP addresses and domain names periodically. Domain generation algorithms (DGAs) are employed to generate a large number of domain names. Detection techniques have been proposed to identify malicious domain names generated by DGAs. Three metrics, Kullback-Leibler (KL) distance, Edit distance (ED), and Jaccard index (JI), are used to detect botnet domains with up to 100% detection rate and 2.5% false-positive rate. In this paper, we propose two DGAs that use hidden Markov models (HMMs) and probabilistic context-free grammars (PCFGs), respectively. Experiment results show that DGA detection metrics (KL, JI, and ED) and detection systems (BotDigger and Pleiades) have difficulty detecting domain names generated using the proposed approaches. Game theory is used to optimize strategies for both botmasters and security personnel. Results show that, to optimize DGA detection, security personnel should use the ED detection technique with probability 0.78 and JI detection with probability 0.22, and botmasters should choose the HMM-based DGA with probability 0.67 and PCFG-based DGA with probability 0.33.