When it comes to malware attacks, one of the more common techniques is “living off the land” (LOtL). Utilizing standard tools or features that already exist in the target environment allows these attacks to blend into the environment and avoid detection. While these techniques can appear normal in isolation, they start looking suspicious when observed in the parent-child context. This is where the ProblemChild framework can help. In this blog, we will talk about how you can use Elastic machine learning to create your own ProblemChild framework to detect LOtL activity in Windows process event data (we will be referring to Windows process events as just “events” throughout this blog). We will talk in detail about the following:
Extracting features from event metadata
Training a supervised model to classify events as malicious vs. benign
Using the trained model to enrich event data at ingest time
Picking out the most unusual events for analysts to triage
If you would like to follow along with this blog, we recommend starting a free 14-day Elastic trial. All the supporting materials for this blog are also available in the examples repository. BackgroundLiving-off-the-land binaries (LOLBins) are Microsoft-signed binaries that come pre-installed on the operating system. These binaries can sometimes have unexpected features outside of their core functionality, which attackers can leverage. For example, the task scheduler in Windows, which allows an admin to create, delete, run, and schedule tasks on a local computer. However, attackers may leverage the binary to bypass User Account Control (UAC) and escalate privileges. The use of these binaries complicates the discovery of the attack, since adversary behavior is mixed with traditional benign operating system activity. Things get a little interesting when viewed from a parent-child lens, since unusual child processes spawned by a parent process can indicate malicious activity. For example, word.exe spawning powershell.exe could indicate a Spearphishing Attachment. Current solutions to detect LOtL attacks using parent-child relationships include writing rules and heuristics. While these solutions work well, they can sometimes be either too rigid or too lax and do not generalize well. There is also a significant amount of manual effort that goes into writing them. With ProblemChild, the goal remains the same: we hope to provide better generalization with the added advantage of ranking and prioritizing events for further investigation using machine learning. The ProblemChild frameworkProblemChild uses data frame analytics available in the Elastic Stack to build a supervised model to classify events as malicious or benign using features extracted from event metadata. It then uses anomaly detection to pick out “high priority” events for further analysis from those detected as malicious by the supervised model. DataFor the supervised model, we gathered Windows process event metadata from a variety of sources like the Splunk Attack data, Splunk botsv1, Red Canary Atomic Red Team, and several internal databases. An example of a raw sample used in training is as follows: { "timestamp_utc": "2019-06-14 15:31:17Z", "pid": 372, "integrity_level": "system", "elevation_type": "default", "signature_status": "trusted", "serial_event_id": 1007, "elevated": true, "signature_signer": "Microsoft Windows Publisher", "event_subtype_full": "already_running", "command_line": "C:\Windows\System32\svchost.exe -k LocalSystemNetworkRestricted -p", "parent_process_name": "services.exe", "ppid": 620, "sha256": "7fd065bac18c5278777ae44908101cdfed72d26fa741367f0ad4d02020787ab6", "user_name": "SYSTEM", "process_path": "C:\Windows\System32\svchost.exe", "user_sid": "S-1-5-18", "timestamp": 132049998770000000, "process_name": "svchost.exe", "original_file_name": "svchost.exe", "parent_process_path": "C:\Windows\System32\services.exe", "unique_pid": 1007, "md5": "8a0a29438052faed8a2532da50455756", "sha1": "a1385ce20ad79f55df235effd9780c31442aa234", "unique_ppid": 1006, "event_type_full": "process_event", "opcode": 3, "user_domain": "NT AUTHORITY" } Sample raw document containing Windows process event metadata Feature engineeringSince we wanted to focus on identifying LOtL activity using parent-child context, we started by extracting features that capture information about the process itself, its parent, and surrounding contextual information (e.g., elevation level, system user, etc.) from the raw event metadata (shown above) as follows:
Process name
Parent process name
Commandline arguments
Process path
Parent process path
Event subtype
Whether event is elevated
Elevation type
Integrity level
Normalized process path
Whether process is signed
Whether signer is trusted
Whether user is running as system
Filename mismatch
Whether process name ends with exe
All of the feature engineering was done using processors already available in the Elastic Stack or using custom scripts written in Painless, which were then used in script processors. A high-level breakdown of the featurization process is as follows: Since the model supports Windows process events for the Elastic Endpoint Security integration, Elastic Endgame, and Winlogbeat, we standardized the field names across the different agents. We did this so the model always has the same set of input fields, regardless of the agent type. This was done using separate script processors for each agent type. We then used script processors to build features that were derived from the common set of fields. Example: The following script processor sets the feature feature_ends_with_exe to true if the process name associated with the event ends with ".exe" and false otherwise. { "script": { "lang": "painless", "source": """ if(ctx.feature_process_name.contains(".exe")) { ctx.feature_ends_with_exe = true } else { ctx.feature_ends_with_exe = false } """ } } Example of using script processors for feature extraction We noticed that minor variations like change in case, usernames, certain special characters (mainly ", /, ), and appearance of random numbers/hexadecimal values in fields like commandline arguments and process paths were affecting the performance of our models, and needed to be normalized and/or obfuscated. We also found that replacing certain Windows directories with appropriate tokens, for example replacing windows/system32 and windows/syswow64 with the token win_system_dir, further improved model performance. These normalizations and obfuscations were done using the lowercase and gsub processors available in the Elastic Stack.
Example: The following processor replaces text matched by the pattern defined in the pattern field with the string 'process_id' in the feature_command_line field. { "gsub": { "field": "feature_command_line", "pattern": "[0-9a-f]{4,}-[0-9a-f]{4,}-[0-9a-f]{4,}-[0-9a-f-]{4,}", "replacement": "process_id" } } Example of using pre-built Elastic Stack processors for normalization of features Finally, we used a series of script processors to extract n-gram features from process and parent process names and paths and commandline arguments. After experimenting with different n-gram lengths, we concluded that bigrams were the most optimum fit and provided the best trade-off between dimensionality of the feature set and model performance. Example: The following processor generates bigrams for the field feature_process_name. { "script": { "id": "ngram-extractor", "params": { "ngram_count": 2, "field": "feature_process_name", "max_length": 100 } } } Script processor for extracting n-grams All the processors mentioned so far were a part of an ingest pipeline used to featurize raw events from the source index and re-index them. Please refer to the examples repository for detailed instructions on featurization and the relevant configurations, scripts, etc. An example of features created by the ingest pipeline is as follows: { "feature_command_line_2-gram_feature10" : "", "feature_process_parent_executable_2-gram_feature53" : ".e", "feature_process_parent_executable_2-gram_feature54" : "ex", "feature_process_parent_executable_2-gram_feature55" : "xe", "feature_process_parent_executable_2-gram_feature56" : "", "feature_process_executable_2-gram_feature49" : "ka", "feature_process_executable_2-gram_feature48" : """\k""", "feature_process_executable_2-gram_feature47" : """r\""", "feature_command_line" : "kaps.exe -u", "feature_process_executable_2-gram_feature46" : "er", "feature_process_executable_2-gram_feature45" : "le", "feature_process_executable_2-gram_feature44" : "ll", "feature_process_executable_2-gram_feature43" : "il", "feature_process_executable_2-gram_feature42" : "ki", "feature_process_executable_2-gram_feature41" : """\k""", "feature_process_executable_2-gram_feature40" : """s\""", "feature_running_as_system" : false, "feature_process_signer_trusted" : true, "feature_process_parent_executable_2-gram_feature46" : "er", "feature_process_parent_executable_2-gram_feature47" : """r\""", "feature_process_parent_executable_2-gram_feature48" : """\k""", "feature_process_parent_executable_2-gram_feature49" : "ka", "feature_process_parent_executable_2-gram_feature42" : "ki", "feature_process_parent_executable_2-gram_feature43" : "il", "feature_process_parent_executable_2-gram_feature44" : "ll", "feature_process_parent_executable" : """c:\win_system_dir\drivers\rivetnetworks\killer\kaps.exe""", "feature_process_parent_executable_2-gram_feature45" : "le", "feature_process_parent_executable_2-gram_feature50" : "ap", "feature_process_parent_executable_2-gram_feature51" : "ps", "feature_process_parent_executable_2-gram_feature52" : "s.", "feature_process_executable_2-gram_feature56" : "", "feature_process_executable_2-gram_feature55" : "xe", "feature_process_executable_2-gram_feature54" : "ex", "feature_process_executable_2-gram_feature53" : ".e", "feature_process_executable_2-gram_feature52" : "s.", "feature_process_executable_2-gram_feature51" : "ps", "feature_process_executable_2-gram_feature50" : "ap", "feature_process_name" : "kaps.exe", "feature_process_executable_2-gram_feature29" : "iv", "feature_process_executable_2-gram_feature28" : "ri", "feature_process_executable_2-gram_feature27" : """\r""", "feature_process_executable_2-gram_feature26" : """s\""", "feature_process_executable_2-gram_feature25" : "rs", "feature_process_executable_2-gram_feature24" : "er", "feature_process_executable_2-gram_feature23" : "ve", "feature_process_executable_2-gram_feature22" : "iv", "feature_process_executable_2-gram_feature21" : "ri", "feature_process_executable_2-gram_feature20" : "dr", "feature_process_name_2-gram_feature4" : ".e", "feature_process_parent_name_2-gram_feature4" : ".e", "feature_process_name_2-gram_feature5" : "ex", "feature_process_parent_name_2-gram_feature3" : "s.", "feature_process_name_2-gram_feature6" : "xe", "feature_process_parent_name_2-gram_feature2" : "ps", "feature_process_name_2-gram_feature7" : "", "feature_process_parent_name_2-gram_feature1" : "ap", "feature_process_parent_name_2-gram_feature7" : "", "feature_process_parent_name_2-gram_feature6" : "xe", "feature_process_parent_name_2-gram_feature5" : "ex", "feature_ends_with_exe" : true, "feature_process_executable_2-gram_feature39" : "ks", "feature_process_executable_2-gram_feature38" : "rk", "feature_process_executable_2-gram_feature37" : "or", "feature_process_executable_2-gram_feature36" : "wo", "feature_process_executable_2-gram_feature35" : "tw", "feature_process_executable_2-gram_feature34" : "et", "feature_process_executable_2-gram_feature33" : "ne", "feature_process_executable_2-gram_feature32" : "tn", "feature_process_name_2-gram_feature0" : "ka", "feature_process_parent_name_2-gram_feature0" : "ka", "feature_process_executable_2-gram_feature31" : "et", "feature_process_name_2-gram_feature1" : "ap", "feature_process_executable_2-gram_feature30" : "ve", "feature_process_name_2-gram_feature2" : "ps", "feature_process_name_2-gram_feature3" : "s.", "feature_process_parent_executable_2-gram_feature17" : "32", "feature_process_parent_executable_2-gram_feature18" : """2\""", "feature_process_parent_executable_2-gram_feature19" : """\d""", "feature_process_parent_executable_2-gram_feature3" : "wi", "feature_process_parent_executable_2-gram_feature13" : "st", "feature_process_parent_executable_2-gram_feature2" : """\w""", "feature_process_parent_executable_2-gram_feature14" : "te", "feature_process_parent_executable_2-gram_feature5" : "nd", "feature_process_parent_executable_2-gram_feature15" : "em", "feature_process_parent_executable_2-gram_feature4" : "in", "feature_process_parent_executable_2-gram_feature16" : "m3", "feature_process_parent_executable_2-gram_feature7" : "ow", "feature_process_parent_executable_2-gram_feature6" : "do", "feature_process_parent_executable_2-gram_feature10" : """\s""", "feature_process_parent_executable_2-gram_feature9" : """s\""", "feature_process_parent_executable_2-gram_feature11" : "sy", "feature_process_parent_executable_2-gram_feature8" : "ws", "feature_process_parent_executable_2-gram_feature12" : "ys", "feature_process_parent_executable_2-gram_feature1" : """:\""", "feature_process_parent_executable_2-gram_feature0" : "c:", "feature_process_signed" : true, "feature_elevation_type" : "limited", "feature_integrity_level" : "medium", "feature_elevated" : false, "feature_process_executable_2-gram_feature19" : """\d""", "feature_process_executable_2-gram_feature18" : """2\""", "feature_process_executable_2-gram_feature17" : "32", "feature_process_executable_2-gram_feature16" : "m3", "feature_process_executable_2-gram_feature15" : "em", "feature_process_executable_2-gram_feature14" : "te", "feature_process_executable_2-gram_feature13" : "st", "feature_process_executable_2-gram_feature12" : "ys", "feature_process_executable_2-gram_feature11" : "sy", "feature_process_executable_2-gram_feature10" : """\s""", "feature_process_executable" : """c:\win_system_dir\drivers\rivetnetworks\killer\kaps.exe""", "feature_filename_mismatch" : false, "feature_process_executable_2-gram_feature8" : "ws", "feature_command_line_2-gram_feature4" : ".e", "feature_process_executable_2-gram_feature7" : "ow", "feature_command_line_2-gram_feature3" : "s.", "feature_process_executable_2-gram_feature6" : "do", "feature_command_line_2-gram_feature6" : "xe", "feature_process_executable_2-gram_feature5" : "nd", "feature_command_line_2-gram_feature5" : "ex", "feature_process_parent_executable_2-gram_feature39" : "ks", "feature_command_line_2-gram_feature0" : "ka", "feature_command_line_2-gram_feature2" : "ps", "feature_process_executable_2-gram_feature9" : """s\""", "feature_command_line_2-gram_feature1" : "ap", "feature_process_parent_executable_2-gram_feature35" : "tw", "feature_normalized_ppath" : "win_system_dir", "feature_process_parent_executable_2-gram_feature36" : "wo", "feature_process_parent_executable_2-gram_feature37" : "or", "feature_process_parent_executable_2-gram_feature38" : "rk", "feature_process_parent_executable_2-gram_feature31" : "et", "feature_process_parent_executable_2-gram_feature32" : "tn", "feature_process_parent_executable_2-gram_feature33" : "ne", "feature_process_parent_executable_2-gram_feature34" : "et", "feature_process_parent_executable_2-gram_feature40" : """s\""", "feature_process_parent_executable_2-gram_feature41" : """\k""", "feature_event_action" : "creation_event", "feature_process_executable_2-gram_feature0" : "c:", "feature_process_executable_2-gram_feature4" : "in", "feature_process_executable_2-gram_feature3" : "wi", "feature_process_executable_2-gram_feature2" : """\w""", "feature_process_parent_name" : "kaps.exe", "feature_process_executable_2-gram_feature1" : """:\""", "feature_process_parent_execut
Autentifică-te pentru a adăuga comentarii
Alte posturi din acest grup
Version 7.17.27 of the Elastic Stack was released today. We recommend you upgrade to this latest version. We recommend 7.17.27 over the previous versi