[+] Malware Analysis Techniques
/ 4 min read
Table of Contents
How I Approach Malware Analysis
I got into malware analysis through CTFs. The forensics challenges hooked me - figuring out what a suspicious binary does felt like solving a crime. Over time I built a structured workflow that I follow for every sample. Here is how I do it.
My Setup
First thing I learned the hard way: never run malware on your host machine. I set up a dedicated analysis environment:
- A Windows 10 VM in VirtualBox with snapshots (I revert after every analysis)
- REMnux as my Linux analysis VM
- FakeNet-NG for simulating network responses
- Host-only networking with no internet access
I snapshot the clean state and revert after every session. Took me one close call to make this a habit.
Static Analysis - My First Pass
I always start static. No execution, just looking at what the file tells me before I let it run.
File Identification
# What am I looking at?file suspicious.exe
# Get hashes for lookupsha256sum suspicious.exe
# Quick check on VirusTotal# If 50+ engines flag it, I already know it is bad# But I still want to know WHAT it doesStrings - My Favorite Starting Point
Strings reveal so much. I run this on every sample:
strings suspicious.exe | less
# For Windows malware, wide strings are commonstrings -el suspicious.exe
# When strings are obfuscated, FLOSS recovers themfloss suspicious.exeWhat I look for:
- IP addresses and domains - tells me where it phones home
- Registry paths - shows me how it persists
- File paths - where it drops stuff
- API names - what it is capable of
- Debug strings the author forgot to remove (happens more than you think)
PE Header Check
For Windows binaries, I check the PE structure:
# Imports tell me capabilitiesrabin2 -i suspicious.exe
# High entropy sections = packed or encryptedrabin2 -S suspicious.exeMy red flags:
- Only imports LoadLibrary and GetProcAddress (resolves everything at runtime to hide)
- Section entropy above 7.0 (packed)
- Weird section names (.UPX, .vmp, random garbage)
- Import of VirtualAlloc + WriteProcessMemory + CreateRemoteThread (process injection)
Loading in Ghidra
If static strings and headers look interesting, I load it in Ghidra:
- Find the entry point, trace main logic
- Look for anti-analysis checks early in execution
- Map out what functions do based on API calls
- Check for encrypted config blobs (common in RATs)
Dynamic Analysis - Watching It Run
Once I understand the structure, I let it execute in my sandbox and watch what happens.
Process Monitoring
I launch ProcMon before executing the sample and filter for the process:
What I watch for:- File creates/writes (dropping payloads, logs)- Registry modifications (persistence)- Network connections (C2 communication)- Process creation (spawning child processes, injection)Process Hacker gives me a live view of the process tree, loaded DLLs, and open handles.
Network Capture
I run Wireshark in the background:
# Or tcpdump on REMnuxtcpdump -i eth0 -w malware_capture.pcapFakeNet-NG simulates responses so the malware thinks it is online. This way I see:
- DNS queries for C2 domains
- HTTP beacons (regular interval callbacks)
- Data being exfiltrated
- The protocol it uses to talk to its server
Debugging Key Moments
When I need to understand specific behavior, I attach x64dbg:
- Set breakpoints on interesting APIs (CreateFile, RegSetValue, send)
- Step through decryption routines to get plaintext configs
- Catch the moment it injects into another process
- Find the unpacked code if it is packed
Memory Analysis
Sometimes the real payload only exists in memory. I use Volatility to analyze dumps:
# List processesvolatility -f memory.dmp --profile=Win10x64 pslist
# Find injected codevolatility -f memory.dmp --profile=Win10x64 malfind
# Check network connectionsvolatility -f memory.dmp --profile=Win10x64 netscanThis caught me several injected DLLs that never touched disk.
What I Document
For every sample I analyze, I record:
Indicators of Compromise (IOCs)
- Hashes (MD5, SHA256)
- C2 IPs and domains
- Dropped file paths
- Registry keys modified
- Mutex names (used to prevent double-infection)
Capabilities
- Persistence mechanism
- C2 protocol
- Data it steals
- Lateral movement techniques
- Evasion methods
Behavior Timeline
1. Execution starts, checks if debugger present2. Decrypts embedded config (XOR key: 0x41)3. Drops payload to %TEMP%\svchost.exe4. Creates registry run key for persistence5. Connects to 192.168.x.x:4436. Sends system info (hostname, username, OS version)7. Waits for commands in a loopWriting Detection Rules
After analysis, I write YARA rules so the sample gets caught next time:
rule Sample_RAT_Variant { meta: author = "jaswanth" description = "Detects RAT variant found during analysis" date = "2026-04"
strings: $mutex = "Global\\xRAT_mutex" $config_xor = { 41 41 41 41 } $api1 = "CreateRemoteThread" $api2 = "VirtualAllocEx" $reg = "Software\\Microsoft\\Windows\\CurrentVersion\\Run"
condition: uint16(0) == 0x5A4D and 3 of them}Tools I Use Daily
| What | Tool |
|---|---|
| File identification | file, DIE, pestudio |
| Strings | strings, FLOSS |
| Disassembly | Ghidra, IDA |
| Debugging | x64dbg |
| Process monitoring | ProcMon, Process Hacker |
| Network | Wireshark, FakeNet-NG |
| Memory forensics | Volatility |
| Detection | YARA |
| Sandbox | FlareVM, REMnux |
What I Learned Along the Way
- Start safe. Always sandbox. No exceptions.
- Document as you go. I forgot details too many times early on.
- Do not skip static analysis. It gives you a map before you explore.
- When stuck, step back and check strings again. Authors leave clues.
- Every sample teaches something. Even commodity malware has tricks worth noting.
Malware analysis is detective work. I enjoy the puzzle of figuring out what someone built and how to stop it.