When AI tooling looks like a LOLBin: a Splunk process hunt on EventID 4688

The goal was straightforward: validate Windows EventID 4688 (process creation) telemetry in Splunk and build analyst-ready process pivot views.

The first query returned 41 grouped process rows. The top entry by volume was splunk-powershell.exe from the Universal Forwarder - infrastructure noise, not analyst signal. That was expected and filtered first.

After removing forwarder self-activity, the next highest-volume entry was this:

NewProcessName:   pwsh.exe
ParentProcessName: [unexpected deep path]
Count:            1,840

pwsh.exe being spawned nearly 2,000 times in a single collection window, from a non-standard parent, with no obvious origin. That is the exact pattern a LOLBin abuse detection rule fires on.

Living-off-the-Land Binary (LOLBin) abuse is one of the most common attacker techniques for execution and defense evasion. The standard pattern:

• A non-standard process spawns powershell.exe or pwsh.exe
• High spawn volume across a short window
• Parent process name is unexpected or obscure
• No corresponding user activity to explain the volume

This hit all four criteria. The parent process was not a known Windows system process, not an application the analyst recognized by name, and the spawn count was high enough to trigger any reasonable threshold rule.

At this point the correct analyst move is not to dismiss it. It is to pull the full parent process path and trace the binary to a known, verifiable origin.

Step 1 - Pull the full parent process path.

The abbreviated display name in the grouped stats view was not enough to classify. The investigation pivoted to the raw field to get the complete executable path:

index=windows sourcetype=XmlWinEventLog:Security
| rex field=_raw "<EventID>(?<EventID>\d+)</EventID>"
| rex field=_raw "<Data Name='NewProcessName'>(?<NewProcessName>[^<]+)</Data>"
| rex field=_raw "<Data Name='ParentProcessName'>(?<ParentProcessName>[^<]+)</Data>"
| search EventID=4688 NewProcessName="*pwsh.exe"
| stats count by NewProcessName ParentProcessName
| sort - count

Step 2 - Evaluate the full path.

The parent process resolved to the full executable path of the OpenAI Codex CLI tool - a Node.js-based AI coding assistant installed via npm:

[npm_modules_path]\@openai\codex\...\codex-win32-x64\vendor\...\codex.exe

This is a known, versioned, published software package. The path is deterministic - it lives inside the npm module tree under a user's AppData directory, not in a temp folder, system directory, or path used by commodity malware loaders.

Step 3 - Correlate spawn count to tool behavior.

The Codex CLI operates by spawning pwsh.exe to execute shell commands on behalf of the AI agent. Each tool call - file write, git command, script run - generates one or more process creation events. A coding session running dozens of operations over hours produces exactly this kind of volume.

The pattern was also present in the secondary spawner chain:

codex.exe -> codex-windows-sandbox-setup.exe  (465 events)
codex-command-runner -> pwsh.exe                          (407 events)
codex.exe -> git.exe                           (427 events)

This is the normal execution tree for an AI coding tool running in sandboxed mode. The whole process family traces back to a single known origin.

Classification decision: expected tool behavior.

The determination was not "this looks safe so I'll ignore it." It was based on three verifiable criteria:

1. Parent binary path is deterministic and traces to a known published package
2. Spawn pattern matches the documented execution model of that tool
3. No indicators of compromise in adjacent process or network activity

After classification, the analyst view was rebuilt with forwarder self-activity and the verified AI tool process family excluded. The before/after comparison:

The resulting view shows analyst-relevant process behavior: expected system activity and user applications. The AI tool process family is excluded by a documented, reproducible filter - not silently dropped.

The exclusion logic is explicit and reviewable:

| search NOT (NewProcessName="*SplunkUniversalForwarder*"
         OR ParentProcessName="*SplunkUniversalForwarder*")
| search NOT (ParentProcessName="*\\@openai\\codex\\*"
         OR ParentProcessName="*codex-command-runner*"
         OR ParentProcessName="*codex-windows-sandbox-setup*")

The filter is scoped to known, versioned tool paths. It does not suppress all pwsh.exe spawns from non-standard parents - only the ones that trace to the specific verified origin. A real malicious spawner would not match this path and would remain visible.

AI-assisted coding tools are entering enterprise environments now. Security teams writing detection rules for LOLBin abuse - rules that correctly fire on any non-standard parent spawning PowerShell - will start seeing this pattern in production logs.

If the analyst has not encountered this process family before, the alert looks real. The volume looks suspicious. The parent name is unfamiliar. It will generate false positives, alert fatigue, or a missed escalation depending on how the rule is tuned.

The detection engineering response is not "add it to a global exclusion list." It is:

1. Identify the full binary path of the spawner
2. Confirm it traces to a known, versioned software package
3. Build a scoped exclusion that targets the specific tool path, not the behavior class
4. Document the decision so the next analyst can trace it

That is the same reasoning used to tune any enterprise detection rule when a legitimate tool matches a malicious pattern. The fact that the tool is an AI coding assistant makes it novel. The triage method is the same.

Base grouped query (all process creation, with noise):

index=windows sourcetype=XmlWinEventLog:Security
| rex field=_raw "<EventID>(?<EventID>\d+)</EventID>"
| rex field=_raw "<Data Name='NewProcessName'>(?<NewProcessName>[^<]+)</Data>"
| rex field=_raw "<Data Name='ParentProcessName'>(?<ParentProcessName>[^<]+)</Data>"
| search EventID=4688
| stats count by NewProcessName ParentProcessName
| sort - count

Analyst view (forwarder and verified AI tool excluded):

index=windows sourcetype=XmlWinEventLog:Security
| rex field=_raw "<EventID>(?<EventID>\d+)</EventID>"
| rex field=_raw "<Data Name='NewProcessName'>(?<NewProcessName>[^<]+)</Data>"
| rex field=_raw "<Data Name='ParentProcessName'>(?<ParentProcessName>[^<]+)</Data>"
| search EventID=4688
| search NOT (NewProcessName="*SplunkUniversalForwarder*"
         OR ParentProcessName="*SplunkUniversalForwarder*")
| search NOT (ParentProcessName="*\\@openai\\codex\\*"
         OR ParentProcessName="*codex-command-runner*"
         OR ParentProcessName="*codex-windows-sandbox-setup*"
         OR NewProcessName="*codex-windows-sandbox-setup*")
| stats count by NewProcessName ParentProcessName
| sort - count

Parsing uses rex field extraction from XmlWinEventLog:Security raw XML. The forwarder exclusion removes collector self-activity. The AI tool exclusion removes a verified-benign process family by specific executable path, not by behavior class. Raw evidence CSVs available in the lab evidence register.

The finding is not that AI coding tools are dangerous. The finding is that they produce process trees that match the signature of known-malicious behavior, and that is going to be a real analyst problem as these tools proliferate in development environments.

The correct response is the same one used for any false-positive triage: trace the binary, verify the origin, build a scoped exclusion, document the decision. The tooling is new. The methodology is not.

The specific exclusion path above is a working artifact. Any environment running the same AI coding tool will see the same spawn pattern in their 4688 data.