How to Preserve Forensic Evidence on a Network Appliance Before You Patch It
A firmware upgrade erases memory, rotates logs, and overwrites the artifacts that prove a breach. A step-by-step capture procedure for appliances, built on CISA's forensic-triage requirement.
The instinct when a critical patch drops is to apply it immediately. For a workstation that is right. For an internet-facing appliance that has been exposed to a zero-day, it is often the single most destructive thing you can do to your own investigation. A firmware upgrade rewrites the system partition, restarts the packet engine, clears process memory, and frequently rotates log files. If the box was compromised, you have just erased the proof and any chance of learning what the attacker took. CISA's BOD 26-04 makes forensic triage a formal requirement for the vulnerabilities it flags, and the September 2026 NetScaler zero-days CVE-2026-88771 and CVE-2026-88772 carried that flag. This guide is the procedure for capturing what you need before the upgrade destroys it, built around Citrix's own compromise-response article but applicable to any appliance.
Why Patching Destroys Evidence
Appliances are not servers with a package manager. A version upgrade typically replaces the entire operating environment, and the vendor's design goal is a clean known state, not evidence preservation. Three categories of evidence are at risk. Volatile memory holds running processes, injected code, open network connections, and decrypted session material; a reboot removes all of it. Local logs are often size-capped and rotate quickly on a busy gateway, so the window that covers the intrusion may already be closing. And filesystem artifacts such as web shells, modified binaries, and attacker-created accounts may live in paths the upgrade overwrites.
There is a second, subtler loss. Attackers who compromise a gateway frequently harvest the credentials and session tokens flowing through it. If you patch without looking, you close the door but leave the attacker holding every key that was inside. The investigation is what tells you which keys to change. Skip it and you are guessing.
Step 1: Decide the Isolation Posture
Before touching anything, decide whether the appliance stays connected during the capture. Citrix's guidance is to remove a suspected-compromised NetScaler from the network to prevent further unauthorized access, and that is the right default when exploitation is confirmed. Isolating means blocking traffic to and from the appliance at the upstream firewall or switch, not powering it off. Powering off loses memory. Pulling the cable loses nothing and stops the attacker's session. The strategic question of who is allowed to make that call, and when, is its own topic; we cover it in why pulling a perimeter appliance offline is a legitimate zero-day response.
Step 2: Record the Clock
Write down the appliance's current system time, its configured time zone, and its NTP servers before you do anything else. Citrix lists this explicitly, and it is the step most teams skip. Every timestamp in every log you collect will need to be correlated against firewall logs, authentication logs, and endpoint telemetry from other systems. If the appliance's clock was drifting, or set to UTC while your SIEM assumes local time, you will misattribute events by hours. Capture the offset now while the box is in its original state.
Step 3: Capture Volatile State
For a virtual appliance such as a NetScaler VPX, take a hypervisor snapshot that includes memory. This is the highest-value single action available, because it preserves the running state completely and can be analyzed offline without disturbing the original. Name the snapshot with the date and the word "forensic" so nobody deletes it during cleanup.
For physical hardware, Citrix's guidance is to generate a core file of the packet engine, which produces a memory dump at the cost of a warm restart of that process. The resulting files land under the highest-numbered directory beneath /var/core/ and carry an NSPPE- prefix, compressed with gzip. Copy them off the appliance immediately. Then generate the technical support bundle, which captures configuration, running processes, and a set of logs in one archive. Do the core dump before the support bundle, since the bundle collection itself changes process state.
Step 4: Secure the Logs
Collect logs from three places, because they will not agree. The appliance's local logs are the most detailed but the most fragile. A remote syslog server, if you have one, holds a copy that the attacker could not easily alter from the appliance and that predates any log rotation. The management platform, NetScaler Console in Citrix's case, holds a third view including configuration change history and any indicator scans that were run. Pull all three into your evidence store, hash them, and record the hashes. If you have never configured remote syslog for your perimeter appliances, that is the first change to make once this incident is over.
Step 5: Image Physical Disks
For MPX and SDX hardware, Citrix's procedure after memory capture is to power down, remove the physical disks, and create bit-for-bit images using a hardware write-blocker. Keep two copies, one for analysis and one untouched, and document the chain of custody: who handled the disks, when, and where they were stored. This step is where most organizations discover they do not own a write-blocker. If law enforcement involvement is possible, Citrix advises consulting legal counsel before rebuilding, because an imaged disk that was handled without documentation may not be usable as evidence.
Step 6: Run the Vendor's Indicator Check
With evidence captured, run whatever indicator-of-compromise check the vendor provides. For NetScaler, that is the IOC scan on the Console security advisory page on 14.1-73.36 and later with telemetry enabled, or indicators obtained from Citrix Support. Treat the result as one input, not a verdict. Citrix itself warns that the indicators do not cover every technique and that a clean result is not proof the appliance was not compromised. Cross-check against the network indicators published by incident responders; Truesec listed three command-and-control addresses for the NetScaler campaign, and an egress-log search for those from the appliance's subnet IP takes minutes.
Step 7: Now Upgrade, Then Rotate
Only now apply the fixed build. Read the vendor's upgrade notes for the branch you are on; watchTowr flagged that 13.1 appliances with configured variables should use 13.1-64.24 rather than 13.1-64.23 to avoid a reboot loop. After the upgrade, rotate everything the appliance held: local account passwords, key encryption keys, the service accounts used for LDAP and RADIUS binds, and every certificate and private key. If the investigation found evidence of compromise, Citrix's guidance is to rebuild from clean firmware, restore a backup that predates the intrusion, and rotate the restored secrets a second time, then monitor for at least ninety days.
Making It Repeatable
None of this is difficult, but all of it is slow when you are inventing it at two in the morning. Write the procedure down per appliance model, with the exact paths and the name of the person who owns the write-blocker. Pre-stage a storage location for evidence with enough space for a memory image. And where a mitigation short of patching exists, such as disabling the DTLS listener that CVE-2026-88772 depends on, apply it first so the capture can proceed without the attacker still inside. Our explainer on how DTLS works and why it widens a VPN gateway's attack surface covers that particular case.