How to Preserve Forensic Evidence on a Network Appliance Before You Patch It

The instinct when a critical patch drops is to apply it immediately. For a workstation that is right. For an internet-facing appliance that has been exposed to a zero-dayZero-Day🛡️A security vulnerability that is exploited or publicly disclosed before the software vendor can release a patch, giving developers 'zero days' to fix it., it is often the single most destructive thing you can do to your own investigation. A firmwareFirmware🏠Permanent software programmed into a device's hardware that controls its basic functions. upgrade rewrites the system partition, restarts the packet engine, clears process memory, and frequently rotates log files. If the box was compromised, you have just erased the proof and any chance of learning what the attacker took. CISA's BOD 26-04BOD 26-04🛡️CISA Binding Operational Directive 26-04, issued 10 June 2026, which replaced BOD 22-01. It assigns federal civilian agencies remediation deadlines of 3, 14, or 60 days based on asset exposure, KEV status, exploit automation, and post-exploitation impact, and requires forensic triage for the 3-day tier. makes forensic triageForensic Triage🛡️A rapid, structured assessment of whether a system was compromised, performed alongside patching rather than after it. It prioritizes evidence the attacker could not have altered, such as network egress and authentication logs on other systems, over the affected host's own logs. a formal requirement for the vulnerabilities it flags, and the September 2026 NetScaler zero-days CVE-2026-88771 and CVE-2026-88772 carried that flag. This guide is the procedure for capturing what you need before the upgrade destroys it, built around Citrix's own compromise-response article but applicable to any appliance.

Why Patching Destroys Evidence

Appliances are not servers with a package manager. A version upgrade typically replaces the entire operating environment, and the vendor's design goal is a clean known state, not evidence preservation. Three categories of evidence are at risk. Volatile memory holds running processes, injected code, open network connections, and decrypted session material; a reboot removes all of it. Local logs are often size-capped and rotate quickly on a busy gateway, so the window that covers the intrusion may already be closing. And filesystem artifacts such as web shells, modified binaries, and attacker-created accounts may live in paths the upgrade overwrites.

There is a second, subtler loss. Attackers who compromise a gateway frequently harvest the credentials and session tokens flowing through it. If you patch without looking, you close the door but leave the attacker holding every key that was inside. The investigation is what tells you which keys to change. Skip it and you are guessing.

Step 1: Decide the Isolation Posture

Before touching anything, decide whether the appliance stays connected during the capture. Citrix's guidance is to remove a suspected-compromised NetScaler from the network to prevent further unauthorized access, and that is the right default when exploitation is confirmed. Isolating means blocking traffic to and from the appliance at the upstream firewallFirewall🌐Security system that monitors and controls network traffic based on predetermined rules. or switch, not powering it off. Powering off loses memory. Pulling the cable loses nothing and stops the attacker's session. The strategic question of who is allowed to make that call, and when, is its own topic; we cover it in why pulling a perimeter appliance offline is a legitimate zero-day response.

Step 2: Record the Clock

Write down the appliance's current system time, its configured time zone, and its NTP servers before you do anything else. Citrix lists this explicitly, and it is the step most teams skip. Every timestamp in every log you collect will need to be correlated against firewall logs, authentication logs, and endpoint telemetry from other systems. If the appliance's clock was drifting, or set to UTC while your SIEM assumes local time, you will misattribute events by hours. Capture the offset now while the box is in its original state.

Step 3: Capture Volatile State

For a virtual appliance such as a NetScaler VPX, take a hypervisorHypervisor🌐Software that creates and manages virtual machines by allocating physical hardware resources among multiple guest operating systems. VMware ESXi is a Type 1 (bare-metal) hypervisor. snapshot that includes memory. This is the highest-value single action available, because it preserves the running state completely and can be analyzed offline without disturbing the original. Name the snapshot with the date and the word "forensic" so nobody deletes it during cleanup.

For physical hardware, Citrix's guidance is to generate a core file of the packet engine, which produces a memory dump at the cost of a warm restart of that process. The resulting files land under the highest-numbered directory beneath `/var/core/` and carry an `NSPPE-` prefix, compressed with gzip. Copy them off the appliance immediately. Then generate the technical support bundle, which captures configuration, running processes, and a set of logs in one archive. Do the core dumpCore Dump🛡️A file containing the memory contents of a process or system at a point in time, generated on a crash or on demand. In incident response a core dump of an appliance's packet engine preserves injected code, open connections, and session data that a reboot or upgrade would erase. before the support bundle, since the bundle collection itself changes process state.

Step 4: Secure the Logs

Collect logs from three places, because they will not agree. The appliance's local logs are the most detailed but the most fragile. A remote syslog server, if you have one, holds a copy that the attacker could not easily alter from the appliance and that predates any log rotation. The management platform, NetScaler Console in Citrix's case, holds a third view including configuration change history and any indicator scans that were run. Pull all three into your evidence store, hash them, and record the hashes. If you have never configured remote syslog for your perimeter appliances, that is the first change to make once this incident is over.

Step 5: Image Physical Disks

For MPX and SDX hardware, Citrix's procedure after memory capture is to power down, remove the physical disks, and create bit-for-bit images using a hardware write-blocker. Keep two copies, one for analysis and one untouched, and document the chain of custodyChain of Custody🛡️A documented record of who handled a piece of evidence, when, and how, from the moment it was collected until it is presented. Disk images and logs taken from a compromised device without chain-of-custody records may be unusable in legal or regulatory proceedings.: who handled the disks, when, and where they were stored. This step is where most organizations discover they do not own a write-blocker. If law enforcement involvement is possible, Citrix advises consulting legal counsel before rebuilding, because an imaged disk that was handled without documentation may not be usable as evidence.

Step 6: Run the Vendor's Indicator Check

With evidence captured, run whatever indicator-of-compromise check the vendor provides. For NetScaler, that is the IOC scan on the Console security advisory page on 14.1-73.36 and later with telemetry enabled, or indicators obtained from Citrix Support. Treat the result as one input, not a verdict. Citrix itself warns that the indicators do not cover every technique and that a clean result is not proof the appliance was not compromised. Cross-check against the network indicators published by incident responders; Truesec listed three command-and-control addresses for the NetScaler campaign, and an egress-log search for those from the appliance's subnet IP takes minutes.

Step 7: Now Upgrade, Then Rotate

Only now apply the fixed build. Read the vendor's upgrade notes for the branch you are on; watchTowr flagged that 13.1 appliances with configured variables should use 13.1-64.24 rather than 13.1-64.23 to avoid a reboot loop. After the upgrade, rotate everything the appliance held: local account passwords, key encryptionEncryption🛡️The process of converting data into a coded format that can only be read with the correct decryption key. keys, the service accounts used for LDAP and RADIUSRADIUS🌐Remote Authentication Dial-In User Service, the protocol network devices use to ask a central server whether a user or device should be granted access and with what attributes. Each device shares a secret with the server, so a compromised server exposes the secrets of every device that trusts it. binds, and every certificate and private key. If the investigation found evidence of compromise, Citrix's guidance is to rebuild from clean firmware, restore a backup that predates the intrusion, and rotate the restored secrets a second time, then monitor for at least ninety days.

Making It Repeatable

None of this is difficult, but all of it is slow when you are inventing it at two in the morning. Write the procedure down per appliance model, with the exact paths and the name of the person who owns the write-blocker. Pre-stage a storage location for evidence with enough space for a memory image. And where a mitigation short of patching exists, such as disabling the DTLS listener that CVE-2026-88772 depends on, apply it first so the capture can proceed without the attacker still inside. Our explainer on how DTLS works and why it widens a VPN gateway's attack surfaceAttack Surface🛡️The sum of all points where an unauthorized user could attempt to enter or extract data from a system: exposed services, interfaces, accounts, and integrations. Reducing attack surface means removing reachability, not just patching. covers that particular case.