anti-debugging-techniques

A security research playbook for detecting and bypassing anti-debugging protections encountered when reverse engineering protected binaries on Linux and Windows. It is intended for loading when a target detects debuggers through mechanisms such as ptrace, PEB flags, timing checks, or signal and exception handlers, and it pairs each detection technique with corresponding bypass strategies.

On Linux it covers the classic ptrace self-attach, reading TracerPid from /proc/self/status, rdtsc and clock timing checks, SIGTRAP signal-based detection, scanning /proc/self/maps for injected libraries, and environment-variable checks. On Windows it covers IsDebuggerPresent and CheckRemoteDebuggerPresent, the relevant PEB flag fields and their debugged values, NtQueryInformationProcess info classes, hardware breakpoint detection via debug registers, INT 2D and UD2 exception tricks, TLS callbacks that run before main, NtSetInformationThread with ThreadHideFromDebugger, and VEH-based checks. It also addresses advanced multi-layer schemes such as fork-and-ptrace self-debugging and mutual multi-process watching.

Bypasses are practical and technique-specific: LD_PRELOAD shims, binary patching and NOPing, GDB syscall catches and signal handling, Frida hooks, and hooking or zeroing the relevant Windows APIs and PEB fields. Quick-pick tables map each detection class to a first and backup bypass, and the document points to a companion ANTI_DEBUG_MATRIX.md for a full cross-reference matrix, reliability ratings, false-positive notes, and tool compatibility across GDB, x64dbg, WinDbg, Frida, and ScyllaHide. The material emphasizes the distinction between user-mode and kernel-mode detection and the correct patching strategy for each. Use it as a reference during malware analysis, software protection assessment, and reverse-engineering engagements.