aoti-debug

AOTI Debugging Guide is a diagnostic playbook for AOTInductor, PyTorch's ahead-of-time Inductor compilation path. It targets the failures teams hit when compiling, packaging, and loading models with aot_compile, aot_load, aoti_compile_and_package, or aoti_load_package: segfaults, device-mismatch errors, constant-loading crashes, and other runtime errors. The guide opens by routing specific error signatures (such as a Triton index-out-of-bounds assertion) to dedicated sub-guides, then falls back to a structured general procedure.

The first and most emphasized step for any AOTI error is to verify device and shape matching: the model must be loaded on the same device type it was compiled on, runtime input devices must match the compile device, and input shapes must match those used at compile time or satisfy dynamic-shape constraints. Because AOTI does not support cross-device loading (compile on CUDA, load on CUDA; compile on CPU, load on CPU, with only the device index allowed to differ), many segfaults and wrong-output bugs trace back to a device-type mismatch. From there it walks through CUDA illegal-memory-access debugging with a sequence of environment flags: AOTI_RUNTIME_CHECK_INPUTS and TORCHINDUCTOR_NAN_ASSERTS for sanity checks, PYTORCH_NO_CUDA_MEMORY_CACHING and CUDA_LAUNCH_BLOCKING to make non-deterministic errors reproducible, and the intermediate-value printer to isolate the offending kernel.

Use it whenever an AOTI compile or load step crashes or produces incorrect results, especially in dynamic-shape or custom-op scenarios that are historically frequent sources of illegal memory access. The guide also lists logging and tracing tools (tlparse/TORCH_TRACE, TORCH_LOGS, TORCH_SHOW_CPP_STACKTRACES) and clarifies the API landscape: the older torch._export.aot_compile and aot_load entry points are deprecated in favor of torch._inductor.aoti_compile_and_package and aoti_load_package, which store device metadata in the package so the correct device type is selected automatically. A summary table maps each environment variable to when it applies and what it does.