PyTorch 2.13 Release Blog – PyTorch

Featured projects

• 
  

We are excited to announce the release of PyTorch® 2.13 (release notes)!

The PyTorch 2.13 release features the following changes:

•   FlexAttention lands on Apple Silicon (MPS), with up to ~12x speedup over SDPA on sparse patterns, and gains a deterministic backward path on CUDA for reproducible gradient computation 
•   CuTeDSL “Native DSL” backend gives Inductor a second high-performance code path (alongside Triton) for key GPU operations, with faster compilation
•   nn.LinearCrossEntropyLoss combines final prediction and loss computation operations to cut peak GPU memory by up to 4x for large-vocabulary language model training
•   torchcomms, a new communications backend for PyTorch Distributed, improves fault tolerance, scalability, and debuggability for large-cluster training
•   FSDP2 now overlaps reduce-scatter and all-gather communications via a dedicated process group (opt-in), increasing distributed training throughput
•   Python 3.15 wheel support for Torch on Linux via the pytorch repository index. This support includes builds compatible with free-threaded 3.15t. 
•   Broader platform support: ROCm gains AOTriton 0.12b with native HIP CMake, Arm adds Armv9-A torch.compile targeting, and Intel XPU exposes new device telemetry APIs

This release is composed of 3,328 commits from 526 contributors since PyTorch 2.12. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.13. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page.

Have questions? Join us on July 22, 2026, at 11 a.m. PT  for a live Q&A with panelists Alban Desmaison, Andrey Talman, Piotr Bialecki and moderator Chris Gottbrath. We will provide a brief overview of the release and answer your questions live. Register today.

Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. PyTorch 2.11 introduced differentiable collectives for distributed training and FlashAttention-4 on next-generation GPUs. PyTorch 2.12 added a device-agnostic torch.accelerator.Graph API, up to 100× faster batched eigendecomposition, and Microscaling quantization export support.

PyTorch 2.13 pushes performance further across platforms and scales: FlexAttention lands on Apple Silicon with up to 12× speedups, CuTeDSL brings CUTLASS-grade GEMM kernels to Inductor, and fused nn.LinearCrossEntropyLoss cuts peak memory up to 4x for large-vocabulary models. On the distributed side, a new torchcomms backend improves fault tolerance and debuggability at cluster scale, and FSDP2 unlocks communication overlap between all-gather and reduce-scatter for higher training throughput. This release also marks ExecuTorch’s integration into PyTorch Core, making on-device inference a first-class capability of the framework.

Performance Improvements

Deterministic Backward for FlexAttention Flash Backend

This improvement focuses on correctness and debugging for the existing CUDA implementation of FlexAttention by making gradient computation reproducible. By default, the FlexAttention flash backend uses atomic operations in the backward pass for dQ accumulation, which introduces non-determinism — repeated runs on the same input can produce slightly different gradients. This makes debugging, regression testing, and reproducible research difficult.

The new deterministic backward path (compute_dq_write_order) replaces atomics with a pre-computed write ordering that guarantees bit-for-bit reproducible gradients without meaningful performance penalty. The measured end-to-end overhead on create_block_mask is well under 1% at longer sequence lengths (e.g., +0.2% at S=32768), making determinism effectively free for most production workloads. Users can opt in via the existing torch.use_deterministic_algorithms(True) setting with no additional code changes. 

API Unstable

(PR #174813 by Driss Guessous, Meta)

Large MPS Op Migration to Native Metal

PyTorch’s MPS backend previously delegated most operations to Apple’s MPSGraph framework, which adds compilation and scheduling overhead per dispatch. For high-frequency, latency-sensitive operations this overhead can dominate execution time. This release migrates a broad set of operations to hand-written Metal compute kernels — copy/cast, uniform/normal/randint, comparisons, reductions (sum/mean), cumsum/cumprod, sort (multi-block and stable), embedding backward, and scatter/gather with bounds checking.

The native Metal path eliminates MPSGraph’s per-op compilation cost and gives PyTorch direct control over thread dispatch and memory access patterns, reducing kernel launch latency across common training and inference workloads on Apple Silicon.

API Unstable

(PR #184740 by Nikita Shulga, Meta, #185609 and #185119 by Irakli Salia, EPAM.)

FlexAttention on Apple Silicon (MPS)

FlexAttention, PyTorch’s unified API for expressing custom attention patterns as plain Python functions compiled into fused kernels, is now available on Metal/MPS. The MPS implementation provides hand-written Metal kernels for both the sparse prefill and decode paths (including GQA and captured buffers). So instead of writing a custom CUDA kernel for every attention variant you need, FlexAttention will write a 2-line Python function, and the compiler builds a fast kernel for you automatically. 

The benchmark numbers are impressive for sparse masks. On long, sparse attention patterns the speedups over SDPA are substantial — e.g. on a 1×8×32768×64 shape with a 256-element sliding window (0.8% density), FlexAttention runs in ~35 ms vs ~431 ms for SDPA (~12.3x); a smaller 8192-length / 64-window case achieves ~4.15x. Dense patterns continue to favor SDPA, as expected. API Unstable
(PR #182552, #186215, and #181575 by Irakli Salia, EPAM)

Core Features

nn.LinearCrossEntropyLoss

In standard large-vocabulary training (e.g., language models with 100K+ token vocabularies), computing the cross-entropy loss requires materializing the entire logits matrix over the vocabulary, which can consume tens of gigabytes of GPU memory. nn.LinearCrossEntropyLoss (and the corresponding linear_cross_entropy functional) fuses the final linear projection and cross-entropy computation into a single module that processes the vocabulary dimension in chunks, never materializing the full logits matrix. This reduces peak memory by up to ~4x for large-vocabulary workloads while maintaining numerical equivalence with the unfused path. The implementation supports label smoothing, weight tying, and z-loss regularization out of the box, and integrates with torch.compile for further optimization. As a drop-in replacement for separate nn.Linear + nn.CrossEntropyLoss, adoption requires no other code changes. 

API Unstable

(See #172446, #172286, and #185852 by Pearu Peterson, OpenTeams.)

Load Safetensors Directly with torch.load

Safetensors has become a widely adopted format for distributing model weights (used by Hugging Face, Stability AI, and others) due to its memory-mapped loading and lack of arbitrary code execution risk. Previously, loading safetensors files required installing and importing a separate library. torch.load(“foo.safetensors”) now works natively, detecting the format automatically and returning tensors directly. This removes a dependency for common workflows and makes PyTorch a seamless drop-in for loading models distributed in safetensors format. 

API Unstable
(PR #170592 by Nikita Shulga, Meta)

Python 3.15 Binary Support – Release Engineering

PyTorch wheels now support Python 3.15, including the experimental free-threaded 3.15t build.  Important Note: Python 3.15 is currently in pre-release (beta), with the final stable release scheduled for October 2026. Support is limited to torch wheels only (torchvision is not built for 3.15 yet) and to Linux builds on x86_64 and aarch64, covering the CPU, CUDA, ROCm, and XPU variants. torch.compile is not yet supported on Python 3.15. There are no Windows or macOS 3.15 wheels for this python version in this release. 

Python 3.15 and 3.15t wheels are not published to PyPI — they are available to download only via download.pytorch.org, using any of the following commands:

  # CPU

  pip3 install torch –index-url

  # CUDA (substitute the CUDA version, e.g. cu126 / cu130)

  pip3 install torch –index-url

  # ROCm (substitute the ROCm version)

  pip3 install torch –index-url

  # XPU

  pip3 install torch –index-url

 The same commands install the free-threaded 3.15t build when run under a free-threaded interpreter.

API Unstable. For updates See tracker issue: #184352
(PR #182954  by Nikita Shulga, PR #184600 and #186244 by Andrey Talman, Meta, #186017 by Rob Timpe, OpenTeams)

Distributed Training

torchcomms Backend

PyTorch Distributed has historically relied on c10d’s ProcessGroup abstraction for collective communications (all-reduce, all-gather, etc.), which was designed primarily around NCCL. As distributed training grows more complex (multi-dimensional parallelism, elastic scaling, heterogeneous interconnects), the original backend’s error handling and observability limitations become bottlenecks for operations teams.

torchcomms is a new communications backend integrated into PyTorch Distributed’s CI and device-mesh paths, providing improved fault tolerance (graceful timeout and partial-group recovery), better scalability across large clusters, and richer debuggability through structured logging and collective tracing. It serves as a modern alternative to the existing c10d backends while maintaining API compatibility. 

API Unstable

(PR #181662 by Tristan Rice, Meta, #178533 Pangiotis Kourdis, Intel, and #182057 by Kapil Sharma, Meta)

FSDP2 Separate Reduce-Scatter Group

In fully-sharded data-parallel (FSDP) training, all-gather and reduce-scatter share a single NCCL communicator by default. Because NCCL serializes operations on the same communicator, these two collectives cannot overlap, leaving communication bandwidth underutilized. FSDPModule.set_separate_reduce_scatter_group(enable=True) gives reduce-scatter its own dedicated NCCL communicator, allowing it to progress concurrently with all-gather operations. This enables AG/RS overlap, improving training throughput for fully-sharded workloads without any changes to model code.

API Unstable

 (PR #186335 by Wei Feng, Meta)

Compilation and Export

torch.compiler.set_default_backend

When using custom or out-of-tree compiler backends (e.g., for specialized hardware), users previously had to pass backend= explicitly to every torch.compile() call, making code verbose and error-prone. torch.compiler.set_default_backend mirrors the pattern established by torch.set_default_dtype and torch.set_default_device, letting backend authors or infrastructure teams set a process-wide default once. All subsequent torch.compile() calls automatically use that backend unless an explicit backend= argument overrides it. This simplifies adoption of custom backends across large codebases.

API Unstable

(PR #178944 by Angela Yi, Meta)

Platform Features and Updates

CUDA

CuTeDSL “Native DSL” Backend for Inductor

CuTeDSL is a Python-native domain-specific language built on NVIDIA’s CuTe (CUDA Templates) library, giving developers direct control over GPU tensor layouts, tiling strategies, and memory access patterns. PyTorch’s Inductor compiler can now use CuTeDSL as an alternative code-generation backend alongside Triton — specifically for matrix multiplication (GEMM) and normalization (RMSNorm), two of the most performance-critical operations in transformer training. These Quack-derived kernel overrides produce higher-quality matrix-multiply code without requiring Triton for these workloads. Kernel compilation has also moved from the thread pool to a subprocess pool, eliminating Python’s GIL bottleneck and improving compile-time parallelism. 

API Unstable
(PR #181267 by Michael Lazos, #182108 by Simon Layton, and #186310 by Driss Guessous, Meta)

ROCm

AOTriton 0.12b, Origami GEMM Selection, Native HIP CMake

The ROCm stack gains three improvements: AOTriton advances to 0.12b with support for asymmetric head dimensions, deterministic algorithms, and new GPU targets (gfx1100/gfx1151 promoted to stable, partial FlashAttention v3 on gfx950). The Origami tool replaces brute-force autotuning with analytical GEMM-configuration selection, cutting tune time. The build system now uses CMake’s native HIP language support.

API Unstable

(PR #184288 and #172512 by Xinya Zhang and Umesh Chand, AMD)

Arm

Armv9-A Target Support for torch.compile

torch.compile on AArch64 now recognizes Armv9-A CPUs (e.g., Neoverse V2 used in AWS Graviton4), propagating the correct target triple and feature set (128-bit and 256-bit SVE) through Inductor codegen. No behavior change for x86. 

API Unstable
(PR #184555 by Zhibo Li, Arm)

XPU (Intel GPUs)

Device Telemetry APIs

New query APIs for Intel GPUs expose runtime device state: memory usage (torch.xpu.device_memory_used), utilization, power draw, clock rate, and temperature, plus device-wide synchronization and hardware properties (last-level cache size, integrated-GPU detection).
API Unstable

(PR #183431, #183429, #183428, and #183427 by Guangye Yu, Intel)

C++ ABI

torch::stable::Generator

Custom kernel authors can now retrieve an at::Generator through the stable C++ ABI (previously always passed as null), enabling RNG support in ABI-stable extensions without recompilation against PyTorch internals. 

API Unstable

(PR #186423 and #183930 by Jane Xu, Meta and Chris Leonard, Red Hat) 

Profiling and Debugging

Experimental CUPTI Monitor Profiler

PyTorch’s existing profiler collects GPU activity through CUPTI’s activity API, which requires synchronization points that can distort timing and contend with the GIL in multi-threaded workloads. The new experimental CUPTI monitor backend collects GPU metrics asynchronously (completely off the GIL), eliminating profiling-induced overhead while reusing the existing CPU profiler path. The result is merged Chrome traces that accurately reflect real-world execution timing, with GPU-side user annotations automatically recreated for record_function scopes. This makes it practical to profile production training loops without measurably perturbing their performance.

API Unstable

(PR #186037 and #186295 by Natalia Gimelshein, Meta)

CUDAGraph.get_graph_data()

When debugging performance issues in CUDA graph captures, understanding the internal structure (which kernels run, their dependencies, execution order) previously required external tools or manual inspection. CUDAGraph.get_graph_data() exposes the full graph topology programmatically: node types, kernel names, dependency edges, and IDs remapped to match CUPTI profiler output. This lets developers correlate captured graph structure with profiled kernel launches, making it straightforward to identify bottlenecks, unnecessary serialization, or suboptimal kernel fusion within a captured graph.

API Unstable

(PR #183165 by Natalia Gimelshein, Meta)

Deprecations and Backwards-Incompatible Changes

• Named tensors removed. The deprecated named-tensor feature (Tensor.names and associated APIs) has been hard-removed to cut overhead and code bloat. See #173895.
• Distributed collectives renamed to a _single scheme. all_gather_into_tensor → all_gather_single and reduce_scatter_tensor → reduce_scatter_single, aligning with torchcomms. The old names remain as thin wrappers marked deprecated via FutureWarning. See #186123.
• Bazel build removed. The Bazel build was never broadly adopted and depended on an antiquated Bazel 6; it has been removed. See #180883.

Non-Feature Updates

• Python support: CPython 3.13t dropped from the Linux binary matrix. (Preliminary Python 3.15 and 3.15t binary support is found above.) See #182951.
• CUDA: CUDA 13.0 remains the default build; small wheels are now always built for CUDA Linux and the CUDA 12.8/12.9 builds were removed; ptxas is no longer bundled in the cu13 binary. See #180612, #174716.
• Triton: pin advanced to 3.7.1. See #186792.
• oneDNN: submodule upgraded to v3.12. See #181222.

Updated (7/8): Removed two instances of prototype which is a designation that we no longer use.