diff --git a/_downloads/662999063954282841dc90b8945f85ce/tutorials_jupyter.zip b/_downloads/662999063954282841dc90b8945f85ce/tutorials_jupyter.zip
index ec4554643..184dfcc42 100644
Binary files a/_downloads/662999063954282841dc90b8945f85ce/tutorials_jupyter.zip and b/_downloads/662999063954282841dc90b8945f85ce/tutorials_jupyter.zip differ
diff --git a/_downloads/763344228ae6bc253ed1a6cf586aa30d/tutorials_python.zip b/_downloads/763344228ae6bc253ed1a6cf586aa30d/tutorials_python.zip
index a5b6a97d1..eea2bf9a4 100644
Binary files a/_downloads/763344228ae6bc253ed1a6cf586aa30d/tutorials_python.zip and b/_downloads/763344228ae6bc253ed1a6cf586aa30d/tutorials_python.zip differ
diff --git a/_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb b/_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb
index a63d64e3a..0fabbe9d0 100644
--- a/_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb
+++ b/_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb
@@ -22,14 +22,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Motivations\nMatrix multiplications are a key building block of most modern high-performance computing systems.\nThey are notoriously hard to optimize, hence their implementation is generally done by\nhardware vendors themselves as part of so-called \"kernel libraries\" (e.g., cuBLAS).\nUnfortunately, these libraries are often proprietary and cannot be easily customized\nto accomodate the needs of modern deep learning workloads (e.g., fused activation functions).\nIn this tutorial, you will learn how to implement efficient matrix multiplications by\nyourself with Triton, in a way that is easy to customize and extend.\n\nRoughly speaking, the kernel that we will write will implement the following blocked\nalgorithm to multiply a (MxK) by a (KxN) matrix:\n\n .. code-block:: python\n\n # do in parallel\n for m in range(0, M, BLOCK_SIZE_M):\n # do in parallel\n for n in range(0, N, BLOCK_SIZE_N):\n acc = zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=float32)\n for k in range(0, K, BLOCK_SIZE_K):\n a = A[m : m+BLOCK_SIZE_M, k : k+BLOCK_SIZE_K]\n b = B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]\n acc += dot(a, b)\n C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;\n\nwhere each iteration of the doubly-nested for-loop corresponds to a Triton program instance.\n\n"
+ "## Motivations\nMatrix multiplications are a key building block of most modern high-performance computing systems.\nThey are notoriously hard to optimize, hence their implementation is generally done by\nhardware vendors themselves as part of so-called \"kernel libraries\" (e.g., cuBLAS).\nUnfortunately, these libraries are often proprietary and cannot be easily customized\nto accomodate the needs of modern deep learning workloads (e.g., fused activation functions).\nIn this tutorial, you will learn how to implement efficient matrix multiplications by\nyourself with Triton, in a way that is easy to customize and extend.\n\nRoughly speaking, the kernel that we will write will implement the following blocked\nalgorithm to multiply a (M, K) by a (K, N) matrix:\n\n .. code-block:: python\n\n # do in parallel\n for m in range(0, M, BLOCK_SIZE_M):\n # do in parallel\n for n in range(0, N, BLOCK_SIZE_N):\n acc = zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=float32)\n for k in range(0, K, BLOCK_SIZE_K):\n a = A[m : m+BLOCK_SIZE_M, k : k+BLOCK_SIZE_K]\n b = B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]\n acc += dot(a, b)\n C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;\n\nwhere each iteration of the doubly-nested for-loop is performed by a dedicated Triton program instance.\n\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Compute Kernel\n\nThe above algorithm is, actually, fairly straightforward to implement in Triton.\nThe main difficulty comes from the computation of the memory locations at which blocks\nof :code:`A` and :code:`B` must be read in the inner loop. For that, we need\nmulti-dimensional pointer arithmetics.\n\n### Pointer Arithmetics\n\nFor a row-major 2D tensor :code:`X`, the memory location of :code:`X[i, j]` is given b\ny :code:`&X[i, j] = X + i*stride_x_0 + j*stride_x_1`.\nTherefore, blocks of pointers for :code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]` and\n:code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]` can be defined in pseudo-code as:\n\n .. code-block:: python\n\n &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = A + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);\n &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = B + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);\n\nWhich means that pointers for blocks of A and B can be initialized (i.e., :code:`k=0`) in Triton as:\n\n .. code-block:: python\n\n pid_m = triton.program_id(0)\n pid_n = triton.program_id(1)\n rm = pid_m * BLOCK_SIZE_M + triton.arange(0, BLOCK_SIZE_M)\n rn = pid_n * BLOCK_SIZE_N + triton.arange(0, BLOCK_SIZE_N)\n rk = triton.arange(0, BLOCK_SIZE_K)\n // pointer for A operand\n pa = A + (rm[:, None] * stride_a_0 + rk[None, :] * stride_a_1);\n // pointer for B operand\n pb = B + (rk[:, None] * stride_b_0 + rn[None, :] * stride_b_1);\n\nAnd then updated in the inner loop as follows:\n\n .. code-block:: python\n\n pa += BLOCK_SIZE_K * stride_a_1;\n pb += BLOCK_SIZE_K * stride_b_0;\n\n\n### L2 Cache Optimizations\n\nAs mentioned above, each program instance computes a :code:`[BLOCK_SIZE_M, BLOCK_SIZE_N]`\nblock of :code:`C`.\nIt is important to remember that the order in which these blocks are computed does\nmatter, since it affects the L2 cache hit rate of our program. and unfortunately, a\na simple row-major ordering\n\n .. code-block:: Python\n\n pid = triton.program_id(0);\n grid_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M;\n grid_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N;\n pid_m = pid / grid_n;\n pid_n = pid % grid_n;\n\nis just not going to cut it.\n\nOne possible solution is to launch blocks in an order that promotes data reuse.\nThis can be done by 'super-grouping' blocks in groups of :code:`GROUP_M` rows before\nswitching to the next column:\n\n .. code-block:: python\n\n pid = triton.program_id(0);\n width = GROUP_M * grid_n;\n group_id = pid // width;\n # we need to handle the case where M % (GROUP_M*BLOCK_SIZE_M) != 0\n group_size = min(grid_m - group_id * GROUP_M, GROUP_M);\n pid_m = group_id * GROUP_M + (pid % group_size);\n pid_n = (pid % width) // (group_size);\n\nFor example, in the following matmul where each matrix is 9 blocks by 9 blocks,\nwe can see that if we compute the output in row-major ordering, we need to load 90\nblocks into SRAM to compute the first 9 output blocks, but if we do it in grouped\nordering, we only need to load 54 blocks.\n .. image:: grouped_vs_row_major_ordering.png\n\nIn practice, this can improve the performance of our matrix multiplication kernel by\nmore than 10\\% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).\n\n\n"
+ "## Compute Kernel\n\nThe above algorithm is, actually, fairly straightforward to implement in Triton.\nThe main difficulty comes from the computation of the memory locations at which blocks\nof :code:`A` and :code:`B` must be read in the inner loop. For that, we need\nmulti-dimensional pointer arithmetics.\n\n### Pointer Arithmetics\n\nFor a row-major 2D tensor :code:`X`, the memory location of :code:`X[i, j]` is given b\ny :code:`&X[i, j] = X + i*stride_xi + j*stride_xj`.\nTherefore, blocks of pointers for :code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]` and\n:code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]` can be defined in pseudo-code as:\n\n .. code-block:: python\n\n &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = a_ptr + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);\n &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = b_ptr + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);\n\nWhich means that pointers for blocks of A and B can be initialized (i.e., :code:`k=0`) in Triton as:\n\n .. code-block:: python\n\n offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)\n offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)\n offs_k = tl.arange(0, BLOCK_SIZE_K)\n a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)\n b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)\n\nAnd then updated in the inner loop as follows:\n\n .. code-block:: python\n\n pa += BLOCK_SIZE_K * stride_ak;\n pb += BLOCK_SIZE_K * stride_bk;\n\n\n### L2 Cache Optimizations\n\nAs mentioned above, each program instance computes a :code:`[BLOCK_SIZE_M, BLOCK_SIZE_N]`\nblock of :code:`C`.\nIt is important to remember that the order in which these blocks are computed does\nmatter, since it affects the L2 cache hit rate of our program. and unfortunately, a\na simple row-major ordering\n\n .. code-block:: Python\n\n pid = triton.program_id(0);\n grid_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M;\n grid_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N;\n pid_m = pid / grid_n;\n pid_n = pid % grid_n;\n\nis just not going to cut it.\n\nOne possible solution is to launch blocks in an order that promotes data reuse.\nThis can be done by 'super-grouping' blocks in groups of :code:`GROUP_M` rows before\nswitching to the next column:\n\n .. code-block:: python\n\n # program ID\n pid = tl.program_id(axis=0)\n # number of program ids along the M axis\n num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)\n # number of programs ids along the N axis\n num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)\n # number of programs in group\n num_pid_in_group = GROUP_SIZE_M * num_pid_n \n # id of the group this program is in\n group_id = pid // num_pid_in_group \n # row-id of the first program in the group\n first_pid_m = group_id * GROUP_SIZE_M \n # if `num_pid_m` isn't divisible by `GROUP_SIZE_M`, the last group is smaller\n group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M) \n # *within groups*, programs are ordered in a column-major order\n # row-id of the program in the *launch grid*\n pid_m = first_pid_m + (pid % group_size_m)\n # col-id of the program in the *launch grid*\n pid_n = (pid % num_pid_in_group) // group_size_m\n\nFor example, in the following matmul where each matrix is 9 blocks by 9 blocks,\nwe can see that if we compute the output in row-major ordering, we need to load 90\nblocks into SRAM to compute the first 9 output blocks, but if we do it in grouped\nordering, we only need to load 54 blocks.\n .. image:: grouped_vs_row_major_ordering.png\n\nIn practice, this can improve the performance of our matrix multiplication kernel by\nmore than 10\\% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).\n\n\n"
]
},
{
@@ -47,7 +47,7 @@
},
"outputs": [],
"source": [
- "import torch\nimport triton\nimport triton.language as tl\n\n# %\n# :code:`triton.jit`'ed functions can be auto-tuned by using the `triton.autotune`\n# decorator, which consumes:\n# - A list of :code:`triton.Config` objects that define different configurations of\n# meta-parameters (e.g., BLOCK_SIZE_M) and compilation options (e.g., num_warps) to try\n# - An autotuning *key* whose change in values will trigger evaluation of all the\n# provided configs\n\n@triton.autotune(\n configs=[\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),\n triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),\n triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 64 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 32 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 32 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),\n triton.Config({'BLOCK_SIZE_M': 32 , 'BLOCK_SIZE_N': 64 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),\n ],\n key=['M', 'N', 'K'],\n)\n# %\n# We can now define our kernel as normal, using all the techniques presented above\n@triton.jit\ndef matmul_kernel(\n # Pointers to matrices\n a_ptr,\n b_ptr,\n c_ptr,\n # Matrix dimensions\n M,\n N,\n K,\n # The stride variables represent how much to increase the ptr by when moving by 1\n # element in a particular dimension. E.g. stride_am is how much to increase a_ptr\n # by to get the element one row down (A has M rows)\n stride_am,\n stride_ak,\n stride_bk,\n stride_bn,\n stride_cm,\n stride_cn,\n **meta,\n):\n \"\"\"Kernel for computing the matmul AB = C\n\n A has shape (M, K), B has shape (K, N) and C has shape (M, N)\n \"\"\"\n # extract meta-parameters\n BLOCK_SIZE_M = meta['BLOCK_SIZE_M']\n BLOCK_SIZE_N = meta['BLOCK_SIZE_N']\n BLOCK_SIZE_K = meta['BLOCK_SIZE_K']\n GROUP_SIZE_M = 8\n pid = tl.program_id(axis=0)\n\n # the number of blocks is the ceil(M / BLOCK_SIZE_M) since we need an extra block\n # Note that this will lead to some quantization in performance where time-taken jumps\n # when you need to add a new block\n n_blocks_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M\n n_blocks_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N\n\n # Map PIDs to the block they should compute. This is done in a grouped ordering\n # to promote L2 cache reuse.\n n_output_blocks_in_group = GROUP_SIZE_M * n_blocks_n\n group_id = pid // n_output_blocks_in_group\n first_m_block_in_group = group_id * GROUP_SIZE_M\n\n # If the number of blocks is not divisible by the group size, the last group is smaller\n group_size_m = min(n_blocks_m - first_m_block_in_group, GROUP_SIZE_M)\n\n # Within a group, we compute in col-major ordering, block_m and block_n are the\n # output row and col that this program is computing in terms of blocks\n block_m = first_m_block_in_group + (pid % group_size_m)\n block_n = (pid % n_output_blocks_in_group) // group_size_m\n\n # Convert from block indices back to element indices\n m_start = block_m * BLOCK_SIZE_M\n n_start = block_n * BLOCK_SIZE_N\n\n # Expand out to all the offsets for each of the elements in this block.\n m_offsets_a = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]\n n_offsets_b = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]\n k_offsets = tl.arange(0, BLOCK_SIZE_K)\n\n # Get the pointers for the first block of each. We will advance this pointer\n # as we move in the K direction and accumulate.\n # a_ptrs should contain BLOCK_SIZE_M * BLOCK_SIZE_K pointers\n a_ptrs = a_ptr + (stride_am * m_offsets_a + stride_ak * k_offsets[None, :])\n # b_ptrs should contain BLOCK_SIZE_K * BLOCK_SIZE_N pointers\n b_ptrs = b_ptr + (stride_bk * k_offsets[:, None] + stride_bn * n_offsets_b)\n # We accumulate internally in fp32, but the output is written out in the dtype\n # of the tensor when it is stored\n accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)\n for k in range(0, K, BLOCK_SIZE_K):\n # Note that for simplicity, we don't apply a mask here. This means that if K is\n # not a multiple of BLOCK_SIZE_K, this will access out-of-bounds memory and\n # accumulate it incorrectly.\n a = tl.load(a_ptrs)\n b = tl.load(b_ptrs)\n # We accumulate along the K dimension\n accumulator += tl.dot(a, b)\n\n # Advance the ptrs to the next K block\n a_ptrs += BLOCK_SIZE_K * stride_ak\n b_ptrs += BLOCK_SIZE_K * stride_bk\n # triton can accept arbitrary activation function via metaparameters!\n if meta['ACTIVATION']:\n accumulator = meta['ACTIVATION'](accumulator)\n\n m_offsets_c = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]\n n_offsets_c = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]\n c_ptrs = c_ptr + stride_cm * m_offsets_c + stride_cn * n_offsets_c\n mask = (m_offsets_c < M) & (n_offsets_c < N)\n tl.store(c_ptrs, accumulator, mask=mask)\n\n\n# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`\n@triton.jit\ndef leaky_relu(x):\n return tl.where(x >= 0, x, 0.01 * x)"
+ "import torch\nimport triton\nimport triton.language as tl\n\n# %\n# :code:`triton.jit`'ed functions can be auto-tuned by using the `triton.autotune`\n# decorator, which consumes:\n# - A list of :code:`triton.Config` objects that define different configurations of\n# meta-parameters (e.g., BLOCK_SIZE_M) and compilation options (e.g., num_warps) to try\n# - An autotuning *key* whose change in values will trigger evaluation of all the\n# provided configs\n\n@triton.autotune(\n configs=[\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),\n triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),\n triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 64 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 32 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),\n triton.Config({'BLOCK_SIZE_M': 64 , 'BLOCK_SIZE_N': 32 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),\n triton.Config({'BLOCK_SIZE_M': 32 , 'BLOCK_SIZE_N': 64 , 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),\n ],\n key=['M', 'N', 'K'],\n)\n# %\n# We can now define our kernel as normal, using all the techniques presented above\n@triton.jit\ndef matmul_kernel(\n # Pointers to matrices\n a_ptr, b_ptr, c_ptr,\n # Matrix dimensions\n M, N, K,\n # The stride variables represent how much to increase the ptr by when moving by 1\n # element in a particular dimension. E.g. stride_am is how much to increase a_ptr\n # by to get the element one row down (A has M rows)\n stride_am, stride_ak,\n stride_bk, stride_bn,\n stride_cm, stride_cn,\n # Meta-parameters\n **meta,\n):\n \"\"\"Kernel for computing the matmul C = A x B.\n A has shape (M, K), B has shape (K, N) and C has shape (M, N)\n \"\"\"\n # extract meta-parameters\n BLOCK_SIZE_M = meta['BLOCK_SIZE_M']\n BLOCK_SIZE_N = meta['BLOCK_SIZE_N']\n BLOCK_SIZE_K = meta['BLOCK_SIZE_K']\n GROUP_SIZE_M = 8\n\n # -----------------------------------------------------------\n # Map program ids `pid` to the block of C it should compute.\n # This is done in a grouped ordering to promote L2 data reuse\n # See above `L2 Cache Optimizations` section for details\n pid = tl.program_id(axis=0)\n num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)\n num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)\n num_pid_in_group = GROUP_SIZE_M * num_pid_n \n group_id = pid // num_pid_in_group \n first_pid_m = group_id * GROUP_SIZE_M \n group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M) \n pid_m = first_pid_m + (pid % group_size_m)\n pid_n = (pid % num_pid_in_group) // group_size_m\n\n # ----------------------------------------------------------\n # Create pointers for the first blocks of A and B.\n # We will advance this pointer as we move in the K direction \n # and accumulate\n # a_ptrs is a block of [BLOCK_SIZE_M, BLOCK_SIZE_K] pointers\n # b_ptrs is a block of [BLOCK_SIZE_K, BLOCK_SIZE_n] pointers\n # see above `Pointer Arithmetics` section for details\n offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)\n offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)\n offs_k = tl.arange(0, BLOCK_SIZE_K)\n a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)\n b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)\n\n # -----------------------------------------------------------\n # Iterate to compute a block of the C matrix\n # We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block\n # of fp32 values for higher accuracy.\n # `accumulator` will be converted back to fp16 after the loop\n accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)\n for k in range(0, K, BLOCK_SIZE_K):\n # Note that for simplicity, we don't apply a mask here. \n # This means that if K is not a multiple of BLOCK_SIZE_K, \n # this will access out-of-bounds memory and produce an\n # error or (worse!) incorrect results.\n a = tl.load(a_ptrs)\n b = tl.load(b_ptrs)\n # We accumulate along the K dimension\n accumulator += tl.dot(a, b)\n # Advance the ptrs to the next K block\n a_ptrs += BLOCK_SIZE_K * stride_ak\n b_ptrs += BLOCK_SIZE_K * stride_bk\n # you can fuse arbitrary activation functions here\n # while the accumulator is still in FP32 !\n if meta['ACTIVATION']: \n accumulator = meta['ACTIVATION'](accumulator)\n c = accumulator.to(tl.float16)\n\n # -----------------------------------------------------------\n # Write back the block of the output matrix C\n offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)\n offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)\n c_ptrs = c_ptr + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]\n c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)\n tl.store(c_ptrs, c, mask=c_mask)\n\n\n# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`\n@triton.jit\ndef leaky_relu(x):\n return tl.where(x >= 0, x, 0.01 * x)"
]
},
{
@@ -65,7 +65,7 @@
},
"outputs": [],
"source": [
- "def matmul(a, b, activation=None):\n # checks constraints\n assert a.shape[1] == b.shape[0], \"incompatible dimensions\"\n assert a.is_contiguous(), \"matrix A must be contiguous\"\n assert b.is_contiguous(), \"matrix B must be contiguous\"\n M, K = a.shape\n K, N = b.shape\n assert (\n K % 32 == 0\n ), \"We don't check memory-out-of-bounds with K so K must be divisible by BLOCK_SIZE_K\"\n # allocates output\n c = torch.empty((M, N), device=a.device, dtype=a.dtype)\n # 1D launch kernel where each block gets its own program.\n grid = lambda META: (\n triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),\n )\n matmul_kernel[grid](\n a,\n b,\n c,\n M,\n N,\n K,\n a.stride(0),\n a.stride(1),\n b.stride(0),\n b.stride(1),\n c.stride(0),\n c.stride(1),\n ACTIVATION=activation,\n )\n return c"
+ "def matmul(a, b, activation=None):\n # checks constraints\n assert a.shape[1] == b.shape[0], \"incompatible dimensions\"\n assert a.is_contiguous(), \"matrix A must be contiguous\"\n assert b.is_contiguous(), \"matrix B must be contiguous\"\n M, K = a.shape\n K, N = b.shape\n assert (\n K % 32 == 0\n ), \"We don't check memory-out-of-bounds with K so K must be divisible by BLOCK_SIZE_K\"\n # allocates output\n c = torch.empty((M, N), device=a.device, dtype=a.dtype)\n # 1D launch kernel where each block gets its own program.\n grid = lambda META: (\n triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),\n )\n matmul_kernel[grid](\n a, b, c,\n M, N, K,\n a.stride(0), a.stride(1),\n b.stride(0), b.stride(1),\n c.stride(0), c.stride(1),\n ACTIVATION=activation,\n )\n return c"
]
},
{
diff --git a/_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py b/_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py
index e71fae2d6..80207d8cf 100644
--- a/_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py
+++ b/_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py
@@ -23,7 +23,7 @@ You will specifically learn about:
# yourself with Triton, in a way that is easy to customize and extend.
#
# Roughly speaking, the kernel that we will write will implement the following blocked
-# algorithm to multiply a (MxK) by a (KxN) matrix:
+# algorithm to multiply a (M, K) by a (K, N) matrix:
#
# .. code-block:: python
#
@@ -38,7 +38,7 @@ You will specifically learn about:
# acc += dot(a, b)
# C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;
#
-# where each iteration of the doubly-nested for-loop corresponds to a Triton program instance.
+# where each iteration of the doubly-nested for-loop is performed by a dedicated Triton program instance.
# %%
# Compute Kernel
@@ -53,35 +53,31 @@ You will specifically learn about:
# ~~~~~~~~~~~~~~~~~~~~
#
# For a row-major 2D tensor :code:`X`, the memory location of :code:`X[i, j]` is given b
-# y :code:`&X[i, j] = X + i*stride_x_0 + j*stride_x_1`.
+# y :code:`&X[i, j] = X + i*stride_xi + j*stride_xj`.
# Therefore, blocks of pointers for :code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]` and
# :code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]` can be defined in pseudo-code as:
#
# .. code-block:: python
#
-# &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = A + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
-# &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = B + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);
+# &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = a_ptr + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
+# &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = b_ptr + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);
#
# Which means that pointers for blocks of A and B can be initialized (i.e., :code:`k=0`) in Triton as:
#
# .. code-block:: python
#
-# pid_m = triton.program_id(0)
-# pid_n = triton.program_id(1)
-# rm = pid_m * BLOCK_SIZE_M + triton.arange(0, BLOCK_SIZE_M)
-# rn = pid_n * BLOCK_SIZE_N + triton.arange(0, BLOCK_SIZE_N)
-# rk = triton.arange(0, BLOCK_SIZE_K)
-# // pointer for A operand
-# pa = A + (rm[:, None] * stride_a_0 + rk[None, :] * stride_a_1);
-# // pointer for B operand
-# pb = B + (rk[:, None] * stride_b_0 + rn[None, :] * stride_b_1);
+# offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+# offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+# offs_k = tl.arange(0, BLOCK_SIZE_K)
+# a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
+# b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)
#
# And then updated in the inner loop as follows:
#
# .. code-block:: python
#
-# pa += BLOCK_SIZE_K * stride_a_1;
-# pb += BLOCK_SIZE_K * stride_b_0;
+# pa += BLOCK_SIZE_K * stride_ak;
+# pb += BLOCK_SIZE_K * stride_bk;
#
#
# L2 Cache Optimizations
@@ -109,13 +105,25 @@ You will specifically learn about:
#
# .. code-block:: python
#
-# pid = triton.program_id(0);
-# width = GROUP_M * grid_n;
-# group_id = pid // width;
-# # we need to handle the case where M % (GROUP_M*BLOCK_SIZE_M) != 0
-# group_size = min(grid_m - group_id * GROUP_M, GROUP_M);
-# pid_m = group_id * GROUP_M + (pid % group_size);
-# pid_n = (pid % width) // (group_size);
+# # program ID
+# pid = tl.program_id(axis=0)
+# # number of program ids along the M axis
+# num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+# # number of programs ids along the N axis
+# num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
+# # number of programs in group
+# num_pid_in_group = GROUP_SIZE_M * num_pid_n
+# # id of the group this program is in
+# group_id = pid // num_pid_in_group
+# # row-id of the first program in the group
+# first_pid_m = group_id * GROUP_SIZE_M
+# # if `num_pid_m` isn't divisible by `GROUP_SIZE_M`, the last group is smaller
+# group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
+# # *within groups*, programs are ordered in a column-major order
+# # row-id of the program in the *launch grid*
+# pid_m = first_pid_m + (pid % group_size_m)
+# # col-id of the program in the *launch grid*
+# pid_n = (pid % num_pid_in_group) // group_size_m
#
# For example, in the following matmul where each matrix is 9 blocks by 9 blocks,
# we can see that if we compute the output in row-major ordering, we need to load 90
@@ -164,26 +172,19 @@ import triton.language as tl
@triton.jit
def matmul_kernel(
# Pointers to matrices
- a_ptr,
- b_ptr,
- c_ptr,
+ a_ptr, b_ptr, c_ptr,
# Matrix dimensions
- M,
- N,
- K,
+ M, N, K,
# The stride variables represent how much to increase the ptr by when moving by 1
# element in a particular dimension. E.g. stride_am is how much to increase a_ptr
# by to get the element one row down (A has M rows)
- stride_am,
- stride_ak,
- stride_bk,
- stride_bn,
- stride_cm,
- stride_cn,
+ stride_am, stride_ak,
+ stride_bk, stride_bn,
+ stride_cm, stride_cn,
+ # Meta-parameters
**meta,
):
- """Kernel for computing the matmul AB = C
-
+ """Kernel for computing the matmul C = A x B.
A has shape (M, K), B has shape (K, N) and C has shape (M, N)
"""
# extract meta-parameters
@@ -191,67 +192,65 @@ def matmul_kernel(
BLOCK_SIZE_N = meta['BLOCK_SIZE_N']
BLOCK_SIZE_K = meta['BLOCK_SIZE_K']
GROUP_SIZE_M = 8
+
+ # -----------------------------------------------------------
+ # Map program ids `pid` to the block of C it should compute.
+ # This is done in a grouped ordering to promote L2 data reuse
+ # See above `L2 Cache Optimizations` section for details
pid = tl.program_id(axis=0)
+ num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+ num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
+ num_pid_in_group = GROUP_SIZE_M * num_pid_n
+ group_id = pid // num_pid_in_group
+ first_pid_m = group_id * GROUP_SIZE_M
+ group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
+ pid_m = first_pid_m + (pid % group_size_m)
+ pid_n = (pid % num_pid_in_group) // group_size_m
- # the number of blocks is the ceil(M / BLOCK_SIZE_M) since we need an extra block
- # Note that this will lead to some quantization in performance where time-taken jumps
- # when you need to add a new block
- n_blocks_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M
- n_blocks_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N
+ # ----------------------------------------------------------
+ # Create pointers for the first blocks of A and B.
+ # We will advance this pointer as we move in the K direction
+ # and accumulate
+ # a_ptrs is a block of [BLOCK_SIZE_M, BLOCK_SIZE_K] pointers
+ # b_ptrs is a block of [BLOCK_SIZE_K, BLOCK_SIZE_n] pointers
+ # see above `Pointer Arithmetics` section for details
+ offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
+ b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)
- # Map PIDs to the block they should compute. This is done in a grouped ordering
- # to promote L2 cache reuse.
- n_output_blocks_in_group = GROUP_SIZE_M * n_blocks_n
- group_id = pid // n_output_blocks_in_group
- first_m_block_in_group = group_id * GROUP_SIZE_M
-
- # If the number of blocks is not divisible by the group size, the last group is smaller
- group_size_m = min(n_blocks_m - first_m_block_in_group, GROUP_SIZE_M)
-
- # Within a group, we compute in col-major ordering, block_m and block_n are the
- # output row and col that this program is computing in terms of blocks
- block_m = first_m_block_in_group + (pid % group_size_m)
- block_n = (pid % n_output_blocks_in_group) // group_size_m
-
- # Convert from block indices back to element indices
- m_start = block_m * BLOCK_SIZE_M
- n_start = block_n * BLOCK_SIZE_N
-
- # Expand out to all the offsets for each of the elements in this block.
- m_offsets_a = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]
- n_offsets_b = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]
- k_offsets = tl.arange(0, BLOCK_SIZE_K)
-
- # Get the pointers for the first block of each. We will advance this pointer
- # as we move in the K direction and accumulate.
- # a_ptrs should contain BLOCK_SIZE_M * BLOCK_SIZE_K pointers
- a_ptrs = a_ptr + (stride_am * m_offsets_a + stride_ak * k_offsets[None, :])
- # b_ptrs should contain BLOCK_SIZE_K * BLOCK_SIZE_N pointers
- b_ptrs = b_ptr + (stride_bk * k_offsets[:, None] + stride_bn * n_offsets_b)
- # We accumulate internally in fp32, but the output is written out in the dtype
- # of the tensor when it is stored
+ # -----------------------------------------------------------
+ # Iterate to compute a block of the C matrix
+ # We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block
+ # of fp32 values for higher accuracy.
+ # `accumulator` will be converted back to fp16 after the loop
accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
for k in range(0, K, BLOCK_SIZE_K):
- # Note that for simplicity, we don't apply a mask here. This means that if K is
- # not a multiple of BLOCK_SIZE_K, this will access out-of-bounds memory and
- # accumulate it incorrectly.
+ # Note that for simplicity, we don't apply a mask here.
+ # This means that if K is not a multiple of BLOCK_SIZE_K,
+ # this will access out-of-bounds memory and produce an
+ # error or (worse!) incorrect results.
a = tl.load(a_ptrs)
b = tl.load(b_ptrs)
# We accumulate along the K dimension
accumulator += tl.dot(a, b)
-
# Advance the ptrs to the next K block
a_ptrs += BLOCK_SIZE_K * stride_ak
b_ptrs += BLOCK_SIZE_K * stride_bk
- # triton can accept arbitrary activation function via metaparameters!
- if meta['ACTIVATION']:
+ # you can fuse arbitrary activation functions here
+ # while the accumulator is still in FP32 !
+ if meta['ACTIVATION']:
accumulator = meta['ACTIVATION'](accumulator)
+ c = accumulator.to(tl.float16)
- m_offsets_c = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]
- n_offsets_c = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]
- c_ptrs = c_ptr + stride_cm * m_offsets_c + stride_cn * n_offsets_c
- mask = (m_offsets_c < M) & (n_offsets_c < N)
- tl.store(c_ptrs, accumulator, mask=mask)
+ # -----------------------------------------------------------
+ # Write back the block of the output matrix C
+ offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ c_ptrs = c_ptr + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]
+ c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+ tl.store(c_ptrs, c, mask=c_mask)
# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`
@@ -282,18 +281,11 @@ def matmul(a, b, activation=None):
triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),
)
matmul_kernel[grid](
- a,
- b,
- c,
- M,
- N,
- K,
- a.stride(0),
- a.stride(1),
- b.stride(0),
- b.stride(1),
- c.stride(0),
- c.stride(1),
+ a, b, c,
+ M, N, K,
+ a.stride(0), a.stride(1),
+ b.stride(0), b.stride(1),
+ c.stride(0), c.stride(1),
ACTIVATION=activation,
)
return c
diff --git a/_images/sphx_glr_01-vector-add_001.png b/_images/sphx_glr_01-vector-add_001.png
index 3d7049a91..f4f36e595 100644
Binary files a/_images/sphx_glr_01-vector-add_001.png and b/_images/sphx_glr_01-vector-add_001.png differ
diff --git a/_images/sphx_glr_01-vector-add_thumb.png b/_images/sphx_glr_01-vector-add_thumb.png
index acc77a248..a2efc1f1d 100644
Binary files a/_images/sphx_glr_01-vector-add_thumb.png and b/_images/sphx_glr_01-vector-add_thumb.png differ
diff --git a/_images/sphx_glr_02-fused-softmax_001.png b/_images/sphx_glr_02-fused-softmax_001.png
index 0ee5b44ef..9424aae06 100644
Binary files a/_images/sphx_glr_02-fused-softmax_001.png and b/_images/sphx_glr_02-fused-softmax_001.png differ
diff --git a/_images/sphx_glr_02-fused-softmax_thumb.png b/_images/sphx_glr_02-fused-softmax_thumb.png
index 41d99308e..de8be8737 100644
Binary files a/_images/sphx_glr_02-fused-softmax_thumb.png and b/_images/sphx_glr_02-fused-softmax_thumb.png differ
diff --git a/_images/sphx_glr_03-matrix-multiplication_001.png b/_images/sphx_glr_03-matrix-multiplication_001.png
index 2140ebaa8..3566f05bc 100644
Binary files a/_images/sphx_glr_03-matrix-multiplication_001.png and b/_images/sphx_glr_03-matrix-multiplication_001.png differ
diff --git a/_images/sphx_glr_03-matrix-multiplication_thumb.png b/_images/sphx_glr_03-matrix-multiplication_thumb.png
index 0bfd1c39a..dcfd633a2 100644
Binary files a/_images/sphx_glr_03-matrix-multiplication_thumb.png and b/_images/sphx_glr_03-matrix-multiplication_thumb.png differ
diff --git a/_sources/getting-started/tutorials/01-vector-add.rst.txt b/_sources/getting-started/tutorials/01-vector-add.rst.txt
index 369f14ead..237282f10 100644
--- a/_sources/getting-started/tutorials/01-vector-add.rst.txt
+++ b/_sources/getting-started/tutorials/01-vector-add.rst.txt
@@ -234,10 +234,10 @@ We can now run the decorated function above. Pass `print_data=True` to see the p
0 4096.0 9.600000 9.600000
1 8192.0 19.200000 19.200000
2 16384.0 38.400001 38.400001
- 3 32768.0 76.800002 76.800002
+ 3 32768.0 63.999998 76.800002
4 65536.0 127.999995 127.999995
5 131072.0 219.428568 219.428568
- 6 262144.0 341.333321 384.000001
+ 6 262144.0 384.000001 384.000001
7 524288.0 472.615390 472.615390
8 1048576.0 614.400016 614.400016
9 2097152.0 722.823517 722.823517
@@ -254,7 +254,7 @@ We can now run the decorated function above. Pass `print_data=True` to see the p
.. rst-class:: sphx-glr-timing
- **Total running time of the script:** ( 0 minutes 10.994 seconds)
+ **Total running time of the script:** ( 0 minutes 10.971 seconds)
.. _sphx_glr_download_getting-started_tutorials_01-vector-add.py:
diff --git a/_sources/getting-started/tutorials/02-fused-softmax.rst.txt b/_sources/getting-started/tutorials/02-fused-softmax.rst.txt
index 902505c37..06b931adc 100644
--- a/_sources/getting-started/tutorials/02-fused-softmax.rst.txt
+++ b/_sources/getting-started/tutorials/02-fused-softmax.rst.txt
@@ -306,10 +306,10 @@ We will then compare its performance against (1) :code:`torch.softmax` and (2) t
3 640.0 682.666684 640.000002 160.000000
4 768.0 702.171410 664.216187 163.839992
.. ... ... ... ...
- 93 12160.0 812.359066 406.179533 198.936606
- 94 12288.0 812.429770 416.101597 199.298541
- 95 12416.0 810.840807 412.149375 198.854847
- 96 12544.0 810.925276 412.971190 199.209928
+ 93 12160.0 812.359066 405.755985 198.936606
+ 94 12288.0 812.429770 415.222812 199.096718
+ 95 12416.0 810.840807 411.296057 198.755369
+ 96 12544.0 810.925276 412.971190 199.012395
97 12672.0 811.007961 412.097543 199.167004
[98 rows x 4 columns]
@@ -328,7 +328,7 @@ In the above plot, we can see that:
.. rst-class:: sphx-glr-timing
- **Total running time of the script:** ( 1 minutes 12.617 seconds)
+ **Total running time of the script:** ( 1 minutes 12.739 seconds)
.. _sphx_glr_download_getting-started_tutorials_02-fused-softmax.py:
diff --git a/_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt b/_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt
index 08614bc72..d0c673c8e 100644
--- a/_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt
+++ b/_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt
@@ -42,7 +42,7 @@ In this tutorial, you will learn how to implement efficient matrix multiplicatio
yourself with Triton, in a way that is easy to customize and extend.
Roughly speaking, the kernel that we will write will implement the following blocked
-algorithm to multiply a (MxK) by a (KxN) matrix:
+algorithm to multiply a (M, K) by a (K, N) matrix:
.. code-block:: python
@@ -57,9 +57,9 @@ algorithm to multiply a (MxK) by a (KxN) matrix:
acc += dot(a, b)
C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;
-where each iteration of the doubly-nested for-loop corresponds to a Triton program instance.
+where each iteration of the doubly-nested for-loop is performed by a dedicated Triton program instance.
-.. GENERATED FROM PYTHON SOURCE LINES 44-129
+.. GENERATED FROM PYTHON SOURCE LINES 44-137
Compute Kernel
----------------
@@ -73,35 +73,31 @@ Pointer Arithmetics
~~~~~~~~~~~~~~~~~~~~
For a row-major 2D tensor :code:`X`, the memory location of :code:`X[i, j]` is given b
-y :code:`&X[i, j] = X + i*stride_x_0 + j*stride_x_1`.
+y :code:`&X[i, j] = X + i*stride_xi + j*stride_xj`.
Therefore, blocks of pointers for :code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]` and
:code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]` can be defined in pseudo-code as:
.. code-block:: python
- &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = A + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
- &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = B + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);
+ &A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = a_ptr + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
+ &B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = b_ptr + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);
Which means that pointers for blocks of A and B can be initialized (i.e., :code:`k=0`) in Triton as:
.. code-block:: python
- pid_m = triton.program_id(0)
- pid_n = triton.program_id(1)
- rm = pid_m * BLOCK_SIZE_M + triton.arange(0, BLOCK_SIZE_M)
- rn = pid_n * BLOCK_SIZE_N + triton.arange(0, BLOCK_SIZE_N)
- rk = triton.arange(0, BLOCK_SIZE_K)
- // pointer for A operand
- pa = A + (rm[:, None] * stride_a_0 + rk[None, :] * stride_a_1);
- // pointer for B operand
- pb = B + (rk[:, None] * stride_b_0 + rn[None, :] * stride_b_1);
+ offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
+ b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)
And then updated in the inner loop as follows:
.. code-block:: python
- pa += BLOCK_SIZE_K * stride_a_1;
- pb += BLOCK_SIZE_K * stride_b_0;
+ pa += BLOCK_SIZE_K * stride_ak;
+ pb += BLOCK_SIZE_K * stride_bk;
L2 Cache Optimizations
@@ -129,13 +125,25 @@ switching to the next column:
.. code-block:: python
- pid = triton.program_id(0);
- width = GROUP_M * grid_n;
- group_id = pid // width;
- # we need to handle the case where M % (GROUP_M*BLOCK_SIZE_M) != 0
- group_size = min(grid_m - group_id * GROUP_M, GROUP_M);
- pid_m = group_id * GROUP_M + (pid % group_size);
- pid_n = (pid % width) // (group_size);
+ # program ID
+ pid = tl.program_id(axis=0)
+ # number of program ids along the M axis
+ num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+ # number of programs ids along the N axis
+ num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
+ # number of programs in group
+ num_pid_in_group = GROUP_SIZE_M * num_pid_n
+ # id of the group this program is in
+ group_id = pid // num_pid_in_group
+ # row-id of the first program in the group
+ first_pid_m = group_id * GROUP_SIZE_M
+ # if `num_pid_m` isn't divisible by `GROUP_SIZE_M`, the last group is smaller
+ group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
+ # *within groups*, programs are ordered in a column-major order
+ # row-id of the program in the *launch grid*
+ pid_m = first_pid_m + (pid % group_size_m)
+ # col-id of the program in the *launch grid*
+ pid_n = (pid % num_pid_in_group) // group_size_m
For example, in the following matmul where each matrix is 9 blocks by 9 blocks,
we can see that if we compute the output in row-major ordering, we need to load 90
@@ -147,13 +155,13 @@ In practice, this can improve the performance of our matrix multiplication kerne
more than 10\% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).
-.. GENERATED FROM PYTHON SOURCE LINES 131-134
+.. GENERATED FROM PYTHON SOURCE LINES 139-142
Final Result
-------------
-.. GENERATED FROM PYTHON SOURCE LINES 134-263
+.. GENERATED FROM PYTHON SOURCE LINES 142-262
.. code-block:: default
@@ -190,26 +198,19 @@ Final Result
@triton.jit
def matmul_kernel(
# Pointers to matrices
- a_ptr,
- b_ptr,
- c_ptr,
+ a_ptr, b_ptr, c_ptr,
# Matrix dimensions
- M,
- N,
- K,
+ M, N, K,
# The stride variables represent how much to increase the ptr by when moving by 1
# element in a particular dimension. E.g. stride_am is how much to increase a_ptr
# by to get the element one row down (A has M rows)
- stride_am,
- stride_ak,
- stride_bk,
- stride_bn,
- stride_cm,
- stride_cn,
+ stride_am, stride_ak,
+ stride_bk, stride_bn,
+ stride_cm, stride_cn,
+ # Meta-parameters
**meta,
):
- """Kernel for computing the matmul AB = C
-
+ """Kernel for computing the matmul C = A x B.
A has shape (M, K), B has shape (K, N) and C has shape (M, N)
"""
# extract meta-parameters
@@ -217,67 +218,65 @@ Final Result
BLOCK_SIZE_N = meta['BLOCK_SIZE_N']
BLOCK_SIZE_K = meta['BLOCK_SIZE_K']
GROUP_SIZE_M = 8
+
+ # -----------------------------------------------------------
+ # Map program ids `pid` to the block of C it should compute.
+ # This is done in a grouped ordering to promote L2 data reuse
+ # See above `L2 Cache Optimizations` section for details
pid = tl.program_id(axis=0)
+ num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+ num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
+ num_pid_in_group = GROUP_SIZE_M * num_pid_n
+ group_id = pid // num_pid_in_group
+ first_pid_m = group_id * GROUP_SIZE_M
+ group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
+ pid_m = first_pid_m + (pid % group_size_m)
+ pid_n = (pid % num_pid_in_group) // group_size_m
- # the number of blocks is the ceil(M / BLOCK_SIZE_M) since we need an extra block
- # Note that this will lead to some quantization in performance where time-taken jumps
- # when you need to add a new block
- n_blocks_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M
- n_blocks_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N
+ # ----------------------------------------------------------
+ # Create pointers for the first blocks of A and B.
+ # We will advance this pointer as we move in the K direction
+ # and accumulate
+ # a_ptrs is a block of [BLOCK_SIZE_M, BLOCK_SIZE_K] pointers
+ # b_ptrs is a block of [BLOCK_SIZE_K, BLOCK_SIZE_n] pointers
+ # see above `Pointer Arithmetics` section for details
+ offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ offs_k = tl.arange(0, BLOCK_SIZE_K)
+ a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
+ b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)
- # Map PIDs to the block they should compute. This is done in a grouped ordering
- # to promote L2 cache reuse.
- n_output_blocks_in_group = GROUP_SIZE_M * n_blocks_n
- group_id = pid // n_output_blocks_in_group
- first_m_block_in_group = group_id * GROUP_SIZE_M
-
- # If the number of blocks is not divisible by the group size, the last group is smaller
- group_size_m = min(n_blocks_m - first_m_block_in_group, GROUP_SIZE_M)
-
- # Within a group, we compute in col-major ordering, block_m and block_n are the
- # output row and col that this program is computing in terms of blocks
- block_m = first_m_block_in_group + (pid % group_size_m)
- block_n = (pid % n_output_blocks_in_group) // group_size_m
-
- # Convert from block indices back to element indices
- m_start = block_m * BLOCK_SIZE_M
- n_start = block_n * BLOCK_SIZE_N
-
- # Expand out to all the offsets for each of the elements in this block.
- m_offsets_a = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]
- n_offsets_b = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]
- k_offsets = tl.arange(0, BLOCK_SIZE_K)
-
- # Get the pointers for the first block of each. We will advance this pointer
- # as we move in the K direction and accumulate.
- # a_ptrs should contain BLOCK_SIZE_M * BLOCK_SIZE_K pointers
- a_ptrs = a_ptr + (stride_am * m_offsets_a + stride_ak * k_offsets[None, :])
- # b_ptrs should contain BLOCK_SIZE_K * BLOCK_SIZE_N pointers
- b_ptrs = b_ptr + (stride_bk * k_offsets[:, None] + stride_bn * n_offsets_b)
- # We accumulate internally in fp32, but the output is written out in the dtype
- # of the tensor when it is stored
+ # -----------------------------------------------------------
+ # Iterate to compute a block of the C matrix
+ # We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block
+ # of fp32 values for higher accuracy.
+ # `accumulator` will be converted back to fp16 after the loop
accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
for k in range(0, K, BLOCK_SIZE_K):
- # Note that for simplicity, we don't apply a mask here. This means that if K is
- # not a multiple of BLOCK_SIZE_K, this will access out-of-bounds memory and
- # accumulate it incorrectly.
+ # Note that for simplicity, we don't apply a mask here.
+ # This means that if K is not a multiple of BLOCK_SIZE_K,
+ # this will access out-of-bounds memory and produce an
+ # error or (worse!) incorrect results.
a = tl.load(a_ptrs)
b = tl.load(b_ptrs)
# We accumulate along the K dimension
accumulator += tl.dot(a, b)
-
# Advance the ptrs to the next K block
a_ptrs += BLOCK_SIZE_K * stride_ak
b_ptrs += BLOCK_SIZE_K * stride_bk
- # triton can accept arbitrary activation function via metaparameters!
- if meta['ACTIVATION']:
+ # you can fuse arbitrary activation functions here
+ # while the accumulator is still in FP32 !
+ if meta['ACTIVATION']:
accumulator = meta['ACTIVATION'](accumulator)
+ c = accumulator.to(tl.float16)
- m_offsets_c = (m_start + tl.arange(0, BLOCK_SIZE_M))[:, None]
- n_offsets_c = (n_start + tl.arange(0, BLOCK_SIZE_N))[None, :]
- c_ptrs = c_ptr + stride_cm * m_offsets_c + stride_cn * n_offsets_c
- mask = (m_offsets_c < M) & (n_offsets_c < N)
- tl.store(c_ptrs, accumulator, mask=mask)
+ # -----------------------------------------------------------
+ # Write back the block of the output matrix C
+ offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+ offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+ c_ptrs = c_ptr + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]
+ c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+ tl.store(c_ptrs, c, mask=c_mask)
# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`
@@ -293,12 +292,12 @@ Final Result
-.. GENERATED FROM PYTHON SOURCE LINES 264-266
+.. GENERATED FROM PYTHON SOURCE LINES 263-265
We can now create a convenience wrapper function that only takes two input tensors
and (1) checks any shape constraint; (2) allocates the output; (3) launches the above kernel
-.. GENERATED FROM PYTHON SOURCE LINES 266-302
+.. GENERATED FROM PYTHON SOURCE LINES 265-294
.. code-block:: default
@@ -321,18 +320,11 @@ and (1) checks any shape constraint; (2) allocates the output; (3) launches the
triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),
)
matmul_kernel[grid](
- a,
- b,
- c,
- M,
- N,
- K,
- a.stride(0),
- a.stride(1),
- b.stride(0),
- b.stride(1),
- c.stride(0),
- c.stride(1),
+ a, b, c,
+ M, N, K,
+ a.stride(0), a.stride(1),
+ b.stride(0), b.stride(1),
+ c.stride(0), c.stride(1),
ACTIVATION=activation,
)
return c
@@ -345,14 +337,14 @@ and (1) checks any shape constraint; (2) allocates the output; (3) launches the
-.. GENERATED FROM PYTHON SOURCE LINES 303-307
+.. GENERATED FROM PYTHON SOURCE LINES 295-299
Unit Test
-----------
We can test our custom matrix multiplication operation against a native torch implementation (i.e., cuBLAS)
-.. GENERATED FROM PYTHON SOURCE LINES 307-320
+.. GENERATED FROM PYTHON SOURCE LINES 299-312
.. code-block:: default
@@ -400,7 +392,7 @@ We can test our custom matrix multiplication operation against a native torch im
-.. GENERATED FROM PYTHON SOURCE LINES 321-327
+.. GENERATED FROM PYTHON SOURCE LINES 313-319
Benchmark
--------------
@@ -409,7 +401,7 @@ Square Matrix Performance
~~~~~~~~~~~~~~~~~~~~~~~~~~
We can now compare the performance of our kernel against that of cuBLAS. Here we focus on square matrices, but feel free to arrange this script as you wish to benchmark any other matrix shape.
-.. GENERATED FROM PYTHON SOURCE LINES 327-368
+.. GENERATED FROM PYTHON SOURCE LINES 319-360
.. code-block:: default
@@ -471,37 +463,37 @@ We can now compare the performance of our kernel against that of cuBLAS. Here we
matmul-performance:
M cuBLAS ... Triton Triton (+ LeakyReLU)
0 128.0 0.455111 ... 0.512000 0.512000
- 1 256.0 2.730667 ... 2.978909 2.978909
+ 1 256.0 2.978909 ... 2.978909 2.978909
2 384.0 7.372800 ... 8.507077 8.507077
- 3 512.0 14.563555 ... 15.420235 16.384000
- 4 640.0 22.260869 ... 24.380953 23.272727
+ 3 512.0 14.563555 ... 16.384000 16.384000
+ 4 640.0 22.260869 ... 24.380953 24.380953
5 768.0 32.768000 ... 34.028308 34.028308
- 6 896.0 39.025776 ... 40.140799 39.025776
- 7 1024.0 49.932191 ... 53.773130 52.428801
- 8 1152.0 45.242181 ... 46.656000 46.656000
- 9 1280.0 51.200001 ... 56.888887 56.888887
+ 6 896.0 39.025776 ... 40.140799 36.023796
+ 7 1024.0 49.932191 ... 52.428801 52.428801
+ 8 1152.0 44.566925 ... 46.656000 46.656000
+ 9 1280.0 51.200001 ... 56.888887 56.109587
10 1408.0 64.138541 ... 64.902096 64.902096
11 1536.0 78.643199 ... 76.106321 75.296679
- 12 1664.0 62.929456 ... 62.061463 62.061463
+ 12 1664.0 63.372618 ... 62.492442 61.636381
13 1792.0 72.983276 ... 69.810085 69.379162
14 1920.0 67.434145 ... 70.892307 70.530615
- 15 2048.0 73.908442 ... 74.898285 74.565406
- 16 2176.0 83.500614 ... 78.916269 79.855747
- 17 2304.0 68.251065 ... 73.275679 72.828879
- 18 2432.0 71.125224 ... 80.731218 80.731218
- 19 2560.0 77.649287 ... 76.560748 76.382283
- 20 2688.0 81.928846 ... 80.366642 82.823267
- 21 2816.0 77.743683 ... 78.868366 78.301990
- 22 2944.0 81.832567 ... 79.610276 78.605729
- 23 3072.0 81.005868 ... 81.005868 82.420822
- 24 3200.0 84.321474 ... 89.635851 85.106381
- 25 3328.0 83.226931 ... 87.156532 86.113988
- 26 3456.0 81.932484 ... 83.632331 85.313831
- 27 3584.0 87.211821 ... 87.211821 91.563533
- 28 3712.0 85.896254 ... 82.491612 84.874549
- 29 3840.0 85.070769 ... 87.493673 87.701820
- 30 3968.0 92.935215 ... 83.865247 83.578035
- 31 4096.0 93.662059 ... 85.926841 84.840533
+ 15 2048.0 73.908442 ... 75.234154 74.898285
+ 16 2176.0 81.472263 ... 80.817862 80.173899
+ 17 2304.0 68.446623 ... 73.501144 73.275679
+ 18 2432.0 71.305746 ... 81.197876 79.362895
+ 19 2560.0 77.649287 ... 77.649287 76.560748
+ 20 2688.0 82.642823 ... 80.708630 82.823267
+ 21 2816.0 79.587973 ... 79.733474 77.605356
+ 22 2944.0 81.967162 ... 78.112900 79.230573
+ 23 3072.0 81.707223 ... 84.135370 79.863336
+ 24 3200.0 84.099871 ... 87.074829 89.136491
+ 25 3328.0 83.905938 ... 84.003845 86.424125
+ 26 3456.0 81.518272 ... 85.494768 81.353753
+ 27 3584.0 86.540320 ... 94.448944 94.847460
+ 28 3712.0 83.947349 ... 88.955779 89.114488
+ 29 3840.0 84.809814 ... 88.191387 87.217666
+ 30 3968.0 93.148045 ... 83.179234 87.409694
+ 31 4096.0 93.531519 ... 89.777746 87.552332
[32 rows x 5 columns]
@@ -511,7 +503,7 @@ We can now compare the performance of our kernel against that of cuBLAS. Here we
.. rst-class:: sphx-glr-timing
- **Total running time of the script:** ( 2 minutes 9.226 seconds)
+ **Total running time of the script:** ( 2 minutes 30.498 seconds)
.. _sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py:
diff --git a/_sources/getting-started/tutorials/sg_execution_times.rst.txt b/_sources/getting-started/tutorials/sg_execution_times.rst.txt
index 41acb9ef9..d16236a4b 100644
--- a/_sources/getting-started/tutorials/sg_execution_times.rst.txt
+++ b/_sources/getting-started/tutorials/sg_execution_times.rst.txt
@@ -5,12 +5,12 @@
Computation times
=================
-**03:32.837** total execution time for **getting-started_tutorials** files:
+**03:54.208** total execution time for **getting-started_tutorials** files:
+---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 02:09.226 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 02:30.498 | 0.0 MB |
+---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``) | 01:12.617 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``) | 01:12.739 | 0.0 MB |
+---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``) | 00:10.994 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``) | 00:10.971 | 0.0 MB |
+---------------------------------------------------------------------------------------------------------+-----------+--------+
diff --git a/getting-started/tutorials/01-vector-add.html b/getting-started/tutorials/01-vector-add.html
index 14cfc89a5..4a2f32168 100644
--- a/getting-started/tutorials/01-vector-add.html
+++ b/getting-started/tutorials/01-vector-add.html
@@ -322,10 +322,10 @@ for different problem sizes.
0 4096.0 9.600000 9.600000
1 8192.0 19.200000 19.200000
2 16384.0 38.400001 38.400001
-3 32768.0 76.800002 76.800002
+3 32768.0 63.999998 76.800002
4 65536.0 127.999995 127.999995
5 131072.0 219.428568 219.428568
-6 262144.0 341.333321 384.000001
+6 262144.0 384.000001 384.000001
7 524288.0 472.615390 472.615390
8 1048576.0 614.400016 614.400016
9 2097152.0 722.823517 722.823517
@@ -337,7 +337,7 @@ for different problem sizes.
15 134217728.0 851.577704 850.656574
-Total running time of the script: ( 0 minutes 10.994 seconds)
+Total running time of the script: ( 0 minutes 10.971 seconds)