gaspersic/triton
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
db7b163cb5291e41ea7aeab738eb1146739e2711
triton/master/.doctrees/getting-started/tutorials/03-matrix-multiplication.doctree

652 lines
55 KiB
Plaintext
Raw Normal View History

[GH-PAGES] Updated website
2022-08-12 00:47:45 +00:00
<EFBFBD><05>k<EFBFBD><00>sphinx.addnodes<65><73>document<6E><74><EFBFBD>)<29><>}<7D>(<28> rawsource<63><65><00><>children<65>]<5D>(<28>docutils.nodes<65><73>comment<6E><74><EFBFBD>)<29><>}<7D>(h<05> DO NOT EDIT.<2E>h]<5D>h <09>Text<78><74><EFBFBD><EFBFBD> DO NOT EDIT.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<06>parent<6E>h uba<62>
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
attributes<EFBFBD>}<7D>(<28>ids<64>]<5D><>classes<65>]<5D><>names<65>]<5D><>dupnames<65>]<5D><>backrefs<66>]<5D><> xml:space<63><65>preserve<76>u<EFBFBD>tagname<6D>h
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
hhhh<03>source<63><65>u/tmp/tmpwu1i38me/7394d732adcec5a67034926a28080af2a7853217/docs/getting-started/tutorials/03-matrix-multiplication.rst<73><74>line<6E>Kubh )<29><>}<7D>(h<05>8THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.<2E>h]<5D>h<11>8THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhh)ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
hhhhh&h'h(Kubh )<29><>}<7D>(h<05>-TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:<3A>h]<5D>h<11>-TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhh7ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hhhhh&h'h(Kubh )<29><>}<7D>(h<05>7"getting-started/tutorials/03-matrix-multiplication.py"<22>h]<5D>h<11>7"getting-started/tutorials/03-matrix-multiplication.py"<22><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhhEubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hhhhh&h'h(Kubh )<29><>}<7D>(h<05>LINE NUMBERS ARE GIVEN BELOW.<2E>h]<5D>h<11>LINE NUMBERS ARE GIVEN BELOW.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhhSubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hhhhh&h'h(Kubh<00>only<6C><79><EFBFBD>)<29><>}<7D>(hhh]<5D>h <09>note<74><65><EFBFBD>)<29><>}<7D>(h<05>}Click :ref:`here <sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py>`
to download the full example code<64>h]<5D>h <09> paragraph<70><68><EFBFBD>)<29><>}<7D>(h<05>}Click :ref:`here <sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py>`
to download the full example code<64>h]<5D>(h<11>Click <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>Click <20>hhnubh<00> pending_xref<65><66><EFBFBD>)<29><>}<7D>(h<05>U:ref:`here <sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py>`<60>h]<5D>h <09>inline<6E><65><EFBFBD>)<29><>}<7D>(hh{h]<5D>h<11>here<72><65><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhhubah}<7D>(h]<5D>h]<5D>(<28>xref<65><66>std<74><64>std-ref<65>eh]<5D>h]<5D>h!]<5D>uh%h}hhyubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>refdoc<6F><63>2getting-started/tutorials/03-matrix-multiplication<6F><6E> refdomain<69>h<EFBFBD><68>reftype<70><65>ref<65><66> refexplicit<69><74><EFBFBD>refwarn<72><6E><EFBFBD> reftarget<65><74>Gsphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py<70>uh%hwh&h'h(K hhnubh<11>"
to download the full example code<64><65><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>"
to download the full example code<64>hhnubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K hhhubah}<7D>(h]<5D>h]<5D><>sphx-glr-download-link-note<74>ah]<5D>h]<5D>h!]<5D>uh%hfhhchhh&h'h(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>expr<70><72>html<6D>uh%hahhh&h'h(Khhubh <09>target<65><74><EFBFBD>)<29><>}<7D>(h<05>C.. _sphx_glr_getting-started_tutorials_03-matrix-multiplication.py:<3A>h]<5D>h}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>refid<69><64>>sphx-glr-getting-started-tutorials-03-matrix-multiplication-py<70>uh%h<>h(Khhhhh&h'ubh <09>section<6F><6E><EFBFBD>)<29><>}<7D>(hhh]<5D>(h <09>title<6C><65><EFBFBD>)<29><>}<7D>(h<05>Matrix Multiplication<6F>h]<5D>h<11>Matrix Multiplication<6F><6E><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<>hh<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(Kubhm)<29><>}<7D>(h<05><>In this tutorial, you will write a 25-lines high-performance FP16 matrix multiplication
kernel that achieves performance on par with cuBLAS.
You will specifically learn about:<3A>h]<5D>h<11><>In this tutorial, you will write a 25-lines high-performance FP16 matrix multiplication
kernel that achieves performance on par with cuBLAS.
You will specifically learn about:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<>hh<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Khh<>hhubh <09> bullet_list<73><74><EFBFBD>)<29><>}<7D>(hhh]<5D>(h <09> list_item<65><6D><EFBFBD>)<29><>}<7D>(h<05>"Block-level matrix multiplications<6E>h]<5D>hm)<29><>}<7D>(hh<>h]<5D>h<11>"Block-level matrix multiplications<6E><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<>hh<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Khh<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(Nubh<62>)<29><>}<7D>(h<05>$Multi-dimensional pointer arithmetic<69>h]<5D>hm)<29><>}<7D>(hj h]<5D>h<11>$Multi-dimensional pointer arithmetic<69><63><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj hj ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Khjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(Nubh<62>)<29><>}<7D>(h<05>2Program re-ordering for improved L2 cache hit rate<74>h]<5D>hm)<29><>}<7D>(hj h]<5D>h<11>2Program re-ordering for improved L2 cache hit rate<74><65><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj hj"ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Khjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(Nubh<62>)<29><>}<7D>(h<05>Automatic performance tuning
<EFBFBD>h]<5D>hm)<29><>}<7D>(h<05>Automatic performance tuning<6E>h]<5D>h<11>Automatic performance tuning<6E><67><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj;hj9ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Khj5ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>bullet<65><74>-<2D>uh%h<>h&h'h(Khh<>hhubh )<29><>}<7D>(h<05>(GENERATED FROM PYTHON SOURCE LINES 15-42<34>h]<5D>h<11>(GENERATED FROM PYTHON SOURCE LINES 15-42<34><32><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjUubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hh<>hhh&h'h(K!ubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05> Motivations<6E>h]<5D>h<11> Motivations<6E><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjhhjfhhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hjchhh&h'h(K#ubhm)<29><>}<7D>(hX[Matrix multiplications are a key building block of most modern high-performance computing systems.
They are notoriously hard to optimize, hence their implementation is generally done by
hardware vendors themselves as part of so-called "kernel libraries" (e.g., cuBLAS).
Unfortunately, these libraries are often proprietary and cannot be easily customized
to accomodate the needs of modern deep learning workloads (e.g., fused activation functions).
In this tutorial, you will learn how to implement efficient matrix multiplications by
yourself with Triton, in a way that is easy to customize and extend.<2E>h]<5D>hX_Matrix multiplications are a key building block of most modern high-performance computing systems.
They are notoriously hard to optimize, hence their implementation is generally done by
hardware vendors themselves as part of so-called “kernel libraries” (e.g., cuBLAS).
Unfortunately, these libraries are often proprietary and cannot be easily customized
to accomodate the needs of modern deep learning workloads (e.g., fused activation functions).
In this tutorial, you will learn how to implement efficient matrix multiplications by
yourself with Triton, in a way that is easy to customize and extend.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjvhjthhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K$hjchhubhm)<29><>}<7D>(h<05><>Roughly speaking, the kernel that we will write will implement the following blocked
algorithm to multiply a (M, K) by a (K, N) matrix:<3A>h]<5D>h<11><>Roughly speaking, the kernel that we will write will implement the following blocked
algorithm to multiply a (M, K) by a (K, N) matrix:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K,hjchhubh <09> block_quote<74><65><EFBFBD>)<29><>}<7D>(hhh]<5D>h <09> literal_block<63><6B><EFBFBD>)<29><>}<7D>(hX<># do in parallel
for m in range(0, M, BLOCK_SIZE_M):
# do in parallel
for n in range(0, N, BLOCK_SIZE_N):
acc = zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=float32)
for k in range(0, K, BLOCK_SIZE_K):
a = A[m : m+BLOCK_SIZE_M, k : k+BLOCK_SIZE_K]
b = B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]
acc += dot(a, b)
C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;<3B>h]<5D>hX<># do in parallel
for m in range(0, M, BLOCK_SIZE_M):
# do in parallel
for n in range(0, N, BLOCK_SIZE_N):
acc = zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=float32)
for k in range(0, K, BLOCK_SIZE_K):
a = A[m : m+BLOCK_SIZE_M, k : k+BLOCK_SIZE_K]
b = B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]
acc += dot(a, b)
C[m : m+BLOCK_SIZE_M, n : n+BLOCK_SIZE_N] = acc;<3B><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$<24>force<63><65><EFBFBD>language<67><65>python<6F><6E>highlight_args<67>}<7D>uh%j<>h&h'h(K/hj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hjchhh&Nh(Nubhm)<29><>}<7D>(h<05>gwhere each iteration of the doubly-nested for-loop is performed by a dedicated Triton program instance.<2E>h]<5D>h<11>gwhere each iteration of the doubly-nested for-loop is performed by a dedicated Triton program instance.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K<hjchhubh )<29><>}<7D>(h<05>)GENERATED FROM PYTHON SOURCE LINES 44-137<33>h]<5D>h<11>)GENERATED FROM PYTHON SOURCE LINES 44-137<33><37><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hjchhh&h'h(K?ubeh}<7D>(h]<5D><> motivations<6E>ah]<5D>h]<5D><> motivations<6E>ah]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(K#ubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05>Compute Kernel<65>h]<5D>h<11>Compute Kernel<65><6C><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hj<>hhh&h'h(KAubhm)<29><>}<7D>(hXThe above algorithm is, actually, fairly straightforward to implement in Triton.
The main difficulty comes from the computation of the memory locations at which blocks
of :code:`A` and :code:`B` must be read in the inner loop. For that, we need
multi-dimensional pointer arithmetics.<2E>h]<5D>(h<11><>The above algorithm is, actually, fairly straightforward to implement in Triton.
The main difficulty comes from the computation of the memory locations at which blocks
of <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>The above algorithm is, actually, fairly straightforward to implement in Triton.
The main difficulty comes from the computation of the memory locations at which blocks
of <20>hj<>hhh&Nh(Nubh <09>literal<61><6C><EFBFBD>)<29><>}<7D>(h<05> :code:`A`<60>h]<5D>h<11>A<><41><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>A<>hj<>ubah}<7D>(h]<5D>h]<5D><>code<64>ah]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubh<11> and <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> and <20>hj<>hhh&Nh(Nubj<62>)<29><>}<7D>(h<05> :code:`B`<60>h]<5D>h<11>B<><42><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>B<>hjubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubh<11>Y must be read in the inner loop. For that, we need
multi-dimensional pointer arithmetics.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>Y must be read in the inner loop. For that, we need
multi-dimensional pointer arithmetics.<2E>hj<>hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(KChj<>hhubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05>Pointer Arithmetics<63>h]<5D>h<11>Pointer Arithmetics<63><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj$hj"hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hjhhh&h'h(KIubhm)<29><>}<7D>(hX4For a row-major 2D tensor :code:`X`, the memory location of :code:`X[i, j]` is given b
y :code:`&X[i, j] = X + i*stride_xi + j*stride_xj`.
Therefore, blocks of pointers for :code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]` and
:code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]` can be defined in pseudo-code as:<3A>h]<5D>(h<11>For a row-major 2D tensor <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>For a row-major 2D tensor <20>hj0hhh&Nh(Nubj<62>)<29><>}<7D>(h<05> :code:`X`<60>h]<5D>h<11>X<><58><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>X<>hj9ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj0ubh<11>, the memory location of <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>, the memory location of <20>hj0hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>:code:`X[i, j]`<60>h]<5D>h<11>X[i, j]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>X[i, j]<5D>hjMubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj0ubh<11> is given b
y <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> is given b
y <20>hj0hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>0:code:`&X[i, j] = X + i*stride_xi + j*stride_xj`<60>h]<5D>h<11>(&X[i, j] = X + i*stride_xi + j*stride_xj<78><6A><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>(&X[i, j] = X + i*stride_xi + j*stride_xj<78>hjaubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj0ubh<11>$.
Therefore, blocks of pointers for <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>$.
Therefore, blocks of pointers for <20>hj0hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>/:code:`A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]`<60>h]<5D>h<11>'A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>'A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K]<5D>hjuubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj0ubh<11> and
<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> and
<EFBFBD>hj0hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>1:code:`B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]`<60>h]<5D>h<11>)B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>)B[k : k+BLOCK_SIZE_K, n : n+BLOCK_SIZE_N]<5D>hj<>ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj0ubh<11>" can be defined in pseudo-code as:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>" can be defined in pseudo-code as:<3A>hj0hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(KKhjhhubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(hX&A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = a_ptr + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
&B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = b_ptr + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);<3B>h]<5D>hX&A[m : m+BLOCK_SIZE_M, k:k+BLOCK_SIZE_K] = a_ptr + (m : m+BLOCK_SIZE_M)[:, None]*A.stride(0) + (k : k+BLOCK_SIZE_K)[None, :]*A.stride(1);
&B[k : k+BLOCK_SIZE_K, n:n+BLOCK_SIZE_N] = b_ptr + (k : k+BLOCK_SIZE_K)[:, None]*B.stride(0) + (n : n+BLOCK_SIZE_N)[None, :]*B.stride(1);<3B><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>python<6F>j<EFBFBD>}<7D>uh%j<>h&h'h(KPhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hjhhh&Nh(Nubhm)<29><>}<7D>(h<05>dWhich means that pointers for blocks of A and B can be initialized (i.e., :code:`k=0`) in Triton as:<3A>h]<5D>(h<11>JWhich means that pointers for blocks of A and B can be initialized (i.e., <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>JWhich means that pointers for blocks of A and B can be initialized (i.e., <20>hj<>hhh&Nh(Nubj<62>)<29><>}<7D>(h<05> :code:`k=0`<60>h]<5D>h<11>k=0<><30><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>k=0<>hj<>ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubh<11>) in Triton as:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>) in Triton as:<3A>hj<>hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(KUhjhhubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(hX1offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
offs_k = tl.arange(0, BLOCK_SIZE_K)
a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)<29>h]<5D>hX1offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
offs_k = tl.arange(0, BLOCK_SIZE_K)
a_ptrs = a_ptr + (offs_am[:, None]*stride_am + offs_k [None, :]*stride_ak)
b_ptrs = b_ptr + (offs_k [:, None]*stride_bk + offs_bn[None, :]*stride_bn)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>python<6F>j<EFBFBD>}<7D>uh%j<>h&h'h(KWhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hjhhh&Nh(Nubhm)<29><>}<7D>(h<05>.And then updated in the inner loop as follows:<3A>h]<5D>h<11>.And then updated in the inner loop as follows:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K_hjhhubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(h<05>?pa += BLOCK_SIZE_K * stride_ak;
pb += BLOCK_SIZE_K * stride_bk;<3B>h]<5D>h<11>?pa += BLOCK_SIZE_K * stride_ak;
pb += BLOCK_SIZE_K * stride_bk;<3B><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>python<6F>j<EFBFBD>}<7D>uh%j<>h&h'h(Kahjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hjhhh&Nh(Nubeh}<7D>(h]<5D><>pointer-arithmetics<63>ah]<5D>h]<5D><>pointer arithmetics<63>ah]<5D>h!]<5D>uh%h<>hj<>hhh&h'h(KIubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05>L2 Cache Optimizations<6E>h]<5D>h<11>L2 Cache Optimizations<6E><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj,hj*hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hj'hhh&h'h(Khubhm)<29><>}<7D>(hX0As mentioned above, each program instance computes a :code:`[BLOCK_SIZE_M, BLOCK_SIZE_N]`
block of :code:`C`.
It is important to remember that the order in which these blocks are computed does
matter, since it affects the L2 cache hit rate of our program. and unfortunately, a
a simple row-major ordering<6E>h]<5D>(h<11>5As mentioned above, each program instance computes a <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>5As mentioned above, each program instance computes a <20>hj8hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>$:code:`[BLOCK_SIZE_M, BLOCK_SIZE_N]`<60>h]<5D>h<11>[BLOCK_SIZE_M, BLOCK_SIZE_N]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>[BLOCK_SIZE_M, BLOCK_SIZE_N]<5D>hjAubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj8ubh<11>
block of <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>
block of <20>hj8hhh&Nh(Nubj<62>)<29><>}<7D>(h<05> :code:`C`<60>h]<5D>h<11>C<><43><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>C<>hjUubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj8ubh<11><>.
It is important to remember that the order in which these blocks are computed does
matter, since it affects the L2 cache hit rate of our program. and unfortunately, a
a simple row-major ordering<6E><67><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>.
It is important to remember that the order in which these blocks are computed does
matter, since it affects the L2 cache hit rate of our program. and unfortunately, a
a simple row-major ordering<6E>hj8hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Kjhj'hhubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(h<05><>pid = triton.program_id(0);
grid_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M;
grid_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N;
pid_m = pid / grid_n;
pid_n = pid % grid_n;<3B>h]<5D>h<11><>pid = triton.program_id(0);
grid_m = (M + BLOCK_SIZE_M - 1) // BLOCK_SIZE_M;
grid_n = (N + BLOCK_SIZE_N - 1) // BLOCK_SIZE_N;
pid_m = pid / grid_n;
pid_n = pid % grid_n;<3B><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjrubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>Python<6F>j<EFBFBD>}<7D>uh%j<>h&h'h(Kphjoubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hj'hhh&Nh(Nubhm)<29><>}<7D>(h<05>is just not going to cut it.<2E>h]<5D>h<11>is just not going to cut it.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Kxhj'hhubhm)<29><>}<7D>(h<05><>One possible solution is to launch blocks in an order that promotes data reuse.
This can be done by 'super-grouping' blocks in groups of :code:`GROUP_M` rows before
switching to the next column:<3A>h]<5D>(h<11><>One possible solution is to launch blocks in an order that promotes data reuse.
This can be done by ‘super-grouping’ blocks in groups of <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>One possible solution is to launch blocks in an order that promotes data reuse.
This can be done by 'super-grouping' blocks in groups of <20>hj<>hhh&Nh(Nubj<62>)<29><>}<7D>(h<05>:code:`GROUP_M`<60>h]<5D>h<11>GROUP_M<5F><4D><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>GROUP_M<5F>hj<>ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubh<11>* rows before
switching to the next column:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>* rows before
switching to the next column:<3A>hj<>hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Kzhj'hhubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(hX&# program ID
pid = tl.program_id(axis=0)
# number of program ids along the M axis
num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
# number of programs ids along the N axis
num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
# number of programs in group
num_pid_in_group = GROUP_SIZE_M * num_pid_n
# id of the group this program is in
group_id = pid // num_pid_in_group
# row-id of the first program in the group
first_pid_m = group_id * GROUP_SIZE_M
# if `num_pid_m` isn't divisible by `GROUP_SIZE_M`, the last group is smaller
group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
# *within groups*, programs are ordered in a column-major order
# row-id of the program in the *launch grid*
pid_m = first_pid_m + (pid % group_size_m)
# col-id of the program in the *launch grid*
pid_n = (pid % num_pid_in_group) // group_size_m<5F>h]<5D>hX&# program ID
pid = tl.program_id(axis=0)
# number of program ids along the M axis
num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
# number of programs ids along the N axis
num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
# number of programs in group
num_pid_in_group = GROUP_SIZE_M * num_pid_n
# id of the group this program is in
group_id = pid // num_pid_in_group
# row-id of the first program in the group
first_pid_m = group_id * GROUP_SIZE_M
# if `num_pid_m` isn't divisible by `GROUP_SIZE_M`, the last group is smaller
group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
# *within groups*, programs are ordered in a column-major order
# row-id of the program in the *launch grid*
pid_m = first_pid_m + (pid % group_size_m)
# col-id of the program in the *launch grid*
pid_n = (pid % num_pid_in_group) // group_size_m<5F><6D><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>python<6F>j<EFBFBD>}<7D>uh%j<>h&h'h(K~hj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hj'hhh&Nh(Nubhm)<29><>}<7D>(hXFor example, in the following matmul where each matrix is 9 blocks by 9 blocks,
we can see that if we compute the output in row-major ordering, we need to load 90
blocks into SRAM to compute the first 9 output blocks, but if we do it in grouped
ordering, we only need to load 54 blocks.<2E>h]<5D>hXFor example, in the following matmul where each matrix is 9 blocks by 9 blocks,
we can see that if we compute the output in row-major ordering, we need to load 90
blocks into SRAM to compute the first 9 output blocks, but if we do it in grouped
ordering, we only need to load 54 blocks.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K<>hj'hhubj<62>)<29><>}<7D>(hhh]<5D>h <09>image<67><65><EFBFBD>)<29><>}<7D>(h<05>-.. image:: grouped_vs_row_major_ordering.png
<EFBFBD>h]<5D>h}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>uri<72><69>;getting-started/tutorials/grouped_vs_row_major_ordering.png<6E><67>
candidates<EFBFBD>}<7D><>*<2A>j<EFBFBD>suh%j<>hj<>h&h'h(Kubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hj'hhh&Nh(Nubhm)<29><>}<7D>(h<05><>In practice, this can improve the performance of our matrix multiplication kernel by
more than 10\% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).<2E>h]<5D>h<11><>In practice, this can improve the performance of our matrix multiplication kernel by
more than 10% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>In practice, this can improve the performance of our matrix multiplication kernel by
more than 10\% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).<2E>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(K<>hj'hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 139-142<34>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 139-142<34><32><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hj'hhh&h'h(K<>ubeh}<7D>(h]<5D><>l2-cache-optimizations<6E>ah]<5D>h]<5D><>l2 cache optimizations<6E>ah]<5D>h!]<5D>uh%h<>hj<>hhh&h'h(Khubeh}<7D>(h]<5D><>compute-kernel<65>ah]<5D>h]<5D><>compute kernel<65>ah]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(KAubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05> Final Result<6C>h]<5D>h<11> Final Result<6C><74><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj,hj*hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hj'hhh&h'h(K<>ubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 142-259<35>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 142-259<35><39><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj8ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
hj'hhh&h'h(K<>ubj<62>)<29><>}<7D>(hXCimport torch
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
import triton
import triton.language as tl
# %
# :code:`triton.jit`'ed functions can be auto-tuned by using the `triton.autotune`
# decorator, which consumes:
# - A list of :code:`triton.Config` objects that define different configurations of
# meta-parameters (e.g., BLOCK_SIZE_M) and compilation options (e.g., num_warps) to try
# - An autotuning *key* whose change in values will trigger evaluation of all the
# provided configs
@triton.autotune(
configs=[
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),
triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),
triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 32, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 32, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),
triton.Config({'BLOCK_SIZE_M': 32, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),
],
key=['M', 'N', 'K'],
)
@triton.jit
def matmul_kernel(
# Pointers to matrices
a_ptr, b_ptr, c_ptr,
# Matrix dimensions
M, N, K,
# The stride variables represent how much to increase the ptr by when moving by 1
# element in a particular dimension. E.g. stride_am is how much to increase a_ptr
# by to get the element one row down (A has M rows)
stride_am, stride_ak,
stride_bk, stride_bn,
stride_cm, stride_cn,
# Meta-parameters
BLOCK_SIZE_M: tl.constexpr, BLOCK_SIZE_N: tl.constexpr, BLOCK_SIZE_K: tl.constexpr,
GROUP_SIZE_M: tl.constexpr,
ACTIVATION: tl.constexpr,
):
"""Kernel for computing the matmul C = A x B.
A has shape (M, K), B has shape (K, N) and C has shape (M, N)
"""
# -----------------------------------------------------------
# Map program ids `pid` to the block of C it should compute.
# This is done in a grouped ordering to promote L2 data reuse
# See above `L2 Cache Optimizations` section for details
pid = tl.program_id(axis=0)
num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
num_pid_in_group = GROUP_SIZE_M * num_pid_n
group_id = pid // num_pid_in_group
first_pid_m = group_id * GROUP_SIZE_M
group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
pid_m = first_pid_m + (pid % group_size_m)
pid_n = (pid % num_pid_in_group) // group_size_m
# ----------------------------------------------------------
# Create pointers for the first blocks of A and B.
# We will advance this pointer as we move in the K direction
# and accumulate
# a_ptrs is a block of [BLOCK_SIZE_M, BLOCK_SIZE_K] pointers
# b_ptrs is a block of [BLOCK_SIZE_K, BLOCK_SIZE_n] pointers
# see above `Pointer Arithmetics` section for details
offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
offs_k = tl.arange(0, BLOCK_SIZE_K)
a_ptrs = a_ptr + (offs_am[:, None] * stride_am + offs_k[None, :] * stride_ak)
b_ptrs = b_ptr + (offs_k[:, None] * stride_bk + offs_bn[None, :] * stride_bn)
# -----------------------------------------------------------
# Iterate to compute a block of the C matrix
# We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block
# of fp32 values for higher accuracy.
# `accumulator` will be converted back to fp16 after the loop
accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
for k in range(0, K, BLOCK_SIZE_K):
# Note that for simplicity, we don't apply a mask here.
# This means that if K is not a multiple of BLOCK_SIZE_K,
# this will access out-of-bounds memory and produce an
# error or (worse!) incorrect results.
a = tl.load(a_ptrs)
b = tl.load(b_ptrs)
# We accumulate along the K dimension
accumulator += tl.dot(a, b)
# Advance the ptrs to the next K block
a_ptrs += BLOCK_SIZE_K * stride_ak
b_ptrs += BLOCK_SIZE_K * stride_bk
# you can fuse arbitrary activation functions here
# while the accumulator is still in FP32!
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
if ACTIVATION == "leaky_relu":
accumulator = leaky_relu(accumulator)
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
c = accumulator.to(tl.float16)
# -----------------------------------------------------------
# Write back the block of the output matrix C
offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
c_ptrs = c_ptr + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]
c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
tl.store(c_ptrs, c, mask=c_mask)
# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`
@triton.jit
def leaky_relu(x):
x = x + 1
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
return tl.where(x >= 0, x, 0.01 * x)<29>h]<5D>hXCimport torch
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
import triton
import triton.language as tl
# %
# :code:`triton.jit`'ed functions can be auto-tuned by using the `triton.autotune`
# decorator, which consumes:
# - A list of :code:`triton.Config` objects that define different configurations of
# meta-parameters (e.g., BLOCK_SIZE_M) and compilation options (e.g., num_warps) to try
# - An autotuning *key* whose change in values will trigger evaluation of all the
# provided configs
@triton.autotune(
configs=[
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),
triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=3, num_warps=8),
triton.Config({'BLOCK_SIZE_M': 256, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 256, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 128, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 128, 'BLOCK_SIZE_N': 32, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=4, num_warps=4),
triton.Config({'BLOCK_SIZE_M': 64, 'BLOCK_SIZE_N': 32, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),
triton.Config({'BLOCK_SIZE_M': 32, 'BLOCK_SIZE_N': 64, 'BLOCK_SIZE_K': 32, 'GROUP_SIZE_M': 8}, num_stages=5, num_warps=2),
],
key=['M', 'N', 'K'],
)
@triton.jit
def matmul_kernel(
# Pointers to matrices
a_ptr, b_ptr, c_ptr,
# Matrix dimensions
M, N, K,
# The stride variables represent how much to increase the ptr by when moving by 1
# element in a particular dimension. E.g. stride_am is how much to increase a_ptr
# by to get the element one row down (A has M rows)
stride_am, stride_ak,
stride_bk, stride_bn,
stride_cm, stride_cn,
# Meta-parameters
BLOCK_SIZE_M: tl.constexpr, BLOCK_SIZE_N: tl.constexpr, BLOCK_SIZE_K: tl.constexpr,
GROUP_SIZE_M: tl.constexpr,
ACTIVATION: tl.constexpr,
):
"""Kernel for computing the matmul C = A x B.
A has shape (M, K), B has shape (K, N) and C has shape (M, N)
"""
# -----------------------------------------------------------
# Map program ids `pid` to the block of C it should compute.
# This is done in a grouped ordering to promote L2 data reuse
# See above `L2 Cache Optimizations` section for details
pid = tl.program_id(axis=0)
num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
num_pid_n = tl.cdiv(N, BLOCK_SIZE_N)
num_pid_in_group = GROUP_SIZE_M * num_pid_n
group_id = pid // num_pid_in_group
first_pid_m = group_id * GROUP_SIZE_M
group_size_m = min(num_pid_m - first_pid_m, GROUP_SIZE_M)
pid_m = first_pid_m + (pid % group_size_m)
pid_n = (pid % num_pid_in_group) // group_size_m
# ----------------------------------------------------------
# Create pointers for the first blocks of A and B.
# We will advance this pointer as we move in the K direction
# and accumulate
# a_ptrs is a block of [BLOCK_SIZE_M, BLOCK_SIZE_K] pointers
# b_ptrs is a block of [BLOCK_SIZE_K, BLOCK_SIZE_n] pointers
# see above `Pointer Arithmetics` section for details
offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
offs_k = tl.arange(0, BLOCK_SIZE_K)
a_ptrs = a_ptr + (offs_am[:, None] * stride_am + offs_k[None, :] * stride_ak)
b_ptrs = b_ptr + (offs_k[:, None] * stride_bk + offs_bn[None, :] * stride_bn)
# -----------------------------------------------------------
# Iterate to compute a block of the C matrix
# We accumulate into a `[BLOCK_SIZE_M, BLOCK_SIZE_N]` block
# of fp32 values for higher accuracy.
# `accumulator` will be converted back to fp16 after the loop
accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
for k in range(0, K, BLOCK_SIZE_K):
# Note that for simplicity, we don't apply a mask here.
# This means that if K is not a multiple of BLOCK_SIZE_K,
# this will access out-of-bounds memory and produce an
# error or (worse!) incorrect results.
a = tl.load(a_ptrs)
b = tl.load(b_ptrs)
# We accumulate along the K dimension
accumulator += tl.dot(a, b)
# Advance the ptrs to the next K block
a_ptrs += BLOCK_SIZE_K * stride_ak
b_ptrs += BLOCK_SIZE_K * stride_bk
# you can fuse arbitrary activation functions here
# while the accumulator is still in FP32!
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
if ACTIVATION == "leaky_relu":
accumulator = leaky_relu(accumulator)
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
c = accumulator.to(tl.float16)
# -----------------------------------------------------------
# Write back the block of the output matrix C
offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
c_ptrs = c_ptr + stride_cm * offs_cm[:, None] + stride_cn * offs_cn[None, :]
c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
tl.store(c_ptrs, c, mask=c_mask)
# we can fuse `leaky_relu` by providing it as an `ACTIVATION` meta-parameter in `_matmul`
@triton.jit
def leaky_relu(x):
x = x + 1
return tl.where(x >= 0, x, 0.01 * x)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjFubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>default<6C>j<EFBFBD>}<7D>uh%j<>h&h'h(K<>hj'hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 260-262<36>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 260-262<36><32><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjVubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hj'hhh&h'h(M%ubhm)<29><>}<7D>(h<05><>We can now create a convenience wrapper function that only takes two input tensors
and (1) checks any shape constraint; (2) allocates the output; (3) launches the above kernel<65>h]<5D>h<11><>We can now create a convenience wrapper function that only takes two input tensors
and (1) checks any shape constraint; (2) allocates the output; (3) launches the above kernel<65><6C><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjfhjdhhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(M&hj'hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 262-291<39>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 262-291<39><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjrubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
hj'hhh&h'h(M*ubj<62>)<29><>}<7D>(hXkdef matmul(a, b, activation=""):
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
# checks constraints
assert a.shape[1] == b.shape[0], "incompatible dimensions"
assert a.is_contiguous(), "matrix A must be contiguous"
assert b.is_contiguous(), "matrix B must be contiguous"
M, K = a.shape
K, N = b.shape
assert (
K % 32 == 0
), "We don't check memory-out-of-bounds with K so K must be divisible by BLOCK_SIZE_K"
# allocates output
c = torch.empty((M, N), device=a.device, dtype=a.dtype)
# 1D launch kernel where each block gets its own program.
grid = lambda META: (
triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),
)
matmul_kernel[grid](
a, b, c,
M, N, K,
a.stride(0), a.stride(1),
b.stride(0), b.stride(1),
c.stride(0), c.stride(1),
ACTIVATION=activation,
)
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
return c<>h]<5D>hXkdef matmul(a, b, activation=""):
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
# checks constraints
assert a.shape[1] == b.shape[0], "incompatible dimensions"
assert a.is_contiguous(), "matrix A must be contiguous"
assert b.is_contiguous(), "matrix B must be contiguous"
M, K = a.shape
K, N = b.shape
assert (
K % 32 == 0
), "We don't check memory-out-of-bounds with K so K must be divisible by BLOCK_SIZE_K"
# allocates output
c = torch.empty((M, N), device=a.device, dtype=a.dtype)
# 1D launch kernel where each block gets its own program.
grid = lambda META: (
triton.cdiv(M, META['BLOCK_SIZE_M']) * triton.cdiv(N, META['BLOCK_SIZE_N']),
)
matmul_kernel[grid](
a, b, c,
M, N, K,
a.stride(0), a.stride(1),
b.stride(0), b.stride(1),
c.stride(0), c.stride(1),
ACTIVATION=activation,
)
return c<><63><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>default<6C>j<EFBFBD>}<7D>uh%j<>h&h'h(M+hj'hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 292-296<39>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 292-296<39><36><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hj'hhh&h'h(MRubeh}<7D>(h]<5D><> final-result<6C>ah]<5D>h]<5D><> final result<6C>ah]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(K<>ubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05> Unit Test<73>h]<5D>h<11> Unit Test<73><74><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hj<>hhh&h'h(MTubhm)<29><>}<7D>(h<05>kWe can test our custom matrix multiplication operation against a native torch implementation (i.e., cuBLAS)<29>h]<5D>h<11>kWe can test our custom matrix multiplication operation against a native torch implementation (i.e., cuBLAS)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(MVhj<>hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 296-309<30>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 296-309<30><39><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hj<>hhh&h'h(MYubj<62>)<29><>}<7D>(hX<>torch.manual_seed(0)
a = torch.randn((512, 512), device='cuda', dtype=torch.float16)
b = torch.randn((512, 512), device='cuda', dtype=torch.float16)
triton_output = matmul(a, b)
torch_output = torch.matmul(a, b)
print(f"triton_output={triton_output}")
print(f"torch_output={torch_output}")
if triton.testing.allclose(triton_output, torch_output):
print("✅ Triton and Torch match")
else:
print("❌ Triton and Torch differ")<29>h]<5D>hX<>torch.manual_seed(0)
a = torch.randn((512, 512), device='cuda', dtype=torch.float16)
b = torch.randn((512, 512), device='cuda', dtype=torch.float16)
triton_output = matmul(a, b)
torch_output = torch.matmul(a, b)
print(f"triton_output={triton_output}")
print(f"torch_output={torch_output}")
if triton.testing.allclose(triton_output, torch_output):
print("✅ Triton and Torch match")
else:
print("❌ Triton and Torch differ")<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>default<6C>j<EFBFBD>}<7D>uh%j<>h&h'h(MZhj<>hhubhm)<29><>}<7D>(h<05>Out:<3A>h]<5D>h<11>Out:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>h]<5D><>sphx-glr-script-out<75>ah]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Mohj<>hhubj<62>)<29><>}<7D>(hX;triton_output=tensor([[ 1.1045, -36.9688, 31.4688, ..., -11.3984, 24.4531, -32.3438],
[ 6.3555, -19.6094, 34.0938, ..., -5.8945, 5.2891, 6.8867],
[-32.0625, 5.9492, 15.3984, ..., -21.3906, -23.9844, -10.1328],
...,
[ -5.7031, 7.4492, 8.2656, ..., -10.6953, -40.0000, 17.7500],
[ 25.5000, 24.3281, -8.4688, ..., -18.9375, 32.5312, -29.9219],
[ -5.3477, 4.9844, 11.8906, ..., 5.5898, 6.4023, -17.3125]],
device='cuda:0', dtype=torch.float16)
torch_output=tensor([[ 1.1045, -36.9688, 31.4688, ..., -11.3906, 24.4531, -32.3438],
[ 6.3516, -19.6094, 34.0938, ..., -5.8906, 5.2812, 6.8828],
[-32.0625, 5.9531, 15.3984, ..., -21.4062, -23.9844, -10.1328],
...,
[ -5.7070, 7.4492, 8.2656, ..., -10.6953, -40.0000, 17.7500],
[ 25.5000, 24.3438, -8.4609, ..., -18.9375, 32.5312, -29.9219],
[ -5.3477, 4.9805, 11.8828, ..., 5.5859, 6.4023, -17.3125]],
device='cuda:0', dtype=torch.float16)
✅ Triton and Torch match<63>h]<5D>hX;triton_output=tensor([[ 1.1045, -36.9688, 31.4688, ..., -11.3984, 24.4531, -32.3438],
[ 6.3555, -19.6094, 34.0938, ..., -5.8945, 5.2891, 6.8867],
[-32.0625, 5.9492, 15.3984, ..., -21.3906, -23.9844, -10.1328],
...,
[ -5.7031, 7.4492, 8.2656, ..., -10.6953, -40.0000, 17.7500],
[ 25.5000, 24.3281, -8.4688, ..., -18.9375, 32.5312, -29.9219],
[ -5.3477, 4.9844, 11.8906, ..., 5.5898, 6.4023, -17.3125]],
device='cuda:0', dtype=torch.float16)
torch_output=tensor([[ 1.1045, -36.9688, 31.4688, ..., -11.3906, 24.4531, -32.3438],
[ 6.3516, -19.6094, 34.0938, ..., -5.8906, 5.2812, 6.8828],
[-32.0625, 5.9531, 15.3984, ..., -21.4062, -23.9844, -10.1328],
...,
[ -5.7070, 7.4492, 8.2656, ..., -10.6953, -40.0000, 17.7500],
[ 25.5000, 24.3438, -8.4609, ..., -18.9375, 32.5312, -29.9219],
[ -5.3477, 4.9805, 11.8828, ..., 5.5859, 6.4023, -17.3125]],
device='cuda:0', dtype=torch.float16)
✅ Triton and Torch match<63><68><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>none<6E>j<EFBFBD>}<7D>uh%j<>h&h'h(Mqhj<>hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 310-316<31>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 310-316<31><36><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
hj<>hhh&h'h(M<>ubeh}<7D>(h]<5D><> unit-test<73>ah]<5D>h]<5D><> unit test<73>ah]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(MTubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05> Benchmark<72>h]<5D>h<11> Benchmark<72><6B><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjhjhhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hjhhh&h'h(M<>ubh<62>)<29><>}<7D>(hhh]<5D>(h<>)<29><>}<7D>(h<05>Square Matrix Performance<63>h]<5D>h<11>Square Matrix Performance<63><65><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj.hj,hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%h<>hj)hhh&h'h(M<>ubhm)<29><>}<7D>(h<05><>We can now compare the performance of our kernel against that of cuBLAS. Here we focus on square matrices, but feel free to arrange this script as you wish to benchmark any other matrix shape.<2E>h]<5D>h<11><>We can now compare the performance of our kernel against that of cuBLAS. Here we focus on square matrices, but feel free to arrange this script as you wish to benchmark any other matrix shape.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<hj:hhh&Nh(Nubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(M<>hj)hhubh )<29><>}<7D>(h<05>*GENERATED FROM PYTHON SOURCE LINES 316-357<35>h]<5D>h<11>*GENERATED FROM PYTHON SOURCE LINES 316-357<35><37><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjHubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$uh%h
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
hj)hhh&h'h(M<>ubj<62>)<29><>}<7D>(hX<>@triton.testing.perf_report(
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
triton.testing.Benchmark(
x_names=['M', 'N', 'K'], # argument names to use as an x-axis for the plot
x_vals=[
128 * i for i in range(2, 33)
], # different possible values for `x_name`
line_arg='provider', # argument name whose value corresponds to a different line in the plot
# possible values for `line_arg``
line_vals=['cublas', 'cublas + relu', 'triton', 'triton + relu'],
# label name for the lines
line_names=["cuBLAS", "cuBLAS (+ torch.nn.LeakyReLU)", "Triton", "Triton (+ LeakyReLU)"],
# line styles
styles=[('green', '-'), ('green', '--'), ('blue', '-'), ('blue', '--')],
ylabel="TFLOPS", # label name for the y-axis
plot_name="matmul-performance", # name for the plot. Used also as a file name for saving the plot.
args={},
)
)
def benchmark(M, N, K, provider):
a = torch.randn((M, K), device='cuda', dtype=torch.float16)
b = torch.randn((K, N), device='cuda', dtype=torch.float16)
if provider == 'cublas':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: torch.matmul(a, b))
if provider == 'triton':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: matmul(a, b))
if provider == 'cublas + relu':
torch_relu = torch.nn.ReLU(inplace=True)
ms, min_ms, max_ms = triton.testing.do_bench(
lambda: torch_relu(torch.matmul(a, b))
)
if provider == 'triton + relu':
ms, min_ms, max_ms = triton.testing.do_bench(
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
lambda: matmul(a, b, activation="leaky_relu")
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
)
perf = lambda ms: 2 * M * N * K * 1e-12 / (ms * 1e-3)
return perf(ms), perf(max_ms), perf(min_ms)
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
benchmark.run(show_plots=True, print_data=True)<29>h]<5D>hX<>@triton.testing.perf_report(
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
triton.testing.Benchmark(
x_names=['M', 'N', 'K'], # argument names to use as an x-axis for the plot
x_vals=[
128 * i for i in range(2, 33)
], # different possible values for `x_name`
line_arg='provider', # argument name whose value corresponds to a different line in the plot
# possible values for `line_arg``
line_vals=['cublas', 'cublas + relu', 'triton', 'triton + relu'],
# label name for the lines
line_names=["cuBLAS", "cuBLAS (+ torch.nn.LeakyReLU)", "Triton", "Triton (+ LeakyReLU)"],
# line styles
styles=[('green', '-'), ('green', '--'), ('blue', '-'), ('blue', '--')],
ylabel="TFLOPS", # label name for the y-axis
plot_name="matmul-performance", # name for the plot. Used also as a file name for saving the plot.
args={},
)
)
def benchmark(M, N, K, provider):
a = torch.randn((M, K), device='cuda', dtype=torch.float16)
b = torch.randn((K, N), device='cuda', dtype=torch.float16)
if provider == 'cublas':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: torch.matmul(a, b))
if provider == 'triton':
ms, min_ms, max_ms = triton.testing.do_bench(lambda: matmul(a, b))
if provider == 'cublas + relu':
torch_relu = torch.nn.ReLU(inplace=True)
ms, min_ms, max_ms = triton.testing.do_bench(
lambda: torch_relu(torch.matmul(a, b))
)
if provider == 'triton + relu':
ms, min_ms, max_ms = triton.testing.do_bench(
[GH-PAGES] Updated website
2022-07-23 00:49:35 +00:00
lambda: matmul(a, b, activation="leaky_relu")
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
)
perf = lambda ms: 2 * M * N * K * 1e-12 / (ms * 1e-3)
return perf(ms), perf(max_ms), perf(min_ms)
benchmark.run(show_plots=True, print_data=True)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjVubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>default<6C>j<EFBFBD>}<7D>uh%j<>h&h'h(M<>hj)hhubj<62>)<29><>}<7D>(h<05><>.. image:: /getting-started/tutorials/images/sphx_glr_03-matrix-multiplication_001.png
:alt: 03 matrix multiplication
:class: sphx-glr-single-img
<EFBFBD>h]<5D>h}<7D>(h]<5D>h]<5D><>sphx-glr-single-img<6D>ah]<5D>h]<5D>h!]<5D><>alt<6C><74>03 matrix multiplication<6F><6E>uri<72><69>Jgetting-started/tutorials/images/sphx_glr_03-matrix-multiplication_001.png<6E>j<EFBFBD>}<7D>j<EFBFBD>jtsuh%j<>hj)hhh&h'h(Nubhm)<29><>}<7D>(h<05>Out:<3A>h]<5D>h<11>Out:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjxhjvhhh&Nh(Nubah}<7D>(h]<5D>h]<5D><>sphx-glr-script-out<75>ah]<5D>h]<5D>h!]<5D>uh%hlh&h'h(M<>hj)hhubj<62>)<29><>}<7D>(hX<>matmul-performance:
M cuBLAS ... Triton Triton (+ LeakyReLU)
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
0 256.0 2.730667 ... 3.276800 3.276800
1 384.0 7.372800 ... 7.899428 7.899428
2 512.0 14.563555 ... 16.384000 15.420235
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
3 640.0 22.260869 ... 24.380953 24.380953
[GH-PAGES] Updated website
2022-07-27 00:51:15 +00:00
4 768.0 32.768000 ... 35.389441 34.028308
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
5 896.0 39.025776 ... 40.140799 39.025776
[GH-PAGES] Updated website
2022-08-14 00:49:28 +00:00
6 1024.0 49.932191 ... 53.773130 52.428801
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
7 1152.0 44.566925 ... 48.161033 47.396572
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
8 1280.0 51.200001 ... 57.690139 57.690139
[GH-PAGES] Updated website
2022-08-17 00:49:36 +00:00
9 1408.0 64.138541 ... 69.009825 68.147202
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
10 1536.0 80.430545 ... 81.355034 79.526831
11 1664.0 63.372618 ... 63.372618 62.492442
12 1792.0 72.983276 ... 73.943582 59.467852
13 1920.0 69.467336 ... 71.257735 71.257735
14 2048.0 73.584279 ... 78.398206 77.314362
15 2176.0 83.500614 ... 87.876193 86.367588
16 2304.0 68.251065 ... 78.064941 77.307030
17 2432.0 71.305746 ... 86.444504 84.621881
18 2560.0 78.019048 ... 82.331658 81.108913
19 2688.0 83.369354 ... 90.966561 89.464755
20 2816.0 79.587973 ... 84.197315 83.074685
21 2944.0 81.967162 ... 83.060049 82.784108
22 3072.0 82.420822 ... 87.924073 85.019675
23 3200.0 84.768213 ... 96.458178 95.238096
24 3328.0 83.905938 ... 85.096096 84.298943
25 3456.0 82.604067 ... 91.304157 87.442050
26 3584.0 87.211821 ... 98.375705 91.656871
27 3712.0 84.874549 ... 89.035062 86.044224
28 3840.0 84.744825 ... 93.484358 84.484863
29 3968.0 92.582651 ... 84.976733 91.472214
30 4096.0 86.256445 ... 88.475759 90.230403
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
[31 rows x 5 columns]<5D>h]<5D>hX<>matmul-performance:
M cuBLAS ... Triton Triton (+ LeakyReLU)
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
0 256.0 2.730667 ... 3.276800 3.276800
1 384.0 7.372800 ... 7.899428 7.899428
2 512.0 14.563555 ... 16.384000 15.420235
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
3 640.0 22.260869 ... 24.380953 24.380953
[GH-PAGES] Updated website
2022-07-27 00:51:15 +00:00
4 768.0 32.768000 ... 35.389441 34.028308
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
5 896.0 39.025776 ... 40.140799 39.025776
[GH-PAGES] Updated website
2022-08-14 00:49:28 +00:00
6 1024.0 49.932191 ... 53.773130 52.428801
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
7 1152.0 44.566925 ... 48.161033 47.396572
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
8 1280.0 51.200001 ... 57.690139 57.690139
[GH-PAGES] Updated website
2022-08-17 00:49:36 +00:00
9 1408.0 64.138541 ... 69.009825 68.147202
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
10 1536.0 80.430545 ... 81.355034 79.526831
11 1664.0 63.372618 ... 63.372618 62.492442
12 1792.0 72.983276 ... 73.943582 59.467852
13 1920.0 69.467336 ... 71.257735 71.257735
14 2048.0 73.584279 ... 78.398206 77.314362
15 2176.0 83.500614 ... 87.876193 86.367588
16 2304.0 68.251065 ... 78.064941 77.307030
17 2432.0 71.305746 ... 86.444504 84.621881
18 2560.0 78.019048 ... 82.331658 81.108913
19 2688.0 83.369354 ... 90.966561 89.464755
20 2816.0 79.587973 ... 84.197315 83.074685
21 2944.0 81.967162 ... 83.060049 82.784108
22 3072.0 82.420822 ... 87.924073 85.019675
23 3200.0 84.768213 ... 96.458178 95.238096
24 3328.0 83.905938 ... 85.096096 84.298943
25 3456.0 82.604067 ... 91.304157 87.442050
26 3584.0 87.211821 ... 98.375705 91.656871
27 3712.0 84.874549 ... 89.035062 86.044224
28 3840.0 84.744825 ... 93.484358 84.484863
29 3968.0 92.582651 ... 84.976733 91.472214
30 4096.0 86.256445 ... 88.475759 90.230403
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
[GH-PAGES] Updated website
2022-08-18 00:48:52 +00:00
[31 rows x 5 columns]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>j<EFBFBD>ah]<5D>h]<5D>h!]<5D>h#h$j<><00>j<EFBFBD><00>none<6E>j<EFBFBD>}<7D>uh%j<>h&h'h(M<>hj)hhubhm)<29><>}<7D>(h<05>B**Total running time of the script:** ( 6 minutes 18.355 seconds)<29>h]<5D>(h <09>strong<6E><67><EFBFBD>)<29><>}<7D>(h<05>%**Total running time of the script:**<2A>h]<5D>h<11>!Total running time of the script:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubh<11> ( 6 minutes 18.355 seconds)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> ( 6 minutes 18.355 seconds)<29>hj<>hhh&Nh(Nubeh}<7D>(h]<5D>h]<5D><>sphx-glr-timing<6E>ah]<5D>h]<5D>h!]<5D>uh%hlh&h'h(M<>hj)hhubh<62>)<29><>}<7D>(h<05>L.. _sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py:<3A>h]<5D>h}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>hČGsphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py<70>uh%h<>h(M<>hj)hhh&h'ubhb)<29><>}<7D>(hhh]<5D>h <09> container<65><72><EFBFBD>)<29><>}<7D>(hXI.. container:: sphx-glr-download sphx-glr-download-python
[GH-PAGES] Updated website
2022-06-05 21:05:02 +00:00
:download:`Download Python source code: 03-matrix-multiplication.py <03-matrix-multiplication.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: 03-matrix-multiplication.ipynb <03-matrix-multiplication.ipynb>`<60>h]<5D>(j<>)<29><>}<7D>(h<05>b:download:`Download Python source code: 03-matrix-multiplication.py <03-matrix-multiplication.py>`<60>h]<5D>hm)<29><>}<7D>(hj<>h]<5D>h<00>download_reference<63><65><EFBFBD>)<29><>}<7D>(hj<>h]<5D>j<EFBFBD>)<29><>}<7D>(hj<>h]<5D>h<11>8Download Python source code: 03-matrix-multiplication.py<70><79><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h]<5D>h]<5D>(h<><68>download<61>eh]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>refdoc<6F>h<EFBFBD><68> refdomain<69>h<06>reftype<70>j<EFBFBD><00> refexplicit<69><74><EFBFBD>refwarn<72><6E>h<EFBFBD><68>03-matrix-multiplication.py<70><79>filename<6D><65><d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py<70>uh%j<>h&h'h(Mhj<>ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(Mhj<>ubah}<7D>(h]<5D>h]<5D>(<28>sphx-glr-download<61><64>sphx-glr-download-python<6F>eh]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubj<62>)<29><>}<7D>(h<05>f:download:`Download Jupyter notebook: 03-matrix-multiplication.ipynb <03-matrix-multiplication.ipynb>`<60>h]<5D>hm)<29><>}<7D>(hjh]<5D>j<EFBFBD>)<29><>}<7D>(hjh]<5D>j<EFBFBD>)<29><>}<7D>(hjh]<5D>h<11>9Download Jupyter notebook: 03-matrix-multiplication.ipynb<6E><62><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj ubah}<7D>(h]<5D>h]<5D>(h<><68>download<61>eh]<5D>h]<5D>h!]<5D>uh%j<>hjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>refdoc<6F>h<EFBFBD><68> refdomain<69>h<06>reftype<70>j<00> refexplicit<69><74><EFBFBD>refwarn<72><6E>h<EFBFBD><68>03-matrix-multiplication.ipynb<6E>j<EFBFBD><00>?b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb<6E>uh%j<>h&h'h(M hjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlh&h'h(M hj<>ubah}<7D>(h]<5D>h]<5D>(<28>sphx-glr-download<61><64>sphx-glr-download-jupyter<65>eh]<5D>h]<5D>h!]<5D>uh%j<>hj<>ubeh}<7D>(h]<5D>h]<5D>(<28>sphx-glr-footer<65><72>class<73><73>sphx-glr-footer-example<6C>eh]<5D>h]<5D>h!]<5D>uh%j<>hj<>hhh&Nh(Nubah}<7D>(h]<5D>j<EFBFBD>ah]<5D>h]<5D><>Gsphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py<70>ah]<5D>h!]<5D>h<EFBFBD><68>html<6D>uh%hahhh&h'h(M<>hj)<00>expect_referenced_by_name<6D>}<7D>j?j<>s<>expect_referenced_by_id<69>}<7D>j<EFBFBD>j<>subhb)<29><>}<7D>(hhh]<5D>hm)<29><>}<7D>(h<05>I`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_<>h]<5D>(h <09> reference<63><65><EFBFBD>)<29><>}<7D>(hjLh]<5D>h<11>#Gallery generated by Sphinx-Gallery<72><79><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>#Gallery generated by Sphinx-Gallery<72>hjPubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>name<6D><65>#Gallery generated by Sphinx-Gallery<72><79>refuri<72><69> https://sphinx-gallery.github.io<69>uh%jNhjJubh<62>)<29><>}<7D>(h<05># <https://sphinx-gallery.github.io><3E>h]<5D>h}<7D>(h]<5D><>#gallery-generated-by-sphinx-gallery<72>ah]<5D>h]<5D><>#gallery generated by sphinx-gallery<72>ah]<5D>h!]<5D><>refuri<72>jauh%h<><68>
referenced<EFBFBD>KhjJubeh}<7D>(h]<5D>h]<5D><>sphx-glr-signature<72>ah]<5D>h]<5D>h!]<5D>uh%hlh&h'h(MhjGhhubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>h<EFBFBD><68>html<6D>uh%hahhh&h'h(Mhj)ubeh}<7D>(h]<5D><>square-matrix-performance<63>ah]<5D>h]<5D><>square matrix performance<63>ah]<5D>h!]<5D>uh%h<>hjhhh&h'h(M<>ubeh}<7D>(h]<5D><> benchmark<72>ah]<5D>h]<5D><> benchmark<72>ah]<5D>h!]<5D>uh%h<>hh<>hhh&h'h(M<>ubeh}<7D>(h]<5D>(<28>matrix-multiplication<6F>h<EFBFBD>eh]<5D><>sphx-glr-example-title<6C>ah]<5D>(<28>matrix multiplication<6F><6E>>sphx_glr_getting-started_tutorials_03-matrix-multiplication.py<70>eh]<5D>h!]<5D>uh%h<>hhhhh&h'h(KjC}<7D>j<EFBFBD>h<>sjE}<7D>h<EFBFBD>h<EFBFBD>subeh}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>source<63>h'uh%h<01>current_source<63>N<EFBFBD> current_line<6E>N<EFBFBD>settings<67><73>docutils.frontend<6E><64>Values<65><73><EFBFBD>)<29><>}<7D>(h<>N<EFBFBD> generator<6F>N<EFBFBD> datestamp<6D>N<EFBFBD> source_link<6E>N<EFBFBD>
source_url<EFBFBD>N<EFBFBD> toc_backlinks<6B><73>entry<72><79>footnote_backlinks<6B>K<01> sectnum_xform<72>K<01>strip_comments<74>N<EFBFBD>strip_elements_with_classes<65>N<EFBFBD> strip_classes<65>N<EFBFBD> report_level<65>K<02>
halt_level<EFBFBD>K<05>exit_status_level<65>K<05>debug<75>N<EFBFBD>warning_stream<61>N<EFBFBD> traceback<63><6B><EFBFBD>input_encoding<6E><67> utf-8-sig<69><67>input_encoding_error_handler<65><72>strict<63><74>output_encoding<6E><67>utf-8<><38>output_encoding_error_handler<65>j<EFBFBD><00>error_encoding<6E><67>utf-8<><38>error_encoding_error_handler<65><72>backslashreplace<63><65> language_code<64><65>en<65><6E>record_dependencies<65>N<EFBFBD>config<69>N<EFBFBD> id_prefix<69>h<06>auto_id_prefix<69><78>id<69><64> dump_settings<67>N<EFBFBD>dump_internals<6C>N<EFBFBD>dump_transforms<6D>N<EFBFBD>dump_pseudo_xml<6D>N<EFBFBD>expose_internals<6C>N<EFBFBD>strict_visitor<6F>N<EFBFBD>_disable_config<69>N<EFBFBD>_source<63>h'<27> _destination<6F>N<EFBFBD> _config_files<65>]<5D><>pep_references<65>N<EFBFBD> pep_base_url<72><6C> https://www.python.org/dev/peps/<2F><>pep_file_url_template<74><65>pep-%04d<34><64>rfc_references<65>N<EFBFBD> rfc_base_url<72><6C>https://tools.ietf.org/html/<2F><> tab_width<74>K<08>trim_footnote_reference_space<63><65><EFBFBD>file_insertion_enabled<65><64><EFBFBD> raw_enabled<65>K<01>syntax_highlight<68><74>long<6E><67> smart_quotes<65><73><EFBFBD>smartquotes_locales<65>]<5D><>character_level_inline_markup<75><70><EFBFBD>doctitle_xform<72><6D><EFBFBD> docinfo_xform<72>K<01>sectsubtitle_xform<72><6D><EFBFBD>embed_stylesheet<65><74><EFBFBD>cloak_email_addresses<65><73><EFBFBD>env<6E>Nub<75>reporter<65>N<EFBFBD>indirect_targets<74>]<5D><>substitution_defs<66>}<7D><>substitution_names<65>}<7D><>refnames<65>}<7D><>refids<64>}<7D>(h<>]<5D>h<EFBFBD>aj<61>]<5D>j<EFBFBD>au<61>nameids<64>}<7D>(j<>h<>j<EFBFBD>j<>j<>j<>j$j!j$j!jjj<>j<>jjj<>j<>j<>j<>j?j<>jkjhu<> nametypes<65>}<7D>(j<><00>j<EFBFBD>Nj<4E>Nj$Nj$NjNj<4E>NjNj<4E>Nj<4E>Nj?<00>jk<00>uh}<7D>(h<>h<EFBFBD>j<EFBFBD>h<>j<EFBFBD>jcj!j<>j!jjj'j<>j'jj<>j<>jj<>j)j<>j<>jhjbu<> footnote_refs<66>}<7D><> citation_refs<66>}<7D><> autofootnotes<65>]<5D><>autofootnote_refs<66>]<5D><>symbol_footnotes<65>]<5D><>symbol_footnote_refs<66>]<5D><> footnotes<65>]<5D><> citations<6E>]<5D><>autofootnote_start<72>K<01>symbol_footnote_start<72>K<00>
id_counter<EFBFBD><EFBFBD> collections<6E><73>Counter<65><72><EFBFBD>}<7D><><EFBFBD>R<EFBFBD><52>parse_messages<65>]<5D>h <09>system_message<67><65><EFBFBD>)<29><>}<7D>(hhh]<5D>hm)<29><>}<7D>(h<05>Unexpected indentation.<2E>h]<5D>h<11>Unexpected indentation.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj!ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlhjubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>level<65>K<03>type<70><65>ERROR<4F><52>source<63>h'<27>line<6E>K<EFBFBD>uh%jhj'hhh&h'h(K<>uba<62>transform_messages<65>]<5D>(j)<29><>}<7D>(hhh]<5D>hm)<29><>}<7D>(hhh]<5D>h<11>dHyperlink target "sphx-glr-getting-started-tutorials-03-matrix-multiplication-py" is not referenced.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj?ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlhj<ubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>level<65>K<01>type<70><65>INFO<46><4F>source<63>h'<27>line<6E>Kuh%jubj)<29><>}<7D>(hhh]<5D>hm)<29><>}<7D>(hhh]<5D>h<11>mHyperlink target "sphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py" is not referenced.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjZubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D>uh%hlhjWubah}<7D>(h]<5D>h]<5D>h]<5D>h]<5D>h!]<5D><>level<65>K<01>type<70>jT<00>source<63>h'<27>line<6E>M<EFBFBD>uh%jube<62> transformer<65>N<EFBFBD>
decoration<EFBFBD>Nhhub.
Reference in New Issue Copy Permalink