2021-03-15 13:58:20 -04:00
<!DOCTYPE html>
< html class = "writer-html5" lang = "en" >
< head >
< meta charset = "utf-8" / >
< meta name = "viewport" content = "width=device-width, initial-scale=1.0" / >
< title > Matrix Multiplication — Triton documentation< / title >
< link rel = "stylesheet" href = "../../_static/css/theme.css" type = "text/css" / >
< link rel = "stylesheet" href = "../../_static/pygments.css" type = "text/css" / >
< link rel = "stylesheet" href = "../../_static/gallery.css" type = "text/css" / >
< link rel = "stylesheet" href = "../../_static/gallery-binder.css" type = "text/css" / >
< link rel = "stylesheet" href = "../../_static/gallery-dataframe.css" type = "text/css" / >
< link rel = "stylesheet" href = "../../_static/gallery-rendered-html.css" type = "text/css" / >
2021-03-19 16:19:37 -04:00
< link rel = "stylesheet" href = "../../_static/css/custom.css" type = "text/css" / >
2021-03-15 13:58:20 -04:00
<!-- [if lt IE 9]>
< script src = "../../_static/js/html5shiv.min.js" > < / script >
<![endif]-->
< script type = "text/javascript" id = "documentation_options" data-url_root = "../../" src = "../../_static/documentation_options.js" > < / script >
< script src = "../../_static/jquery.js" > < / script >
< script src = "../../_static/underscore.js" > < / script >
< script src = "../../_static/doctools.js" > < / script >
< script type = "text/javascript" src = "../../_static/js/theme.js" > < / script >
< link rel = "index" title = "Index" href = "../../genindex.html" / >
< link rel = "search" title = "Search" href = "../../search.html" / >
2021-04-23 16:42:55 -04:00
< link rel = "next" title = "triton" href = "../../python-api/triton.html" / >
2021-03-15 13:58:20 -04:00
< link rel = "prev" title = "Fused Softmax" href = "02-fused-softmax.html" / >
< / head >
< body class = "wy-body-for-nav" >
< div class = "wy-grid-for-nav" >
< nav data-toggle = "wy-nav-shift" class = "wy-nav-side" >
< div class = "wy-side-scroll" >
< div class = "wy-side-nav-search" >
< a href = "../../index.html" class = "icon icon-home" > Triton
< / a >
< div role = "search" >
< form id = "rtd-search-form" class = "wy-form" action = "../../search.html" method = "get" >
< input type = "text" name = "q" placeholder = "Search docs" / >
< input type = "hidden" name = "check_keywords" value = "yes" / >
< input type = "hidden" name = "area" value = "default" / >
< / form >
< / div >
< / div >
< div class = "wy-menu wy-menu-vertical" data-spy = "affix" role = "navigation" aria-label = "main navigation" >
< p class = "caption" > < span class = "caption-text" > Getting Started< / span > < / p >
< ul class = "current" >
< li class = "toctree-l1" > < a class = "reference internal" href = "../installation.html" > Installation< / a > < / li >
< li class = "toctree-l1 current" > < a class = "reference internal" href = "index.html" > Tutorials< / a > < ul class = "current" >
< li class = "toctree-l2" > < a class = "reference internal" href = "01-vector-add.html" > Vector Addition< / a > < / li >
< li class = "toctree-l2" > < a class = "reference internal" href = "02-fused-softmax.html" > Fused Softmax< / a > < / li >
< li class = "toctree-l2 current" > < a class = "current reference internal" href = "#" > Matrix Multiplication< / a > < ul >
< li class = "toctree-l3" > < a class = "reference internal" href = "#motivations" > Motivations< / a > < / li >
< li class = "toctree-l3" > < a class = "reference internal" href = "#compute-kernel" > Compute Kernel< / a > < ul >
< li class = "toctree-l4" > < a class = "reference internal" href = "#pointer-arithmetics" > Pointer Arithmetics< / a > < / li >
< li class = "toctree-l4" > < a class = "reference internal" href = "#l2-cache-optimizations" > L2 Cache Optimizations< / a > < / li >
< / ul >
< / li >
2021-04-21 01:40:29 -04:00
< li class = "toctree-l3" > < a class = "reference internal" href = "#final-result" > Final Result< / a > < / li >
2021-03-15 13:58:20 -04:00
< li class = "toctree-l3" > < a class = "reference internal" href = "#unit-test" > Unit Test< / a > < / li >
< li class = "toctree-l3" > < a class = "reference internal" href = "#benchmark" > Benchmark< / a > < ul >
< li class = "toctree-l4" > < a class = "reference internal" href = "#square-matrix-performance" > Square Matrix Performance< / a > < / li >
< / ul >
< / li >
< / ul >
< / li >
< / ul >
< / li >
< / ul >
2021-04-23 16:42:55 -04:00
< p class = "caption" > < span class = "caption-text" > Python API< / span > < / p >
2021-04-21 01:40:29 -04:00
< ul >
2021-04-23 16:42:55 -04:00
< li class = "toctree-l1" > < a class = "reference internal" href = "../../python-api/triton.html" > triton< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../python-api/triton.language.html" > triton.language< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../python-api/triton.testing.html" > triton.testing< / a > < / li >
2021-04-21 01:40:29 -04:00
< / ul >
2021-03-19 16:19:37 -04:00
< p class = "caption" > < span class = "caption-text" > Programming Guide< / span > < / p >
< ul >
2021-03-23 17:10:07 -04:00
< li class = "toctree-l1" > < a class = "reference internal" href = "../../programming-guide/chapter-1/introduction.html" > Introduction< / a > < / li >
< li class = "toctree-l1" > < a class = "reference internal" href = "../../programming-guide/chapter-2/related-work.html" > Related Work< / a > < / li >
2021-03-19 16:19:37 -04:00
< / ul >
2021-03-15 13:58:20 -04:00
< / div >
< / div >
< / nav >
< section data-toggle = "wy-nav-shift" class = "wy-nav-content-wrap" >
< nav class = "wy-nav-top" aria-label = "top navigation" >
< i data-toggle = "wy-nav-top" class = "fa fa-bars" > < / i >
< a href = "../../index.html" > Triton< / a >
< / nav >
< div class = "wy-nav-content" >
< div class = "rst-content" >
< div role = "navigation" aria-label = "breadcrumbs navigation" >
< ul class = "wy-breadcrumbs" >
< li > < a href = "../../index.html" class = "icon icon-home" > < / a > » < / li >
< li > < a href = "index.html" > Tutorials< / a > » < / li >
< li > Matrix Multiplication< / li >
< li class = "wy-breadcrumbs-aside" >
< a href = "../../_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt" rel = "nofollow" > View page source< / a >
< / li >
< / ul >
< hr / >
< / div >
< div role = "main" class = "document" itemscope = "itemscope" itemtype = "http://schema.org/Article" >
< div itemprop = "articleBody" >
< div class = "sphx-glr-download-link-note admonition note" >
< p class = "admonition-title" > Note< / p >
< p > Click < a class = "reference internal" href = "#sphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py" > < span class = "std std-ref" > here< / span > < / a >
to download the full example code< / p >
< / div >
< div class = "sphx-glr-example-title section" id = "matrix-multiplication" >
< span id = "sphx-glr-getting-started-tutorials-03-matrix-multiplication-py" > < / span > < h1 > Matrix Multiplication< a class = "headerlink" href = "#matrix-multiplication" title = "Permalink to this headline" > ¶< / a > < / h1 >
2021-04-21 01:40:29 -04:00
< p > In this tutorial, you will write a 25-lines high-performance matrix multiplication kernel that achieves close to peak performance on modern GPUs.
2021-03-15 13:58:20 -04:00
You will specifically learn about:< / p >
< ul class = "simple" >
2021-04-21 01:40:29 -04:00
< li > < p > Block-level matrix multiplications< / p > < / li >
2021-03-15 13:58:20 -04:00
< li > < p > Multi-dimensional pointer arithmetic< / p > < / li >
< li > < p > Program re-ordering for improved L2 cache hit rate< / p > < / li >
< li > < p > Automatic performance tuning< / p > < / li >
< / ul >
< div class = "section" id = "motivations" >
< h2 > Motivations< a class = "headerlink" href = "#motivations" title = "Permalink to this headline" > ¶< / a > < / h2 >
< p > Matrix multiplications are a key building block of most modern high-performance computing systems.
They are notoriously hard to optimize, hence their implementation is typically done by hardware vendors themselves as part of so-called “kernel libraries” (e.g., cuBLAS).
2021-04-21 01:40:29 -04:00
Unfortunately, these libraries are often proprietary and cannot be easily customized to accomodate the needs of modern deep learning workloads (e.g., mixture of experts, fused activation functions, etc.).
2021-03-15 13:58:20 -04:00
For this reason, this tutorial will show you how to implement efficient matrix multiplications yourself with Triton, in a way that is easy to customize and extend.< / p >
< p > Roughly speaking, the kernel that we will write will implement the following blocked algorithm:< / p >
< blockquote >
< div > < div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > # do in parallel< / span >
2021-04-21 01:40:29 -04:00
< span class = "k" > for< / span > < span class = "n" > m< / span > < span class = "ow" > in< / span > < span class = "nb" > range< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ):< / span >
2021-03-15 13:58:20 -04:00
< span class = "c1" > # do in parallel< / span >
2021-04-21 01:40:29 -04:00
< span class = "k" > for< / span > < span class = "n" > n< / span > < span class = "ow" > in< / span > < span class = "nb" > range< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ):< / span >
< span class = "n" > acc< / span > < span class = "o" > =< / span > < span class = "n" > zeros< / span > < span class = "p" > ((< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ),< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > float32< / span > < span class = "p" > )< / span >
< span class = "k" > for< / span > < span class = "n" > k< / span > < span class = "ow" > in< / span > < span class = "nb" > range< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ):< / span >
< span class = "n" > a< / span > < span class = "o" > =< / span > < span class = "n" > A< / span > < span class = "p" > [< / span > < span class = "n" > m< / span > < span class = "p" > :< / span > < span class = "n" > m< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ,< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ]< / span >
< span class = "n" > b< / span > < span class = "o" > =< / span > < span class = "n" > B< / span > < span class = "p" > [< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ,< / span > < span class = "n" > n< / span > < span class = "p" > :< / span > < span class = "n" > n< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ]< / span >
< span class = "n" > acc< / span > < span class = "o" > +=< / span > < span class = "n" > dot< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > )< / span >
< span class = "n" > C< / span > < span class = "p" > [< / span > < span class = "n" > m< / span > < span class = "p" > :< / span > < span class = "n" > m< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ,< / span > < span class = "n" > n< / span > < span class = "p" > :< / span > < span class = "n" > n< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > acc< / span > < span class = "p" > ;< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
2021-04-21 01:40:29 -04:00
< p > where each iteration of the doubly-nested for-loop corresponds to a Triton program instance.< / p >
2021-03-15 13:58:20 -04:00
< / div >
< div class = "section" id = "compute-kernel" >
< h2 > Compute Kernel< a class = "headerlink" href = "#compute-kernel" title = "Permalink to this headline" > ¶< / a > < / h2 >
2021-04-21 01:40:29 -04:00
< p > The above algorithm is actually fairly straightforward to implement in Triton.
The main difficulty comes from the 2D pointer arithmetic that must be done to specify the memory locations for the blocks of < code class = "code docutils literal notranslate" > < span class = "pre" > A< / span > < / code > and < code class = "code docutils literal notranslate" > < span class = "pre" > B< / span > < / code > that we need to read in the inner loop.< / p >
2021-03-15 13:58:20 -04:00
< div class = "section" id = "pointer-arithmetics" >
< h3 > Pointer Arithmetics< a class = "headerlink" href = "#pointer-arithmetics" title = "Permalink to this headline" > ¶< / a > < / h3 >
2021-04-21 01:40:29 -04:00
< p > For a row-major 2D tensor < code class = "code docutils literal notranslate" > < span class = "pre" > X< / span > < / code > , the memory location of < code class = "code docutils literal notranslate" > < span class = "pre" > X[i,< / span > < span class = "pre" > j]< / span > < / code > is given by < code class = "code docutils literal notranslate" > < span class = "pre" > & X[i,< / span > < span class = "pre" > j]< / span > < span class = "pre" > =< / span > < span class = "pre" > X< / span > < span class = "pre" > +< / span > < span class = "pre" > i*stride_x_0< / span > < span class = "pre" > +< / span > < span class = "pre" > j*stride_x_1< / span > < / code > .
Therefore, blocks of pointers for < code class = "code docutils literal notranslate" > < span class = "pre" > A[m< / span > < span class = "pre" > :< / span > < span class = "pre" > m+BLOCK_M,< / span > < span class = "pre" > k:k+BLOCK_K]< / span > < / code > and < code class = "code docutils literal notranslate" > < span class = "pre" > B[k< / span > < span class = "pre" > :< / span > < span class = "pre" > k+BLOCK_K,< / span > < span class = "pre" > n< / span > < span class = "pre" > :< / span > < span class = "pre" > n+BLOCK_N]< / span > < / code > can be defined in pseudo-code as:< / p >
2021-03-15 13:58:20 -04:00
< blockquote >
2021-04-21 01:40:29 -04:00
< div > < div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "o" > & < / span > < span class = "n" > A< / span > < span class = "p" > [< / span > < span class = "n" > m< / span > < span class = "p" > :< / span > < span class = "n" > m< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ,< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > A< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > m< / span > < span class = "p" > :< / span > < span class = "n" > m< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > )[:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > A< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > )[< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :];< / span >
< span class = "o" > & < / span > < span class = "n" > B< / span > < span class = "p" > [< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ,< / span > < span class = "n" > n< / span > < span class = "p" > :< / span > < span class = "n" > n< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ]< / span > < span class = "o" > =< / span > < span class = "n" > B< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > k< / span > < span class = "p" > :< / span > < span class = "n" > k< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > )[:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > B< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > n< / span > < span class = "p" > :< / span > < span class = "n" > n< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > )[< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :];< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
< p > Which means that, at initialization (i.e., < code class = "code docutils literal notranslate" > < span class = "pre" > k< / span > < span class = "pre" > =< / span > < span class = "pre" > 0< / span > < / code > ), pointers for blocks of A and B can be initialized in Triton as:< / p >
< blockquote >
2021-04-21 01:40:29 -04:00
< div > < div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "n" > pid_m< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > program_id< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
< span class = "n" > pid_n< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > program_id< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "p" > )< / span >
< span class = "n" > rm< / span > < span class = "o" > =< / span > < span class = "n" > pid_m< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_M< / span > < span class = "o" > +< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > )< / span >
< span class = "n" > rn< / span > < span class = "o" > =< / span > < span class = "n" > pid_n< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_N< / span > < span class = "o" > +< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > )< / span >
< span class = "n" > rk< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > )< / span >
< span class = "o" > //< / span > < span class = "n" > pointer< / span > < span class = "k" > for< / span > < span class = "n" > A< / span > < span class = "n" > operand< / span >
< span class = "n" > pa< / span > < span class = "o" > =< / span > < span class = "n" > A< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > rm< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > stride_a_0< / span > < span class = "o" > +< / span > < span class = "n" > rk< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > *< / span > < span class = "n" > stride_a_1< / span > < span class = "p" > );< / span >
< span class = "o" > //< / span > < span class = "n" > pointer< / span > < span class = "k" > for< / span > < span class = "n" > B< / span > < span class = "n" > operand< / span >
< span class = "n" > pb< / span > < span class = "o" > =< / span > < span class = "n" > B< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > rk< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > stride_b_0< / span > < span class = "o" > +< / span > < span class = "n" > rn< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > *< / span > < span class = "n" > stride_b_1< / span > < span class = "p" > );< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
< p > These pointers can then be updated in the inner loop as:< / p >
< blockquote >
2021-04-21 01:40:29 -04:00
< div > < div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "n" > pa< / span > < span class = "o" > +=< / span > < span class = "n" > BLOCK_K< / span > < span class = "o" > *< / span > < span class = "n" > stride_a_1< / span > < span class = "p" > ;< / span >
< span class = "n" > pb< / span > < span class = "o" > +=< / span > < span class = "n" > BLOCK_K< / span > < span class = "o" > *< / span > < span class = "n" > stride_b_0< / span > < span class = "p" > ;< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
< / div >
< div class = "section" id = "l2-cache-optimizations" >
< h3 > L2 Cache Optimizations< a class = "headerlink" href = "#l2-cache-optimizations" title = "Permalink to this headline" > ¶< / a > < / h3 >
2021-04-21 01:40:29 -04:00
< p > As mentioned above, each program instance computes an < code class = "code docutils literal notranslate" > < span class = "pre" > [BLOCK_M,< / span > < span class = "pre" > BLOCK_N]< / span > < / code > block of < code class = "code docutils literal notranslate" > < span class = "pre" > C< / span > < / code > .
2021-03-15 13:58:20 -04:00
However, the order in which these blocks are computer matters, since it affects the L2 cache hit rate of our program.
This means that a naive row-major ordering:< / p >
< blockquote >
2021-04-21 01:40:29 -04:00
< div > < div class = "highlight-Python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "n" > pid< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > program_id< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > );< / span >
< span class = "n" > grid_m< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > M< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "o" > -< / span > < span class = "mi" > 1< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ;< / span >
< span class = "n" > grid_n< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > N< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "o" > -< / span > < span class = "mi" > 1< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ;< / span >
< span class = "n" > pid_m< / span > < span class = "o" > =< / span > < span class = "n" > pid< / span > < span class = "o" > /< / span > < span class = "n" > grid_n< / span > < span class = "p" > ;< / span >
< span class = "n" > pid_n< / span > < span class = "o" > =< / span > < span class = "n" > pid< / span > < span class = "o" > %< / span > < span class = "n" > grid_n< / span > < span class = "p" > ;< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
< p > is unlikely to result in optimal performance.< / p >
< p > One possible solution is to launch blocks in an order that promotes data reuse.
2021-04-21 01:40:29 -04:00
This can be done by ‘ ’ < code class = "code docutils literal notranslate" > < span class = "pre" > GROUP_M< / span > < / code > rows before switching to the next column:< / p >
2021-03-15 13:58:20 -04:00
< blockquote >
2021-04-21 01:40:29 -04:00
< div > < div class = "highlight-python notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "n" > pid< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > program_id< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > );< / span >
< span class = "n" > width< / span > < span class = "o" > =< / span > < span class = "n" > GROUP_M< / span > < span class = "o" > *< / span > < span class = "n" > grid_n< / span > < span class = "p" > ;< / span >
< span class = "n" > group_id< / span > < span class = "o" > =< / span > < span class = "n" > pid< / span > < span class = "o" > //< / span > < span class = "n" > width< / span > < span class = "p" > ;< / span >
< span class = "c1" > # we need to handle the case where M % (GROUP_M*BLOCK_M) != 0< / span >
< span class = "n" > group_size< / span > < span class = "o" > =< / span > < span class = "nb" > min< / span > < span class = "p" > (< / span > < span class = "n" > grid_m< / span > < span class = "o" > -< / span > < span class = "n" > group_id< / span > < span class = "o" > *< / span > < span class = "n" > GROUP_M< / span > < span class = "p" > ,< / span > < span class = "n" > GROUP_M< / span > < span class = "p" > );< / span >
< span class = "n" > pid_m< / span > < span class = "o" > =< / span > < span class = "n" > group_id< / span > < span class = "o" > *< / span > < span class = "n" > GROUP_M< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > pid< / span > < span class = "o" > %< / span > < span class = "n" > group_size< / span > < span class = "p" > );< / span >
< span class = "n" > pid_n< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > pid< / span > < span class = "o" > %< / span > < span class = "n" > width< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "p" > (< / span > < span class = "n" > group_size< / span > < span class = "p" > );< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div > < / blockquote >
< p > In practice, this can improve the performance of our matrix multiplication kernel by > 10% on some hardware architecture (e.g., 220 to 245 TFLOPS on A100).< / p >
< / div >
< / div >
2021-04-21 01:40:29 -04:00
< div class = "section" id = "final-result" >
< h2 > Final Result< a class = "headerlink" href = "#final-result" title = "Permalink to this headline" > ¶< / a > < / h2 >
2021-03-15 13:58:20 -04:00
< div class = "highlight-default notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "kn" > import< / span > < span class = "nn" > torch< / span >
< span class = "kn" > import< / span > < span class = "nn" > triton< / span >
2021-04-23 16:42:55 -04:00
< span class = "kn" > import< / span > < span class = "nn" > triton.language< / span > < span class = "k" > as< / span > < span class = "nn" > tl< / span >
2021-03-15 13:58:20 -04:00
2021-04-21 01:40:29 -04:00
< span class = "c1" > # %< / span >
< span class = "c1" > # :code:`triton.jit`' ed functions can be auto-tuned by using the `triton.autotune` decorator, which consumes:< / span >
< span class = "c1" > # - A list of :code:`triton.Config` objects that define different configurations of meta-parameters (e.g., BLOCK_M) and compilation options (e.g., num_warps) to try< / span >
< span class = "c1" > # - A autotuning *key* whose change in values will trigger evaluation of all the provided configs< / span >
< span class = "nd" > @triton< / span > < span class = "o" > .< / span > < span class = "n" > jit< / span >
< span class = "k" > def< / span > < span class = "nf" > sigmoid< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "p" > ):< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > ret_true< / span > < span class = "o" > =< / span > < span class = "mi" > 1< / span > < span class = "o" > /< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > exp< / span > < span class = "p" > (< / span > < span class = "o" > -< / span > < span class = "n" > x< / span > < span class = "p" > ))< / span >
< span class = "n" > ret_false< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > exp< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "p" > )< / span > < span class = "o" > /< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > exp< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "p" > ))< / span >
< span class = "k" > return< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > where< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "o" > > =< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > ret_true< / span > < span class = "p" > ,< / span > < span class = "n" > ret_false< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "nd" > @triton< / span > < span class = "o" > .< / span > < span class = "n" > jit< / span >
< span class = "k" > def< / span > < span class = "nf" > swish< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "p" > ):< / span >
< span class = "k" > return< / span > < span class = "n" > x< / span > < span class = "o" > *< / span > < span class = "n" > sigmoid< / span > < span class = "p" > (< / span > < span class = "n" > x< / span > < span class = "p" > )< / span >
< span class = "nd" > @triton< / span > < span class = "o" > .< / span > < span class = "n" > autotune< / span > < span class = "p" > (< / span >
< span class = "n" > configs< / span > < span class = "o" > =< / span > < span class = "p" > [< / span >
< span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > Config< / span > < span class = "p" > ({< / span > < span class = "s1" > ' BLOCK_M' < / span > < span class = "p" > :< / span > < span class = "mi" > 128< / span > < span class = "p" > ,< / span > < span class = "s1" > ' BLOCK_N' < / span > < span class = "p" > :< / span > < span class = "mi" > 128< / span > < span class = "p" > ,< / span > < span class = "s1" > ' BLOCK_K' < / span > < span class = "p" > :< / span > < span class = "mi" > 32< / span > < span class = "p" > ,< / span > < span class = "s1" > ' GROUP_M' < / span > < span class = "p" > :< / span > < span class = "mi" > 8< / span > < span class = "p" > },< / span > < span class = "n" > num_warps< / span > < span class = "o" > =< / span > < span class = "mi" > 4< / span > < span class = "p" > ),< / span >
< span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > Config< / span > < span class = "p" > ({< / span > < span class = "s1" > ' BLOCK_M' < / span > < span class = "p" > :< / span > < span class = "mi" > 64< / span > < span class = "p" > ,< / span > < span class = "s1" > ' BLOCK_N' < / span > < span class = "p" > :< / span > < span class = "mi" > 128< / span > < span class = "p" > ,< / span > < span class = "s1" > ' BLOCK_K' < / span > < span class = "p" > :< / span > < span class = "mi" > 32< / span > < span class = "p" > ,< / span > < span class = "s1" > ' GROUP_M' < / span > < span class = "p" > :< / span > < span class = "mi" > 8< / span > < span class = "p" > },< / span > < span class = "n" > num_warps< / span > < span class = "o" > =< / span > < span class = "mi" > 4< / span > < span class = "p" > ),< / span >
< span class = "p" > ],< / span >
< span class = "n" > key< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "s1" > ' M' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' N' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' K' < / span > < span class = "p" > ],< / span >
< span class = "p" > )< / span >
< span class = "c1" > # %< / span >
< span class = "c1" > # We can now define our kernel as normal, using all the techniques presented above< / span >
< span class = "nd" > @triton< / span > < span class = "o" > .< / span > < span class = "n" > jit< / span >
< span class = "k" > def< / span > < span class = "nf" > _matmul< / span > < span class = "p" > (< / span > < span class = "n" > A< / span > < span class = "p" > ,< / span > < span class = "n" > B< / span > < span class = "p" > ,< / span > < span class = "n" > C< / span > < span class = "p" > ,< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > < span class = "n" > stride_am< / span > < span class = "p" > ,< / span > < span class = "n" > stride_ak< / span > < span class = "p" > ,< / span > < span class = "n" > stride_bk< / span > < span class = "p" > ,< / span > < span class = "n" > stride_bn< / span > < span class = "p" > ,< / span > < span class = "n" > stride_cm< / span > < span class = "p" > ,< / span > < span class = "n" > stride_cn< / span > < span class = "p" > ,< / span > < span class = "o" > **< / span > < span class = "n" > META< / span > < span class = "p" > ):< / span >
< span class = "c1" > # extract meta-parameters< / span >
< span class = "n" > BLOCK_M< / span > < span class = "o" > =< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' BLOCK_M' < / span > < span class = "p" > ]< / span >
< span class = "n" > BLOCK_N< / span > < span class = "o" > =< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' BLOCK_N' < / span > < span class = "p" > ]< / span >
< span class = "n" > BLOCK_K< / span > < span class = "o" > =< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' BLOCK_K' < / span > < span class = "p" > ]< / span >
< span class = "n" > GROUP_M< / span > < span class = "o" > =< / span > < span class = "mi" > 8< / span >
< span class = "c1" > # matrix multiplication< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > pid< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > program_id< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > grid_m< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > M< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_M< / span > < span class = "o" > -< / span > < span class = "mi" > 1< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "n" > BLOCK_M< / span >
< span class = "n" > grid_n< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > N< / span > < span class = "o" > +< / span > < span class = "n" > BLOCK_N< / span > < span class = "o" > -< / span > < span class = "mi" > 1< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "n" > BLOCK_N< / span >
< span class = "c1" > # re-order program ID for better L2 performance< / span >
< span class = "n" > width< / span > < span class = "o" > =< / span > < span class = "n" > GROUP_M< / span > < span class = "o" > *< / span > < span class = "n" > grid_n< / span >
< span class = "n" > group_id< / span > < span class = "o" > =< / span > < span class = "n" > pid< / span > < span class = "o" > //< / span > < span class = "n" > width< / span >
< span class = "n" > group_size< / span > < span class = "o" > =< / span > < span class = "nb" > min< / span > < span class = "p" > (< / span > < span class = "n" > grid_m< / span > < span class = "o" > -< / span > < span class = "n" > group_id< / span > < span class = "o" > *< / span > < span class = "n" > GROUP_M< / span > < span class = "p" > ,< / span > < span class = "n" > GROUP_M< / span > < span class = "p" > )< / span >
< span class = "n" > pid_m< / span > < span class = "o" > =< / span > < span class = "n" > group_id< / span > < span class = "o" > *< / span > < span class = "n" > GROUP_M< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > pid< / span > < span class = "o" > %< / span > < span class = "n" > group_size< / span > < span class = "p" > )< / span >
< span class = "n" > pid_n< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > pid< / span > < span class = "o" > %< / span > < span class = "n" > width< / span > < span class = "p" > )< / span > < span class = "o" > //< / span > < span class = "p" > (< / span > < span class = "n" > group_size< / span > < span class = "p" > )< / span >
< span class = "c1" > # do matrix multiplication< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > rm< / span > < span class = "o" > =< / span > < span class = "n" > pid_m< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_M< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > )< / span >
< span class = "n" > rn< / span > < span class = "o" > =< / span > < span class = "n" > pid_n< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_N< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > )< / span >
< span class = "n" > rk< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > A< / span > < span class = "o" > =< / span > < span class = "n" > A< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > rm< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > stride_am< / span > < span class = "o" > +< / span > < span class = "n" > rk< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > *< / span > < span class = "n" > stride_ak< / span > < span class = "p" > )< / span >
< span class = "n" > B< / span > < span class = "o" > =< / span > < span class = "n" > B< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > rk< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > stride_bk< / span > < span class = "o" > +< / span > < span class = "n" > rn< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > *< / span > < span class = "n" > stride_bn< / span > < span class = "p" > )< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > acc< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > zeros< / span > < span class = "p" > ((< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > ),< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > float32< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "k" > for< / span > < span class = "n" > k< / span > < span class = "ow" > in< / span > < span class = "nb" > range< / span > < span class = "p" > (< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "o" > -< / span > < span class = "n" > BLOCK_K< / span > < span class = "p" > ):< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > a< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > load< / span > < span class = "p" > (< / span > < span class = "n" > A< / span > < span class = "p" > )< / span >
< span class = "n" > b< / span > < span class = "o" > =< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > load< / span > < span class = "p" > (< / span > < span class = "n" > B< / span > < span class = "p" > )< / span >
< span class = "n" > acc< / span > < span class = "o" > +=< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > dot< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > A< / span > < span class = "o" > +=< / span > < span class = "n" > BLOCK_K< / span > < span class = "o" > *< / span > < span class = "n" > stride_ak< / span >
< span class = "n" > B< / span > < span class = "o" > +=< / span > < span class = "n" > BLOCK_K< / span > < span class = "o" > *< / span > < span class = "n" > stride_bk< / span >
< span class = "c1" > # triton can accept arbitrary activation function< / span >
< span class = "c1" > # via metaparameters!< / span >
< span class = "k" > if< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' ACTIVATION' < / span > < span class = "p" > ]:< / span >
< span class = "n" > acc< / span > < span class = "o" > =< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' ACTIVATION' < / span > < span class = "p" > ](< / span > < span class = "n" > acc< / span > < span class = "p" > )< / span >
< span class = "c1" > # rematerialize rm and rn to save registers< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > rm< / span > < span class = "o" > =< / span > < span class = "n" > pid_m< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_M< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_M< / span > < span class = "p" > )< / span >
< span class = "n" > rn< / span > < span class = "o" > =< / span > < span class = "n" > pid_n< / span > < span class = "o" > *< / span > < span class = "n" > BLOCK_N< / span > < span class = "o" > +< / span > < span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > arange< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ,< / span > < span class = "n" > BLOCK_N< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > C< / span > < span class = "o" > =< / span > < span class = "n" > C< / span > < span class = "o" > +< / span > < span class = "p" > (< / span > < span class = "n" > rm< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > *< / span > < span class = "n" > stride_cm< / span > < span class = "o" > +< / span > < span class = "n" > rn< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > *< / span > < span class = "n" > stride_cn< / span > < span class = "p" > )< / span >
< span class = "n" > mask< / span > < span class = "o" > =< / span > < span class = "p" > (< / span > < span class = "n" > rm< / span > < span class = "p" > [:,< / span > < span class = "kc" > None< / span > < span class = "p" > ]< / span > < span class = "o" > < < / span > < span class = "n" > M< / span > < span class = "p" > )< / span > < span class = "o" > & < / span > < span class = "p" > (< / span > < span class = "n" > rn< / span > < span class = "p" > [< / span > < span class = "kc" > None< / span > < span class = "p" > ,< / span > < span class = "p" > :]< / span > < span class = "o" > < < / span > < span class = "n" > N< / span > < span class = "p" > )< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > tl< / span > < span class = "o" > .< / span > < span class = "n" > store< / span > < span class = "p" > (< / span > < span class = "n" > C< / span > < span class = "p" > ,< / span > < span class = "n" > acc< / span > < span class = "p" > ,< / span > < span class = "n" > mask< / span > < span class = "o" > =< / span > < span class = "n" > mask< / span > < span class = "p" > )< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
2021-04-21 01:40:29 -04:00
< p > We can also create a convenience wrapper function that only takes two input tensors
and (1) checks any shape constraint; (2) allocates the output; (3) launches the kernel< / p >
< div class = "highlight-default notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "k" > def< / span > < span class = "nf" > matmul< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ,< / span > < span class = "n" > activation< / span > < span class = "o" > =< / span > < span class = "kc" > None< / span > < span class = "p" > ):< / span >
< span class = "c1" > # checks constraints< / span >
< span class = "k" > assert< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > [< / span > < span class = "mi" > 1< / span > < span class = "p" > ]< / span > < span class = "o" > ==< / span > < span class = "n" > b< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span > < span class = "p" > [< / span > < span class = "mi" > 0< / span > < span class = "p" > ],< / span > < span class = "s2" > " incompatible dimensions" < / span >
< span class = "k" > assert< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > is_contiguous< / span > < span class = "p" > (),< / span > < span class = "s2" > " matrix A must be contiguous" < / span >
< span class = "k" > assert< / span > < span class = "n" > b< / span > < span class = "o" > .< / span > < span class = "n" > is_contiguous< / span > < span class = "p" > (),< / span > < span class = "s2" > " matrix B must be contiguous" < / span >
< span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "o" > =< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span >
< span class = "n" > _< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "o" > =< / span > < span class = "n" > b< / span > < span class = "o" > .< / span > < span class = "n" > shape< / span >
< span class = "c1" > # allocates output< / span >
< span class = "n" > c< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > empty< / span > < span class = "p" > ((< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ),< / span > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > device< / span > < span class = "p" > ,< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > dtype< / span > < span class = "p" > )< / span >
< span class = "c1" > # launch kernel< / span >
< span class = "n" > grid< / span > < span class = "o" > =< / span > < span class = "k" > lambda< / span > < span class = "n" > META< / span > < span class = "p" > :< / span > < span class = "p" > (< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > cdiv< / span > < span class = "p" > (< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' BLOCK_M' < / span > < span class = "p" > ])< / span > < span class = "o" > *< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > cdiv< / span > < span class = "p" > (< / span > < span class = "n" > N< / span > < span class = "p" > ,< / span > < span class = "n" > META< / span > < span class = "p" > [< / span > < span class = "s1" > ' BLOCK_N' < / span > < span class = "p" > ]),< / span > < span class = "p" > )< / span >
< span class = "n" > _matmul< / span > < span class = "p" > [< / span > < span class = "n" > grid< / span > < span class = "p" > ](< / span >
< span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ,< / span > < span class = "n" > c< / span > < span class = "p" > ,< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > \
< span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ),< / span > < span class = "n" > a< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "p" > ),< / span > < span class = "n" > b< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ),< / span > < span class = "n" > b< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "p" > ),< / span > < span class = "n" > c< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 0< / span > < span class = "p" > ),< / span > < span class = "n" > c< / span > < span class = "o" > .< / span > < span class = "n" > stride< / span > < span class = "p" > (< / span > < span class = "mi" > 1< / span > < span class = "p" > ),< / span > \
< span class = "n" > ACTIVATION< / span > < span class = "o" > =< / span > < span class = "n" > activation< / span >
< span class = "p" > )< / span >
< span class = "c1" > # return output< / span >
< span class = "k" > return< / span > < span class = "n" > c< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div >
< div class = "section" id = "unit-test" >
< h2 > Unit Test< a class = "headerlink" href = "#unit-test" title = "Permalink to this headline" > ¶< / a > < / h2 >
2021-04-21 01:40:29 -04:00
< p > We can test our custom matrix multiplication operation against a native torch implementation (i.e., cuBLAS + custom element-wise swish kernel)< / p >
< div class = "highlight-default notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "c1" > #torch.manual_seed(0)< / span >
< span class = "n" > a< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > randn< / span > < span class = "p" > ((< / span > < span class = "mi" > 512< / span > < span class = "p" > ,< / span > < span class = "mi" > 512< / span > < span class = "p" > ),< / span > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > float16< / span > < span class = "p" > )< / span >
< span class = "n" > b< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > randn< / span > < span class = "p" > ((< / span > < span class = "mi" > 512< / span > < span class = "p" > ,< / span > < span class = "mi" > 512< / span > < span class = "p" > ),< / span > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > float16< / span > < span class = "p" > )< / span >
< span class = "n" > c_0< / span > < span class = "o" > =< / span > < span class = "n" > matmul< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ,< / span > < span class = "n" > activation< / span > < span class = "o" > =< / span > < span class = "n" > swish< / span > < span class = "p" > )< / span >
< span class = "n" > c_1< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > nn< / span > < span class = "o" > .< / span > < span class = "n" > SiLU< / span > < span class = "p" > ()(< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > matmul< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ))< / span >
2021-03-15 13:58:20 -04:00
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "n" > c_0< / span > < span class = "p" > )< / span >
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "n" > c_1< / span > < span class = "p" > )< / span >
2021-04-21 01:40:29 -04:00
< span class = "nb" > print< / span > < span class = "p" > (< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > testing< / span > < span class = "o" > .< / span > < span class = "n" > allclose< / span > < span class = "p" > (< / span > < span class = "n" > c_0< / span > < span class = "p" > ,< / span > < span class = "n" > c_1< / span > < span class = "p" > ))< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< p class = "sphx-glr-script-out" > Out:< / p >
2021-04-23 16:42:55 -04:00
< div class = "sphx-glr-script-out highlight-none notranslate" > < div class = "highlight" > < pre > < span > < / span > tensor([[-4.5061e-05, 4.1656e+01, 1.7500e+01, ..., -2.7405e-02,
-2.3251e-03, -0.0000e+00],
[-1.0967e-04, -4.2915e-06, -0.0000e+00, ..., -1.4901e-06,
-0.0000e+00, 1.4367e+01],
[ 5.8156e+01, -0.0000e+00, -1.4603e-04, ..., 1.3930e+01,
-2.1362e-01, 9.4062e+00],
2021-03-15 13:58:20 -04:00
...,
2021-04-23 16:42:55 -04:00
[ 2.3703e+01, -9.2163e-02, -1.3471e-05, ..., -9.5215e-02,
2.0047e+01, 1.4891e+01],
[-1.9073e-06, 5.0664e+00, -0.0000e+00, ..., 2.0281e+01,
-1.7583e-05, 3.8000e+01],
[-1.7285e-05, 5.3945e+00, -1.3916e-01, ..., -2.0984e-01,
5.3750e+00, -1.5993e-03]], device=' cuda:0' , dtype=torch.float16)
tensor([[-4.4942e-05, 4.1656e+01, 1.7500e+01, ..., -2.7405e-02,
-2.3232e-03, -0.0000e+00],
[-1.1003e-04, -4.2915e-06, -0.0000e+00, ..., -1.4901e-06,
-0.0000e+00, 1.4367e+01],
[ 5.8156e+01, -0.0000e+00, -1.4639e-04, ..., 1.3930e+01,
-2.1362e-01, 9.4062e+00],
2021-03-15 13:58:20 -04:00
...,
2021-04-23 16:42:55 -04:00
[ 2.3703e+01, -9.2163e-02, -1.3471e-05, ..., -9.5276e-02,
2.0047e+01, 1.4891e+01],
[-1.9073e-06, 5.0664e+00, -0.0000e+00, ..., 2.0281e+01,
-1.7583e-05, 3.8000e+01],
[-1.7345e-05, 5.3945e+00, -1.3916e-01, ..., -2.0984e-01,
5.3750e+00, -1.6031e-03]], device=' cuda:0' , dtype=torch.float16)
2021-04-21 01:40:29 -04:00
tensor(True, device=' cuda:0' )
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
< / div >
< div class = "section" id = "benchmark" >
< h2 > Benchmark< a class = "headerlink" href = "#benchmark" title = "Permalink to this headline" > ¶< / a > < / h2 >
< div class = "section" id = "square-matrix-performance" >
< h3 > Square Matrix Performance< a class = "headerlink" href = "#square-matrix-performance" title = "Permalink to this headline" > ¶< / a > < / h3 >
< p > We can now compare the performance of our kernel against CUTLASS. Here we focus on square matrices, but feel free to arrange the script as you wish to compare any other matrix shape.#< / p >
< div class = "highlight-default notranslate" > < div class = "highlight" > < pre > < span > < / span > < span class = "nd" > @triton< / span > < span class = "o" > .< / span > < span class = "n" > testing< / span > < span class = "o" > .< / span > < span class = "n" > perf_report< / span > < span class = "p" > (< / span >
< span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > testing< / span > < span class = "o" > .< / span > < span class = "n" > Benchmark< / span > < span class = "p" > (< / span >
< span class = "n" > x_names< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "s1" > ' M' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' N' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' K' < / span > < span class = "p" > ],< / span > < span class = "c1" > # argument names to use as an x-axis for the plot< / span >
< span class = "n" > x_vals< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "mi" > 256< / span > < span class = "o" > *< / span > < span class = "n" > i< / span > < span class = "k" > for< / span > < span class = "n" > i< / span > < span class = "ow" > in< / span > < span class = "nb" > range< / span > < span class = "p" > (< / span > < span class = "mi" > 2< / span > < span class = "p" > ,< / span > < span class = "mi" > 33< / span > < span class = "p" > )],< / span > < span class = "c1" > # different possible values for `x_name`< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > line_arg< / span > < span class = "o" > =< / span > < span class = "s1" > ' provider' < / span > < span class = "p" > ,< / span > < span class = "c1" > # argument name whose value corresponds to a different line in the plot< / span >
< span class = "n" > line_vals< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "s1" > ' cublas' < / span > < span class = "p" > ,< / span > < span class = "s1" > ' triton' < / span > < span class = "p" > ],< / span > < span class = "c1" > # possible values for `line_arg``< / span >
< span class = "n" > line_names< / span > < span class = "o" > =< / span > < span class = "p" > [< / span > < span class = "s2" > " cuBLAS" < / span > < span class = "p" > ,< / span > < span class = "s2" > " Triton" < / span > < span class = "p" > ],< / span > < span class = "c1" > # label name for the lines< / span >
2021-03-15 13:58:20 -04:00
< span class = "n" > ylabel< / span > < span class = "o" > =< / span > < span class = "s2" > " TFLOPS" < / span > < span class = "p" > ,< / span > < span class = "c1" > # label name for the y-axis< / span >
< span class = "n" > plot_name< / span > < span class = "o" > =< / span > < span class = "s2" > " matmul-performance" < / span > < span class = "p" > ,< / span > < span class = "c1" > # name for the plot. Used also as a file name for saving the plot.< / span >
< span class = "n" > args< / span > < span class = "o" > =< / span > < span class = "p" > {}< / span >
< span class = "p" > )< / span >
< span class = "p" > )< / span >
< span class = "k" > def< / span > < span class = "nf" > benchmark< / span > < span class = "p" > (< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > < span class = "n" > provider< / span > < span class = "p" > ):< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > silu< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > nn< / span > < span class = "o" > .< / span > < span class = "n" > SiLU< / span > < span class = "p" > ()< / span >
2021-03-15 13:58:20 -04:00
< span class = "n" > a< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > randn< / span > < span class = "p" > ((< / span > < span class = "n" > M< / span > < span class = "p" > ,< / span > < span class = "n" > K< / span > < span class = "p" > ),< / span > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > float16< / span > < span class = "p" > )< / span >
< span class = "n" > b< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > randn< / span > < span class = "p" > ((< / span > < span class = "n" > K< / span > < span class = "p" > ,< / span > < span class = "n" > N< / span > < span class = "p" > ),< / span > < span class = "n" > device< / span > < span class = "o" > =< / span > < span class = "s1" > ' cuda' < / span > < span class = "p" > ,< / span > < span class = "n" > dtype< / span > < span class = "o" > =< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > float16< / span > < span class = "p" > )< / span >
2021-03-29 11:59:18 -04:00
< span class = "k" > if< / span > < span class = "n" > provider< / span > < span class = "o" > ==< / span > < span class = "s1" > ' cublas' < / span > < span class = "p" > :< / span >
2021-03-15 13:58:20 -04:00
< span class = "n" > ms< / span > < span class = "p" > ,< / span > < span class = "n" > min_ms< / span > < span class = "p" > ,< / span > < span class = "n" > max_ms< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > testing< / span > < span class = "o" > .< / span > < span class = "n" > do_bench< / span > < span class = "p" > (< / span > < span class = "k" > lambda< / span > < span class = "p" > :< / span > < span class = "n" > torch< / span > < span class = "o" > .< / span > < span class = "n" > matmul< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ))< / span >
< span class = "k" > if< / span > < span class = "n" > provider< / span > < span class = "o" > ==< / span > < span class = "s1" > ' triton' < / span > < span class = "p" > :< / span >
2021-04-21 01:40:29 -04:00
< span class = "n" > ms< / span > < span class = "p" > ,< / span > < span class = "n" > min_ms< / span > < span class = "p" > ,< / span > < span class = "n" > max_ms< / span > < span class = "o" > =< / span > < span class = "n" > triton< / span > < span class = "o" > .< / span > < span class = "n" > testing< / span > < span class = "o" > .< / span > < span class = "n" > do_bench< / span > < span class = "p" > (< / span > < span class = "k" > lambda< / span > < span class = "p" > :< / span > < span class = "n" > matmul< / span > < span class = "p" > (< / span > < span class = "n" > a< / span > < span class = "p" > ,< / span > < span class = "n" > b< / span > < span class = "p" > ))< / span >
2021-03-15 13:58:20 -04:00
< span class = "n" > perf< / span > < span class = "o" > =< / span > < span class = "k" > lambda< / span > < span class = "n" > ms< / span > < span class = "p" > :< / span > < span class = "mi" > 2< / span > < span class = "o" > *< / span > < span class = "n" > M< / span > < span class = "o" > *< / span > < span class = "n" > N< / span > < span class = "o" > *< / span > < span class = "n" > K< / span > < span class = "o" > *< / span > < span class = "mf" > 1e-12< / span > < span class = "o" > /< / span > < span class = "p" > (< / span > < span class = "n" > ms< / span > < span class = "o" > *< / span > < span class = "mf" > 1e-3< / span > < span class = "p" > )< / span >
< span class = "k" > return< / span > < span class = "n" > perf< / span > < span class = "p" > (< / span > < span class = "n" > ms< / span > < span class = "p" > ),< / span > < span class = "n" > perf< / span > < span class = "p" > (< / span > < span class = "n" > max_ms< / span > < span class = "p" > ),< / span > < span class = "n" > perf< / span > < span class = "p" > (< / span > < span class = "n" > min_ms< / span > < span class = "p" > )< / span >
2021-04-23 16:42:55 -04:00
< span class = "n" > benchmark< / span > < span class = "o" > .< / span > < span class = "n" > run< / span > < span class = "p" > (< / span > < span class = "n" > show_plots< / span > < span class = "o" > =< / span > < span class = "kc" > True< / span > < span class = "p" > ,< / span > < span class = "n" > print_data< / span > < span class = "o" > =< / span > < span class = "kc" > True< / span > < span class = "p" > )< / span >
2021-03-15 13:58:20 -04:00
< / pre > < / div >
< / div >
2021-03-29 11:59:18 -04:00
< img alt = "03 matrix multiplication" class = "sphx-glr-single-img" src = "../../_images/sphx_glr_03-matrix-multiplication_001.png" / >
2021-04-21 01:40:29 -04:00
< p class = "sphx-glr-script-out" > Out:< / p >
< div class = "sphx-glr-script-out highlight-none notranslate" > < div class = "highlight" > < pre > < span > < / span > M cuBLAS Triton
0 512.0 20.164923 15.420235
2021-04-21 01:58:48 -04:00
1 768.0 58.982401 40.215272
2021-04-23 16:42:55 -04:00
2 1024.0 95.325090 72.315584
3 1280.0 151.703703 117.028568
4 1536.0 153.867127 150.593357
2021-04-21 01:58:48 -04:00
5 1792.0 208.137481 190.498706
2021-04-23 16:42:55 -04:00
6 2048.0 202.135135 151.146088
7 2304.0 251.451276 178.267699
8 2560.0 237.449270 218.453323
9 2816.0 238.329010 200.987140
10 3072.0 243.017615 223.806730
11 3328.0 244.868356 210.500857
12 3584.0 250.460703 232.941430
13 3840.0 256.593972 225.697957
14 4096.0 266.305018 247.634187
15 4352.0 247.675667 237.797917
16 4608.0 280.621108 260.713476
17 4864.0 272.431168 252.534501
18 5120.0 265.596772 245.223576
19 5376.0 261.381955 244.335299
20 5632.0 283.439220 260.383339
21 5888.0 276.674704 254.103421
22 6144.0 274.869441 252.078378
23 6400.0 269.190319 249.027231
24 6656.0 269.252160 249.104840
25 6912.0 267.069377 247.115909
26 7168.0 268.504352 246.006552
27 7424.0 267.373291 246.355964
28 7680.0 266.406511 245.760004
29 7936.0 228.348876 248.331598
30 8192.0 227.680622 247.977332
2021-04-21 01:40:29 -04:00
< / pre > < / div >
< / div >
2021-04-23 16:42:55 -04:00
< p class = "sphx-glr-timing" > < strong > Total running time of the script:< / strong > ( 0 minutes 37.657 seconds)< / p >
2021-03-15 13:58:20 -04:00
< div class = "sphx-glr-footer class sphx-glr-footer-example docutils container" id = "sphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py" >
< div class = "sphx-glr-download sphx-glr-download-python docutils container" >
< p > < a class = "reference download internal" download = "" href = "../../_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py" > < code class = "xref download docutils literal notranslate" > < span class = "pre" > Download< / span > < span class = "pre" > Python< / span > < span class = "pre" > source< / span > < span class = "pre" > code:< / span > < span class = "pre" > 03-matrix-multiplication.py< / span > < / code > < / a > < / p >
< / div >
< div class = "sphx-glr-download sphx-glr-download-jupyter docutils container" >
< p > < a class = "reference download internal" download = "" href = "../../_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb" > < code class = "xref download docutils literal notranslate" > < span class = "pre" > Download< / span > < span class = "pre" > Jupyter< / span > < span class = "pre" > notebook:< / span > < span class = "pre" > 03-matrix-multiplication.ipynb< / span > < / code > < / a > < / p >
< / div >
< / div >
< p class = "sphx-glr-signature" > < a class = "reference external" href = "https://sphinx-gallery.github.io" > Gallery generated by Sphinx-Gallery< / a > < / p >
< / div >
< / div >
< / div >
< / div >
< / div >
< footer >
< div class = "rst-footer-buttons" role = "navigation" aria-label = "footer navigation" >
2021-04-23 16:42:55 -04:00
< a href = "../../python-api/triton.html" class = "btn btn-neutral float-right" title = "triton" accesskey = "n" rel = "next" > Next < span class = "fa fa-arrow-circle-right" aria-hidden = "true" > < / span > < / a >
2021-03-15 13:58:20 -04:00
< a href = "02-fused-softmax.html" class = "btn btn-neutral float-left" title = "Fused Softmax" accesskey = "p" rel = "prev" > < span class = "fa fa-arrow-circle-left" aria-hidden = "true" > < / span > Previous< / a >
< / div >
< hr / >
< div role = "contentinfo" >
< p >
© Copyright 2020, Philippe Tillet.
< / p >
< / div >
Built with < a href = "https://www.sphinx-doc.org/" > Sphinx< / a > using a
< a href = "https://github.com/readthedocs/sphinx_rtd_theme" > theme< / a >
provided by < a href = "https://readthedocs.org" > Read the Docs< / a > .
< / footer >
< / div >
< / div >
< / section >
< / div >
< script type = "text/javascript" >
jQuery(function () {
SphinxRtdTheme.Navigation.enable(true);
});
< / script >
< / body >
< / html >
