[GH-PAGES] Updated website

_sources/getting-started/tutorials/01-vector-add.rst.txt

@@ -216,13 +216,13 @@ We can now run the decorated function above. Pass `print_data=True` to see the p
 
     vector-add-performance:
                size      Triton       Torch
-    0        4096.0    9.540372    9.600000
+    0        4096.0    9.600000    9.600000
     1        8192.0   19.200000   19.200000
     2       16384.0   38.400001   38.400001
     3       32768.0   76.800002   76.800002
     4       65536.0  127.999995  127.999995
     5      131072.0  219.428568  219.428568
-    6      262144.0  341.333321  341.333321
+    6      262144.0  341.333321  384.000001
     7      524288.0  472.615390  472.615390
     8     1048576.0  614.400016  614.400016
     9     2097152.0  722.823517  722.823517
@@ -239,7 +239,7 @@ We can now run the decorated function above. Pass `print_data=True` to see the p
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 0 minutes  11.067 seconds)
+   **Total running time of the script:** ( 0 minutes  11.032 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_01-vector-add.py:

_sources/getting-started/tutorials/02-fused-softmax.rst.txt

@@ -262,15 +262,15 @@ We will then compare its performance against (1) :code:`torch.softmax` and (2) t
     softmax-performance:
               N      Triton  Torch (native)  Torch (jit)
     0     256.0  512.000001      546.133347   273.066674
-    1     384.0  585.142862      585.142862   261.446801
+    1     384.0  585.142862      585.142862   267.130429
     2     512.0  630.153853      606.814814   264.258068
-    3     640.0  682.666684      640.000002   265.974036
+    3     640.0  682.666684      640.000002   269.473696
     4     768.0  702.171410      664.216187   273.066663
     ..      ...         ...             ...          ...
-    93  12160.0  812.359066      405.755985   329.483481
+    93  12160.0  812.359066      406.179533   329.483481
-    94  12288.0  812.429770      415.222812   329.602681
+    94  12288.0  812.429770      415.661740   329.602681
     95  12416.0  810.840807      412.149375   329.173158
-    96  12544.0  810.925276      412.971190   329.022957
+    96  12544.0  810.925276      412.546756   329.292871
     97  12672.0  811.007961      412.097543   329.410251
 
     [98 rows x 4 columns]
@@ -290,7 +290,7 @@ In the above plot, we can see that:
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 1 minutes  8.169 seconds)
+   **Total running time of the script:** ( 1 minutes  8.174 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_02-fused-softmax.py:

_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt

@@ -371,37 +371,37 @@ We can now compare the performance of our kernel against that of cuBLAS. Here we
     matmul-performance:
              M     cuBLAS  ...     Triton  Triton (+ LeakyReLU)
     0    128.0   0.455111  ...   0.512000              0.512000
-    1    256.0   2.730667  ...   2.978909              2.978909
+    1    256.0   2.978909  ...   2.978909              2.978909
-    2    384.0   7.372800  ...   8.507077              8.507077
+    2    384.0   7.372800  ...   7.899428              7.899428
-    3    512.0  14.563555  ...  15.420235             15.420235
+    3    512.0  14.563555  ...  16.384000             15.420235
-    4    640.0  22.260869  ...  23.272727             23.272727
+    4    640.0  22.260869  ...  24.380953             24.380953
     5    768.0  32.768000  ...  34.028308             34.028308
     6    896.0  39.025776  ...  39.025776             39.025776
-    7   1024.0  49.932191  ...  52.428801             52.428801
+    7   1024.0  51.150050  ...  52.428801             52.428801
     8   1152.0  44.566925  ...  46.656000             46.656000
     9   1280.0  51.200001  ...  56.109587             56.109587
     10  1408.0  64.138541  ...  65.684049             65.684049
-    11  1536.0  80.430545  ...  76.106321             76.106321
+    11  1536.0  80.430545  ...  76.106321             75.296679
-    12  1664.0  63.372618  ...  61.636381             61.636381
+    12  1664.0  63.372618  ...  62.061463             61.636381
-    13  1792.0  72.983276  ...  69.379162             68.953520
+    13  1792.0  72.983276  ...  68.953520             68.533074
-    14  1920.0  68.098521  ...  69.818184             69.818184
+    14  1920.0  69.120002  ...  68.435645             68.435645
-    15  2048.0  73.584279  ...  75.234154             75.573044
+    15  2048.0  73.908442  ...  75.573044             75.234154
-    16  2176.0  83.500614  ...  80.817862             80.494588
+    16  2176.0  83.500614  ...  80.173899             79.855747
-    17  2304.0  68.251065  ...  73.051599             73.051599
+    17  2304.0  68.446623  ...  73.051599             72.607513
-    18  2432.0  71.305746  ...  81.197876             81.197876
+    18  2432.0  71.125224  ...  81.197876             80.963875
-    19  2560.0  77.833728  ...  76.740048             76.382283
+    19  2560.0  77.649287  ...  76.027843             76.740048
-    20  2688.0  83.004501  ...  83.369354             85.051697
+    20  2688.0  83.552988  ...  83.186525             82.823267
-    21  2816.0  81.067298  ...  75.982940             78.868366
+    21  2816.0  84.035084  ...  76.921000             79.733474
-    22  2944.0  79.483304  ...  79.865439             79.230573
+    22  2944.0  82.102191  ...  80.122235             78.729910
-    23  3072.0  81.589488  ...  83.146995             83.025078
+    23  3072.0  82.540970  ...  82.661468             82.661468
-    24  3200.0  84.432717  ...  86.956520             89.385477
+    24  3200.0  84.432717  ...  89.385477             84.432717
-    25  3328.0  84.003845  ...  82.843841             85.806075
+    25  3328.0  83.905938  ...  86.113988             86.528001
-    26  3456.0  82.350937  ...  84.686523             84.508982
+    26  3456.0  82.015834  ...  83.545665             84.156124
-    27  3584.0  84.111686  ...  92.032132             96.475743
+    27  3584.0  87.466332  ...  92.600816             84.988707
-    28  3712.0  86.416391  ...  84.874549             88.170647
+    28  3712.0  85.163978  ...  82.902362             83.666116
-    29  3840.0  85.005380  ...  87.771425             87.493673
+    29  3840.0  84.292684  ...  84.550462             85.070769
-    30  3968.0  92.512459  ...  80.703662             84.797731
+    30  3968.0  89.921841  ...  87.472354             87.409694
-    31  4096.0  93.727466  ...  90.995066             91.741443
+    31  4096.0  93.792965  ...  89.478485             90.260743
 
     [32 rows x 5 columns]
 
@@ -411,7 +411,7 @@ We can now compare the performance of our kernel against that of cuBLAS. Here we
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 1 minutes  58.805 seconds)
+   **Total running time of the script:** ( 2 minutes  2.376 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py:

_sources/getting-started/tutorials/sg_execution_times.rst.txt

@@ -5,12 +5,12 @@
 
 Computation times
 =================
-**03:18.042** total execution time for **getting-started_tutorials** files:
+**03:21.583** total execution time for **getting-started_tutorials** files:
 
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 01:58.805 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 02:02.376 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``)                 | 01:08.169 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``)                 | 01:08.174 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``)                       | 00:11.067 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``)                       | 00:11.032 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+

_sources/programming-guide/chapter-2/related-work.rst.txt

@@ -2,7 +2,7 @@
 Related Work
 ==============
 
-At first sight, Triton may seem like just yet another DSL for DNNs. The purpose of this section is to contextualize Triton and highlights its differences with the two leading approaches in this domain: polyhedral compilation and scheduling languages.
+At first sight, Triton may seem like just yet another DSL for DNNs. The purpose of this section is to contextualize Triton and highlight its differences with the two leading approaches in this domain: polyhedral compilation and scheduling languages.
 
 -----------------------
 Polyhedral Compilation
@@ -121,7 +121,7 @@ Limitations
 
 Unfortunately, polyhedral compilers suffer from two major limitations that have prevented its adoption as a universal method for code generation in neural networks.
 
-First, the set of possible program transformations $\Omega = \{ \Theta_S ~|~ S \in \text{program} \}$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures [SATO2019]_.
+First, the set of possible program transformations :math:`\Omega = \{ \Theta_S ~|~ S \in \text{program} \}` is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures [SATO2019]_.
 
 Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common [GIRBAL2006]_ but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse -- or even structured-sparse -- neural networks, whose importance has been rapidly rising over the past few years.
 
@@ -131,7 +131,7 @@ On the other hand, blocked program representations advocated by this dissertatio
 Scheduling Languages
 -----------------------
 
-Separation of concerns \cite{dijkstra82} is a well-known design principle in computer science: programs should be decomposed into modular layers of abstraction that separate the semantics of their algorithms from the details of their implementation. Systems like Halide and TVM push this philosophy one step further, and enforce this separation at the grammatical level through the use of a  **scheduling language**. The benefits of this methodology are particularly visible in the case of matrix multiplication, where, as one can see below, the definition of the algorithm (Line 1-7) is completely disjoint from its implementation (Line 8-16), meaning that both can be maintained, optimized and distributed independently. 
+Separation of concerns [DIJKSTRA82]_ is a well-known design principle in computer science: programs should be decomposed into modular layers of abstraction that separate the semantics of their algorithms from the details of their implementation. Systems like Halide and TVM push this philosophy one step further, and enforce this separation at the grammatical level through the use of a  **scheduling language**. The benefits of this methodology are particularly visible in the case of matrix multiplication, where, as one can see below, the definition of the algorithm (Line 1-7) is completely disjoint from its implementation (Line 8-16), meaning that both can be maintained, optimized and distributed independently. 
 
 .. code-block:: python
   :linenos:
@@ -168,7 +168,7 @@ Scheduling languages are, without a doubt, one of the most popular approaches fo
 Limitations
 ++++++++++++
 
-This ease-of-development comes at a cost. First of all, existing systems that follow this paradigm tend to be noticeably slower than Triton on modern hardware when applicable (e.g., V100/A100 tensor cores w/ equal tile sizes). I do believe that this is not a fundamental issue of scheduling languages -- in the sense that it could probably be solved with more efforts -- but it could mean that these systems are harder to engineer. More importantly, existing scheduling languages generate loops whose bounds and increments cannot depend on surrounding loop indice without at least imposing severe constraints on possible schedules -- if not breaking the system entirely. This is problematic for sparse com-putations, whose iteration spaces may be irregular.
+This ease-of-development comes at a cost. First of all, existing systems that follow this paradigm tend to be noticeably slower than Triton on modern hardware when applicable (e.g., V100/A100 tensor cores w/ equal tile sizes). I do believe that this is not a fundamental issue of scheduling languages -- in the sense that it could probably be solved with more efforts -- but it could mean that these systems are harder to engineer. More importantly, existing scheduling languages generate loops whose bounds and increments cannot depend on surrounding loop indice without at least imposing severe constraints on possible schedules -- if not breaking the system entirely. This is problematic for sparse computations, whose iteration spaces may be irregular.
 
 .. table::
     :widths: 50 50
@@ -206,4 +206,5 @@ References
 .. [GROSSER2012] T. Grosser et al., "Polly - Performing Polyhedral Optimizations on a Low-Level Intermediate Representation", Parallel Processing Letters 2012
 .. [SATO2019] Y. Sato et al., "An Autotuning Framework for Scalable Execution of Tiled Code via Iterative Polyhedral Compilation", TACO 2019
 .. [GIRBAL2006] S. Girbal et al., "Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies", International Journal of Parallel Programming 2006
-.. [MULLAPUDI2016] R. Mullapudi et al., "Automatically scheduling halide image processing pipelines", TOG 2016
+.. [DIJKSTRA82] E. W. Dijkstra et al., "On the role of scientific thought", Selected writings on computing: a personal perspective 1982
+.. [MULLAPUDI2016] R. Mullapudi et al., "Automatically scheduling halide image processing pipelines", TOG 2016

getting-started/tutorials/01-vector-add.html

@@ -305,13 +305,13 @@ for different problem sizes.</p>
 <p class="sphx-glr-script-out">Out:</p>
 <div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>vector-add-performance:
            size      Triton       Torch
-0        4096.0    9.540372    9.600000
+0        4096.0    9.600000    9.600000
 1        8192.0   19.200000   19.200000
 2       16384.0   38.400001   38.400001
 3       32768.0   76.800002   76.800002
 4       65536.0  127.999995  127.999995
 5      131072.0  219.428568  219.428568
-6      262144.0  341.333321  341.333321
+6      262144.0  341.333321  384.000001
 7      524288.0  472.615390  472.615390
 8     1048576.0  614.400016  614.400016
 9     2097152.0  722.823517  722.823517
@@ -323,7 +323,7 @@ for different problem sizes.</p>
 15  134217728.0  851.577704  850.656574
 </pre></div>
 </div>
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  11.067 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  11.032 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-01-vector-add-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/62d97d49a32414049819dd8bb8378080/01-vector-add.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">01-vector-add.py</span></code></a></p>

getting-started/tutorials/02-fused-softmax.html

@@ -347,15 +347,15 @@ We will then compare its performance against (1) <code class="code docutils lite
 <div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>softmax-performance:
           N      Triton  Torch (native)  Torch (jit)
 0     256.0  512.000001      546.133347   273.066674
-1     384.0  585.142862      585.142862   261.446801
+1     384.0  585.142862      585.142862   267.130429
 2     512.0  630.153853      606.814814   264.258068
-3     640.0  682.666684      640.000002   265.974036
+3     640.0  682.666684      640.000002   269.473696
 4     768.0  702.171410      664.216187   273.066663
 ..      ...         ...             ...          ...
-93  12160.0  812.359066      405.755985   329.483481
+93  12160.0  812.359066      406.179533   329.483481
-94  12288.0  812.429770      415.222812   329.602681
+94  12288.0  812.429770      415.661740   329.602681
 95  12416.0  810.840807      412.149375   329.173158
-96  12544.0  810.925276      412.971190   329.022957
+96  12544.0  810.925276      412.546756   329.292871
 97  12672.0  811.007961      412.097543   329.410251
 
 [98 rows x 4 columns]
@@ -370,7 +370,7 @@ This means that – when temporary data is too large to fit entirely in the GPU
 Note that our Triton kernel is not only faster than PyTorch’s CUDA kernel, it is also <strong>easier to read, understand and maintain</strong>.</p></li>
 </ul>
 </div></blockquote>
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 1 minutes  8.169 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 1 minutes  8.174 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-02-fused-softmax-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/d91442ac2982c4e0cc3ab0f43534afbc/02-fused-softmax.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">02-fused-softmax.py</span></code></a></p>

getting-started/tutorials/03-matrix-multiplication.html

@@ -476,42 +476,42 @@ tensor(True, device='cuda:0')
 <div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>matmul-performance:
          M     cuBLAS  ...     Triton  Triton (+ LeakyReLU)
 0    128.0   0.455111  ...   0.512000              0.512000
-1    256.0   2.730667  ...   2.978909              2.978909
+1    256.0   2.978909  ...   2.978909              2.978909
-2    384.0   7.372800  ...   8.507077              8.507077
+2    384.0   7.372800  ...   7.899428              7.899428
-3    512.0  14.563555  ...  15.420235             15.420235
+3    512.0  14.563555  ...  16.384000             15.420235
-4    640.0  22.260869  ...  23.272727             23.272727
+4    640.0  22.260869  ...  24.380953             24.380953
 5    768.0  32.768000  ...  34.028308             34.028308
 6    896.0  39.025776  ...  39.025776             39.025776
-7   1024.0  49.932191  ...  52.428801             52.428801
+7   1024.0  51.150050  ...  52.428801             52.428801
 8   1152.0  44.566925  ...  46.656000             46.656000
 9   1280.0  51.200001  ...  56.109587             56.109587
 10  1408.0  64.138541  ...  65.684049             65.684049
-11  1536.0  80.430545  ...  76.106321             76.106321
+11  1536.0  80.430545  ...  76.106321             75.296679
-12  1664.0  63.372618  ...  61.636381             61.636381
+12  1664.0  63.372618  ...  62.061463             61.636381
-13  1792.0  72.983276  ...  69.379162             68.953520
+13  1792.0  72.983276  ...  68.953520             68.533074
-14  1920.0  68.098521  ...  69.818184             69.818184
+14  1920.0  69.120002  ...  68.435645             68.435645
-15  2048.0  73.584279  ...  75.234154             75.573044
+15  2048.0  73.908442  ...  75.573044             75.234154
-16  2176.0  83.500614  ...  80.817862             80.494588
+16  2176.0  83.500614  ...  80.173899             79.855747
-17  2304.0  68.251065  ...  73.051599             73.051599
+17  2304.0  68.446623  ...  73.051599             72.607513
-18  2432.0  71.305746  ...  81.197876             81.197876
+18  2432.0  71.125224  ...  81.197876             80.963875
-19  2560.0  77.833728  ...  76.740048             76.382283
+19  2560.0  77.649287  ...  76.027843             76.740048
-20  2688.0  83.004501  ...  83.369354             85.051697
+20  2688.0  83.552988  ...  83.186525             82.823267
-21  2816.0  81.067298  ...  75.982940             78.868366
+21  2816.0  84.035084  ...  76.921000             79.733474
-22  2944.0  79.483304  ...  79.865439             79.230573
+22  2944.0  82.102191  ...  80.122235             78.729910
-23  3072.0  81.589488  ...  83.146995             83.025078
+23  3072.0  82.540970  ...  82.661468             82.661468
-24  3200.0  84.432717  ...  86.956520             89.385477
+24  3200.0  84.432717  ...  89.385477             84.432717
-25  3328.0  84.003845  ...  82.843841             85.806075
+25  3328.0  83.905938  ...  86.113988             86.528001
-26  3456.0  82.350937  ...  84.686523             84.508982
+26  3456.0  82.015834  ...  83.545665             84.156124
-27  3584.0  84.111686  ...  92.032132             96.475743
+27  3584.0  87.466332  ...  92.600816             84.988707
-28  3712.0  86.416391  ...  84.874549             88.170647
+28  3712.0  85.163978  ...  82.902362             83.666116
-29  3840.0  85.005380  ...  87.771425             87.493673
+29  3840.0  84.292684  ...  84.550462             85.070769
-30  3968.0  92.512459  ...  80.703662             84.797731
+30  3968.0  89.921841  ...  87.472354             87.409694
-31  4096.0  93.727466  ...  90.995066             91.741443
+31  4096.0  93.792965  ...  89.478485             90.260743
 
 [32 rows x 5 columns]
 </pre></div>
 </div>
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 1 minutes  58.805 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 2 minutes  2.376 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">03-matrix-multiplication.py</span></code></a></p>

getting-started/tutorials/sg_execution_times.html

@@ -174,7 +174,7 @@
             
   <div class="section" id="computation-times">
 <span id="sphx-glr-getting-started-tutorials-sg-execution-times"></span><h1>Computation times<a class="headerlink" href="#computation-times" title="Permalink to this headline">¶</a></h1>
-<p><strong>03:18.042</strong> total execution time for <strong>getting-started_tutorials</strong> files:</p>
+<p><strong>03:21.583</strong> total execution time for <strong>getting-started_tutorials</strong> files:</p>
 <table class="docutils align-default">
 <colgroup>
 <col style="width: 85%" />
@@ -183,15 +183,15 @@
 </colgroup>
 <tbody>
 <tr class="row-odd"><td><p><a class="reference internal" href="03-matrix-multiplication.html#sphx-glr-getting-started-tutorials-03-matrix-multiplication-py"><span class="std std-ref">Matrix Multiplication</span></a> (<code class="docutils literal notranslate"><span class="pre">03-matrix-multiplication.py</span></code>)</p></td>
-<td><p>01:58.805</p></td>
+<td><p>02:02.376</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
 <tr class="row-even"><td><p><a class="reference internal" href="02-fused-softmax.html#sphx-glr-getting-started-tutorials-02-fused-softmax-py"><span class="std std-ref">Fused Softmax</span></a> (<code class="docutils literal notranslate"><span class="pre">02-fused-softmax.py</span></code>)</p></td>
-<td><p>01:08.169</p></td>
+<td><p>01:08.174</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
 <tr class="row-odd"><td><p><a class="reference internal" href="01-vector-add.html#sphx-glr-getting-started-tutorials-01-vector-add-py"><span class="std std-ref">Vector Addition</span></a> (<code class="docutils literal notranslate"><span class="pre">01-vector-add.py</span></code>)</p></td>
-<td><p>00:11.067</p></td>
+<td><p>00:11.032</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
 </tbody>

programming-guide/chapter-2/related-work.html

@@ -114,8 +114,8 @@
 </ul>
 </li>
 <li class="toctree-l2"><a class="reference internal" href="#scheduling-languages">Scheduling Languages</a><ul>
-<li class="toctree-l3"><a class="reference internal" href="#id15">Advantages</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#id16">Advantages</a></li>
-<li class="toctree-l3"><a class="reference internal" href="#id16">Limitations</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#id17">Limitations</a></li>
 </ul>
 </li>
 <li class="toctree-l2"><a class="reference internal" href="#references">References</a></li>
@@ -190,7 +190,7 @@
             
   <div class="section" id="related-work">
 <h1>Related Work<a class="headerlink" href="#related-work" title="Permalink to this headline">¶</a></h1>
-<p>At first sight, Triton may seem like just yet another DSL for DNNs. The purpose of this section is to contextualize Triton and highlights its differences with the two leading approaches in this domain: polyhedral compilation and scheduling languages.</p>
+<p>At first sight, Triton may seem like just yet another DSL for DNNs. The purpose of this section is to contextualize Triton and highlight its differences with the two leading approaches in this domain: polyhedral compilation and scheduling languages.</p>
 <div class="section" id="polyhedral-compilation">
 <h2>Polyhedral Compilation<a class="headerlink" href="#polyhedral-compilation" title="Permalink to this headline">¶</a></h2>
 <p>Traditional compilers typically rely on intermediate representations, such as LLVM-IR <a class="reference internal" href="#lattner2004" id="id1"><span>[LATTNER2004]</span></a>, that encode control flow information using (un)conditional branches. This relatively low-level format makes it difficult to statically analyze the runtime behavior (e.g., cache misses) of input programs, and to  automatically optimize loops accordingly through the use of tiling <a class="reference internal" href="#wolfe1989" id="id2"><span>[WOLFE1989]</span></a>, fusion <a class="reference internal" href="#darte1999" id="id3"><span>[DARTE1999]</span></a> and interchange <a class="reference internal" href="#allen1984" id="id4"><span>[ALLEN1984]</span></a>. To solve this issue, polyhedral compilers <a class="reference internal" href="#ancourt1991" id="id5"><span>[ANCOURT1991]</span></a> rely on program representations that have statically predictable control flow, thereby enabling aggressive compile-time program transformations for data locality and parallelism. Though this strategy has been adopted by many languages and compilers for DNNs such as Tiramisu <a class="reference internal" href="#baghdadi2021" id="id6"><span>[BAGHDADI2021]</span></a>, Tensor Comprehensions <a class="reference internal" href="#vasilache2018" id="id7"><span>[VASILACHE2018]</span></a>, Diesel <a class="reference internal" href="#elango2018" id="id8"><span>[ELANGO2018]</span></a> and the Affine dialect in MLIR <a class="reference internal" href="#lattner2019" id="id9"><span>[LATTNER2019]</span></a>, it also comes with a number of limitations that will be described later in this section.</p>
@@ -282,14 +282,14 @@ i &amp; j
 <div class="section" id="limitations">
 <h3>Limitations<a class="headerlink" href="#limitations" title="Permalink to this headline">¶</a></h3>
 <p>Unfortunately, polyhedral compilers suffer from two major limitations that have prevented its adoption as a universal method for code generation in neural networks.</p>
-<p>First, the set of possible program transformations $Omega = { Theta_S ~|~ S in text{program} }$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures <a class="reference internal" href="#sato2019" id="id12"><span>[SATO2019]</span></a>.</p>
+<p>First, the set of possible program transformations <span class="math notranslate nohighlight">\(\Omega = \{ \Theta_S ~|~ S \in \text{program} \}\)</span> is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures <a class="reference internal" href="#sato2019" id="id12"><span>[SATO2019]</span></a>.</p>
 <p>Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common <a class="reference internal" href="#girbal2006" id="id13"><span>[GIRBAL2006]</span></a> but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse – or even structured-sparse – neural networks, whose importance has been rapidly rising over the past few years.</p>
 <p>On the other hand, blocked program representations advocated by this dissertation are less restricted in scope and can achieve close to peak performance using standard dataflow analysis.</p>
 </div>
 </div>
 <div class="section" id="scheduling-languages">
 <h2>Scheduling Languages<a class="headerlink" href="#scheduling-languages" title="Permalink to this headline">¶</a></h2>
-<p>Separation of concerns cite{dijkstra82} is a well-known design principle in computer science: programs should be decomposed into modular layers of abstraction that separate the semantics of their algorithms from the details of their implementation. Systems like Halide and TVM push this philosophy one step further, and enforce this separation at the grammatical level through the use of a  <strong>scheduling language</strong>. The benefits of this methodology are particularly visible in the case of matrix multiplication, where, as one can see below, the definition of the algorithm (Line 1-7) is completely disjoint from its implementation (Line 8-16), meaning that both can be maintained, optimized and distributed independently.</p>
+<p>Separation of concerns <a class="reference internal" href="#dijkstra82" id="id14"><span>[DIJKSTRA82]</span></a> is a well-known design principle in computer science: programs should be decomposed into modular layers of abstraction that separate the semantics of their algorithms from the details of their implementation. Systems like Halide and TVM push this philosophy one step further, and enforce this separation at the grammatical level through the use of a  <strong>scheduling language</strong>. The benefits of this methodology are particularly visible in the case of matrix multiplication, where, as one can see below, the definition of the algorithm (Line 1-7) is completely disjoint from its implementation (Line 8-16), meaning that both can be maintained, optimized and distributed independently.</p>
 <div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="linenos"> 1</span><span class="o">//</span> <span class="n">algorithm</span>
 <span class="linenos"> 2</span><span class="n">Var</span> <span class="n">x</span><span class="p">(</span><span class="s2">&quot;x&quot;</span><span class="p">),</span> <span class="n">y</span><span class="p">(</span><span class="s2">&quot;y&quot;</span><span class="p">);</span>
 <span class="linenos"> 3</span><span class="n">Func</span> <span class="n">matmul</span><span class="p">(</span><span class="s2">&quot;matmul&quot;</span><span class="p">);</span>
@@ -308,15 +308,15 @@ i &amp; j
 <span class="linenos">16</span>    <span class="o">.</span><span class="n">parallel</span><span class="p">(</span><span class="n">y</span><span class="p">)</span><span class="o">.</span><span class="n">vectorize</span><span class="p">(</span><span class="n">xii</span><span class="p">)</span><span class="o">.</span><span class="n">unroll</span><span class="p">(</span><span class="n">xi</span><span class="p">)</span><span class="o">.</span><span class="n">unroll</span><span class="p">(</span><span class="n">yii</span><span class="p">);</span>
 </pre></div>
 </div>
-<p>The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms <a class="reference internal" href="#mullapudi2016" id="id14"><span>[MULLAPUDI2016]</span></a>.</p>
+<p>The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms <a class="reference internal" href="#mullapudi2016" id="id15"><span>[MULLAPUDI2016]</span></a>.</p>
-<div class="section" id="id15">
+<div class="section" id="id16">
-<h3>Advantages<a class="headerlink" href="#id15" title="Permalink to this headline">¶</a></h3>
+<h3>Advantages<a class="headerlink" href="#id16" title="Permalink to this headline">¶</a></h3>
 <p>The main advantage of this approach is that it allows programmers to write an algorithm <em>only once</em>, and focus on performance optimization separately. It makes it possible to manually specify optimizations that a polyhedral compiler wouldn’t be able to figure out automatically using static data-flow analysis.</p>
 <p>Scheduling languages are, without a doubt, one of the most popular approaches for neural network code generation. The most popular system for this purpose is probably TVM, which provides good performance across a wide range of platforms as well as built-in automatic scheduling mechanisms.</p>
 </div>
-<div class="section" id="id16">
+<div class="section" id="id17">
-<h3>Limitations<a class="headerlink" href="#id16" title="Permalink to this headline">¶</a></h3>
+<h3>Limitations<a class="headerlink" href="#id17" title="Permalink to this headline">¶</a></h3>
-<p>This ease-of-development comes at a cost. First of all, existing systems that follow this paradigm tend to be noticeably slower than Triton on modern hardware when applicable (e.g., V100/A100 tensor cores w/ equal tile sizes). I do believe that this is not a fundamental issue of scheduling languages – in the sense that it could probably be solved with more efforts – but it could mean that these systems are harder to engineer. More importantly, existing scheduling languages generate loops whose bounds and increments cannot depend on surrounding loop indice without at least imposing severe constraints on possible schedules – if not breaking the system entirely. This is problematic for sparse com-putations, whose iteration spaces may be irregular.</p>
+<p>This ease-of-development comes at a cost. First of all, existing systems that follow this paradigm tend to be noticeably slower than Triton on modern hardware when applicable (e.g., V100/A100 tensor cores w/ equal tile sizes). I do believe that this is not a fundamental issue of scheduling languages – in the sense that it could probably be solved with more efforts – but it could mean that these systems are harder to engineer. More importantly, existing scheduling languages generate loops whose bounds and increments cannot depend on surrounding loop indice without at least imposing severe constraints on possible schedules – if not breaking the system entirely. This is problematic for sparse computations, whose iteration spaces may be irregular.</p>
 <table class="colwidths-given docutils align-default">
 <colgroup>
 <col style="width: 50%" />
@@ -402,7 +402,15 @@ i &amp; j
 <li><p>Girbal et al., “Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies”, International Journal of Parallel Programming 2006</p></li>
 </ol>
 </dd>
-<dt class="label" id="mullapudi2016"><span class="brackets"><a class="fn-backref" href="#id14">MULLAPUDI2016</a></span></dt>
+<dt class="label" id="dijkstra82"><span class="brackets"><a class="fn-backref" href="#id14">DIJKSTRA82</a></span></dt>
+<dd><ol class="upperalpha simple" start="5">
+<li><ol class="upperalpha simple" start="23">
+<li><p>Dijkstra et al., “On the role of scientific thought”, Selected writings on computing: a personal perspective 1982</p></li>
+</ol>
+</li>
+</ol>
+</dd>
+<dt class="label" id="mullapudi2016"><span class="brackets"><a class="fn-backref" href="#id15">MULLAPUDI2016</a></span></dt>
 <dd><ol class="upperalpha simple" start="18">
 <li><p>Mullapudi et al., “Automatically scheduling halide image processing pipelines”, TOG 2016</p></li>
 </ol>