[GH-PAGES] Updated website

_downloads/b51b68bc1c6b1a5e509f67800b6235af/03-matrix-multiplication.ipynb

@@ -47,7 +47,7 @@
       },
       "outputs": [],
       "source": [
-        "import torch\nimport triton\n\nautotune_configs = [\n    triton.config(defines={\n        \"MB\": \"128\",\n        \"NB\": \"128\",\n        \"KB\": \"32\"\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '64',\n        'NB': '128',\n        'KB': '32'\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '128',\n        'NB': '64',\n        'KB': '32'\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '64',\n        'NB': '64',\n        'KB': '64'\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '32',\n        'NB': '128',\n        'KB': '64'\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '128',\n        'NB': '32',\n        'KB': '64'\n    }, num_warps=4),\n    triton.config(defines={\n        'MB': '64',\n        'NB': '32',\n        'KB': '64'\n    }, num_warps=2),\n    triton.config(defines={\n        'MB': '32',\n        'NB': '64',\n        'KB': '64'\n    }, num_warps=2)\n]"
+        "import torch\nimport triton\n\nautotune_configs = [\n    triton.config(defines={\"MB\": \"128\", \"NB\": \"128\", \"KB\": \"32\"}, num_warps=4),\n    triton.config(defines={'MB': '64', 'NB': '128', 'KB': '32'}, num_warps=4),\n    triton.config(defines={'MB': '128', 'NB': '64', 'KB': '32'}, num_warps=4),\n    triton.config(defines={'MB': '64', 'NB': '64', 'KB': '64'}, num_warps=4),\n    triton.config(defines={'MB': '32', 'NB': '128', 'KB': '64'}, num_warps=4),\n    triton.config(defines={'MB': '128', 'NB': '32', 'KB': '64'}, num_warps=4),\n    triton.config(defines={'MB': '64', 'NB': '32', 'KB': '64'}, num_warps=2),\n    triton.config(defines={'MB': '32', 'NB': '64', 'KB': '64'}, num_warps=2)\n]"
       ]
     },
     {
@@ -126,7 +126,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "## Benchmark\n\n### Installing The CUTLASS Bindings\n\nThe cuBLAS library (used by :code:`torch.matmul`) uses handwritten assembly-level optimizations that cannot be replicated using publicly available tools.\nFor this reason, we will instead compare the performance of our kernel against `CUTLASS <https://github.com/NVIDIA/cutlass/>`_ , a highly optimized CUDA library for matrix multiplication written by NVIDIA themselves._\nTo install CUTLASS, you need a recent version of cmake:\n\n .. code-block:: bash\n\n   cd /tmp/\n   git clone https://github.com/NVIDIA/cutlass.git\n   cd cutlass\n   mkdir build\n   cd build\n   wget https://github.com/Kitware/CMake/releases/download/v3.19.4/cmake-3.19.4-Linux-x86_64.tar.gz\n   tar xzvf *.tar.gz\n\nYou can then install CUTLASS as follows for V100\n\n .. code-block:: bash\n\n   ./cmake-3.19.4-Linux-x86_64/bin/cmake ../ -DCUTLASS_NVCC_ARCHS_ENABLED=70 -DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_f16_s884gemm_f16_*_align8\n   make -j8 install\n\nOr as follows for A100:\n\n .. code-block:: bash\n\n   ./cmake-3.19.4-Linux-x86_64/bin/cmake ../ -DCUTLASS_NVCC_ARCHS_ENABLED=80 -DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_f16_s16816gemm_*align8\n   make -j8 install\n\nWhere you can change CUTLASS_LIBRARY_KERNELS as you desire. Here, we are only interested in FP16 tensor core performance.\nTriton comes with some basic Python bindings for benchmarking CUTLASS. These will be compiled when the environment variables :code:`CUTLASS_INCLUDE_DIR` and :code:`CUTLASS_LIBRARY_DIR` are set during the installation process.\nTo re-install Triton with the updated CUTLASS bindings, run the following command:\n\n.. code-block:: bash\n\n   export CUTLASS_INCLUDE_DIR=/tmp/cutlass/build/install/include/\n   export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/\n   pip uninstall -y triton\n   pip install -e \"git+https://github.com/ptillet/triton.git#egg=triton&subdirectory=python\"\n\nWhich we can test as follows:\n\n"
+        "## Benchmark\n\n### Installing The CUTLASS Bindings\n\nThe cuBLAS library (used by :code:`torch.matmul`) uses handwritten assembly-level optimizations that cannot be replicated using publicly available tools.\nFor this reason, we will instead compare the performance of our kernel against `CUTLASS <https://github.com/NVIDIA/cutlass/>`_ , a highly optimized CUDA library for matrix multiplication written by NVIDIA themselves._\nTo install CUTLASS, you need a recent version of cmake:\n\n .. code-block:: bash\n\n   cd /path/to/cutlass/\n   git clone https://github.com/NVIDIA/cutlass.git\n   cd cutlass\n   mkdir build\n   cd build\n   wget https://github.com/Kitware/CMake/releases/download/v3.19.4/cmake-3.19.4-Linux-x86_64.tar.gz\n   tar xzvf *.tar.gz\n\nYou can then install CUTLASS as follows for V100\n\n .. code-block:: bash\n\n   ./cmake-3.19.4-Linux-x86_64/bin/cmake ../ -DCUTLASS_NVCC_ARCHS_ENABLED=70 -DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_f16_s884gemm_f16_*_align8\n   make -j8 install\n\nOr as follows for A100:\n\n .. code-block:: bash\n\n   ./cmake-3.19.4-Linux-x86_64/bin/cmake ../ -DCUTLASS_NVCC_ARCHS_ENABLED=80 -DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_f16_s16816gemm_*align8\n   make -j8 install\n\nWhere you can change CUTLASS_LIBRARY_KERNELS as you desire. Here, we are only interested in FP16 tensor core performance.\nTriton comes with some basic Python bindings for benchmarking CUTLASS. These will be compiled when the environment variables :code:`CUTLASS_INCLUDE_DIR` and :code:`CUTLASS_LIBRARY_DIR` are set during the installation process.\nTo re-install Triton with the updated CUTLASS bindings, run the following command:\n\n.. code-block:: bash\n\n   export CUTLASS_INCLUDE_DIR=/tmp/cutlass/build/install/include/\n   export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/a\n   pip uninstall -y triton\n   pip install -e \"git+https://github.com/ptillet/triton.git#egg=triton&subdirectory=python\"\n\nWhich we can test as follows:\n\n"
       ]
     },
     {

_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py

@@ -172,46 +172,14 @@ import torch
 import triton
 
 autotune_configs = [
-    triton.config(defines={
-        "MB": "128",
-        "NB": "128",
-        "KB": "32"
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '64',
-        'NB': '128',
-        'KB': '32'
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '128',
-        'NB': '64',
-        'KB': '32'
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '64',
-        'NB': '64',
-        'KB': '64'
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '32',
-        'NB': '128',
-        'KB': '64'
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '128',
-        'NB': '32',
-        'KB': '64'
-    }, num_warps=4),
-    triton.config(defines={
-        'MB': '64',
-        'NB': '32',
-        'KB': '64'
-    }, num_warps=2),
-    triton.config(defines={
-        'MB': '32',
-        'NB': '64',
-        'KB': '64'
-    }, num_warps=2)
+    triton.config(defines={"MB": "128", "NB": "128", "KB": "32"}, num_warps=4),
+    triton.config(defines={'MB': '64', 'NB': '128', 'KB': '32'}, num_warps=4),
+    triton.config(defines={'MB': '128', 'NB': '64', 'KB': '32'}, num_warps=4),
+    triton.config(defines={'MB': '64', 'NB': '64', 'KB': '64'}, num_warps=4),
+    triton.config(defines={'MB': '32', 'NB': '128', 'KB': '64'}, num_warps=4),
+    triton.config(defines={'MB': '128', 'NB': '32', 'KB': '64'}, num_warps=4),
+    triton.config(defines={'MB': '64', 'NB': '32', 'KB': '64'}, num_warps=2),
+    triton.config(defines={'MB': '32', 'NB': '64', 'KB': '64'}, num_warps=2)
 ]
 
 # %%
@@ -322,7 +290,7 @@ print(torch.allclose(c_0, c_1, rtol=1e-3, atol=1e-3))
 #
 #  .. code-block:: bash
 #
-#    cd /tmp/
+#    cd /path/to/cutlass/
 #    git clone https://github.com/NVIDIA/cutlass.git
 #    cd cutlass
 #    mkdir build
@@ -351,7 +319,7 @@ print(torch.allclose(c_0, c_1, rtol=1e-3, atol=1e-3))
 # .. code-block:: bash
 #
 #    export CUTLASS_INCLUDE_DIR=/tmp/cutlass/build/install/include/
-#    export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/
+#    export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/a
 #    pip uninstall -y triton
 #    pip install -e "git+https://github.com/ptillet/triton.git#egg=triton&subdirectory=python"
 #

_sources/getting-started/installation.rst.txt

@@ -2,6 +2,17 @@
 Installation
 ==============
 
+---------------------
+Binary Distributions
+---------------------
+
+You can install the latest nightly release of Triton from pip:
+
+.. code-block:: bash
+  
+      pip install -U --pre triton
+
+
 --------------
 From Source
 --------------
@@ -14,11 +25,12 @@ You can install the Python package from source by running the following commands
 
 .. code-block:: bash
 
-      sudo apt-get install llvm-10-dev
       git clone https://github.com/ptillet/triton.git;
       cd triton/python;
       pip install -e .
 
+This may take a while (10-20 minutes) as it will download and compile LLVM from source.
+
 You can then test your installation by running the unit tests:
 
 .. code-block:: bash
@@ -40,17 +52,10 @@ Those not interested in Python integration may want to use the internals of Trit
 
 .. code-block:: bash
 
-      sudo apt-get install llvm-10-dev
       git clone https://github.com/ptillet/triton.git;
       mkdir build;
       cd build;
       cmake ../;
       make -j8;
 
-A custom llvm-config binary can also be provided:
-
-.. code-block:: bash
-      
-      cmake ../ -DLLVM_CONFIG=/path/to/llvm-config
-
 Note that while direct usage of the C++ API is not officially supported, a usage tutorial can be found  `here <https://github.com/ptillet/triton/blob/master/tutorials/01-matmul.cc>`_

_sources/getting-started/tutorials/01-vector-add.rst.txt

@@ -268,7 +268,7 @@ We can now run the decorated function above. Pass `show_plots=True` to see the p
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 0 minutes  8.442 seconds)
+   **Total running time of the script:** ( 0 minutes  7.756 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_01-vector-add.py:

_sources/getting-started/tutorials/02-fused-softmax.rst.txt

@@ -314,7 +314,7 @@ In the above plot, we can see that:
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 0 minutes  20.299 seconds)
+   **Total running time of the script:** ( 0 minutes  19.933 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_02-fused-softmax.py:

_sources/getting-started/tutorials/03-matrix-multiplication.rst.txt

@@ -189,7 +189,7 @@ Auto-Tuning
 
 In order to use Triton's built-in auto-tuner in the above kernel, we need to define a list of :code:`triton.config` objects. that can be constructed as follows:
 
-.. GENERATED FROM PYTHON SOURCE LINES 170-217
+.. GENERATED FROM PYTHON SOURCE LINES 170-185
 
 .. code-block:: default
 
@@ -198,46 +198,14 @@ In order to use Triton's built-in auto-tuner in the above kernel, we need to def
     import triton
 
     autotune_configs = [
-        triton.config(defines={
-            "MB": "128",
-            "NB": "128",
-            "KB": "32"
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '64',
-            'NB': '128',
-            'KB': '32'
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '128',
-            'NB': '64',
-            'KB': '32'
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '64',
-            'NB': '64',
-            'KB': '64'
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '32',
-            'NB': '128',
-            'KB': '64'
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '128',
-            'NB': '32',
-            'KB': '64'
-        }, num_warps=4),
-        triton.config(defines={
-            'MB': '64',
-            'NB': '32',
-            'KB': '64'
-        }, num_warps=2),
-        triton.config(defines={
-            'MB': '32',
-            'NB': '64',
-            'KB': '64'
-        }, num_warps=2)
+        triton.config(defines={"MB": "128", "NB": "128", "KB": "32"}, num_warps=4),
+        triton.config(defines={'MB': '64', 'NB': '128', 'KB': '32'}, num_warps=4),
+        triton.config(defines={'MB': '128', 'NB': '64', 'KB': '32'}, num_warps=4),
+        triton.config(defines={'MB': '64', 'NB': '64', 'KB': '64'}, num_warps=4),
+        triton.config(defines={'MB': '32', 'NB': '128', 'KB': '64'}, num_warps=4),
+        triton.config(defines={'MB': '128', 'NB': '32', 'KB': '64'}, num_warps=4),
+        triton.config(defines={'MB': '64', 'NB': '32', 'KB': '64'}, num_warps=2),
+        triton.config(defines={'MB': '32', 'NB': '64', 'KB': '64'}, num_warps=2)
     ]
 
 
@@ -247,12 +215,12 @@ In order to use Triton's built-in auto-tuner in the above kernel, we need to def
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 218-220
+.. GENERATED FROM PYTHON SOURCE LINES 186-188
 
 we also need to define a list of :code:`string` (i.e., "autotuning key") that specifies the set of argument names whose change in value will trigger the auto-tuner to kick in.
 Here, we want to re-tune our kernel only when the shape of input matrices changes.
 
-.. GENERATED FROM PYTHON SOURCE LINES 220-223
+.. GENERATED FROM PYTHON SOURCE LINES 188-191
 
 .. code-block:: default
 
@@ -266,11 +234,11 @@ Here, we want to re-tune our kernel only when the shape of input matrices change
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 224-225
+.. GENERATED FROM PYTHON SOURCE LINES 192-193
 
 We can now create an auto-tuned kernel by passing the `autotune_configs` and `autotune_key` lists to the constructor of the :code:`triton.kernel` class.
 
-.. GENERATED FROM PYTHON SOURCE LINES 225-270
+.. GENERATED FROM PYTHON SOURCE LINES 193-238
 
 .. code-block:: default
 
@@ -326,7 +294,7 @@ We can now create an auto-tuned kernel by passing the `autotune_configs` and `au
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 271-276
+.. GENERATED FROM PYTHON SOURCE LINES 239-244
 
 Autograd Function
 ~~~~~~~~~~~~~~~~~~
@@ -334,7 +302,7 @@ Autograd Function
 Now we are ready to expose our auto-tuned kernel as a `torch.autograd.Function`.
 To do so, we just need to define a `forward` function that takes a two tensors as input and returns a tensor as output.
 
-.. GENERATED FROM PYTHON SOURCE LINES 276-297
+.. GENERATED FROM PYTHON SOURCE LINES 244-265
 
 .. code-block:: default
 
@@ -366,7 +334,7 @@ To do so, we just need to define a `forward` function that takes a two tensors a
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 298-303
+.. GENERATED FROM PYTHON SOURCE LINES 266-271
 
 Unit Test
 -----------
@@ -374,7 +342,7 @@ Unit Test
 We can test our custom matrix multiplication operation against cuBLAS (i.e., :code:`torch.matmul`).
 Note that we need to modify the :code`atol` and :code:`rtol` parameters of `torch.allclose` to account for the fact that we are comparing FP16 tensors.
 
-.. GENERATED FROM PYTHON SOURCE LINES 303-312
+.. GENERATED FROM PYTHON SOURCE LINES 271-280
 
 .. code-block:: default
 
@@ -397,28 +365,28 @@ Note that we need to modify the :code`atol` and :code:`rtol` parameters of `torc
 
  .. code-block:: none
 
-    tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-            [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-            [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+    tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+            [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+            [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
             ...,
-            [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-            [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-            [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+            [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+            [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+            [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
            device='cuda:0', dtype=torch.float16)
-    tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-            [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-            [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+    tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+            [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+            [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
             ...,
-            [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-            [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-            [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+            [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+            [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+            [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
            device='cuda:0', dtype=torch.float16)
     True
 
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 313-359
+.. GENERATED FROM PYTHON SOURCE LINES 281-327
 
 Benchmark
 --------------
@@ -432,7 +400,7 @@ To install CUTLASS, you need a recent version of cmake:
 
  .. code-block:: bash
 
-   cd /tmp/
+   cd /path/to/cutlass/
    git clone https://github.com/NVIDIA/cutlass.git
    cd cutlass
    mkdir build
@@ -461,13 +429,13 @@ To re-install Triton with the updated CUTLASS bindings, run the following comman
 .. code-block:: bash
 
    export CUTLASS_INCLUDE_DIR=/tmp/cutlass/build/install/include/
-   export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/
+   export CUTLASS_LIBRARY_DIR=/tmp/cutlass/build/install/lib/a
    pip uninstall -y triton
    pip install -e "git+https://github.com/ptillet/triton.git#egg=triton&subdirectory=python"
 
 Which we can test as follows:
 
-.. GENERATED FROM PYTHON SOURCE LINES 359-365
+.. GENERATED FROM PYTHON SOURCE LINES 327-333
 
 .. code-block:: default
 
@@ -487,20 +455,20 @@ Which we can test as follows:
 
  .. code-block:: none
 
-    tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-            [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-            [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+    tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+            [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+            [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
             ...,
-            [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-            [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-            [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+            [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+            [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+            [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
            device='cuda:0', dtype=torch.float16)
     True
 
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 366-371
+.. GENERATED FROM PYTHON SOURCE LINES 334-339
 
 Note that this wrapper for CUTLASS was written for benchmarking purposes and is probably not production-ready.
 
@@ -508,7 +476,7 @@ Square Matrix Performance
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 We can now compare the performance of our kernel against CUTLASS. Here we focus on square matrices, but feel free to arrange the script as you wish to compare any other matrix shape.#
 
-.. GENERATED FROM PYTHON SOURCE LINES 371-400
+.. GENERATED FROM PYTHON SOURCE LINES 339-368
 
 .. code-block:: default
 
@@ -552,14 +520,14 @@ We can now compare the performance of our kernel against CUTLASS. Here we focus
 
 
 
-.. GENERATED FROM PYTHON SOURCE LINES 401-401
+.. GENERATED FROM PYTHON SOURCE LINES 369-369
 
 As we can see, the performance of our kernel is pretty good. It is in fact faster than CUTLASS, and therefore probably comparable to the absolute best CUDA code an expert could write.
 
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 1 minutes  10.094 seconds)
+   **Total running time of the script:** ( 1 minutes  6.502 seconds)
 
 
 .. _sphx_glr_download_getting-started_tutorials_03-matrix-multiplication.py:

_sources/getting-started/tutorials/sg_execution_times.rst.txt

@@ -5,12 +5,12 @@
 
 Computation times
 =================
-**01:10.094** total execution time for **getting-started_tutorials** files:
+**01:34.190** total execution time for **getting-started_tutorials** files:
 
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 01:10.094 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_03-matrix-multiplication.py` (``03-matrix-multiplication.py``) | 01:06.502 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``)                       | 00:00.000 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``)                 | 00:19.933 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+
-| :ref:`sphx_glr_getting-started_tutorials_02-fused-softmax.py` (``02-fused-softmax.py``)                 | 00:00.000 | 0.0 MB |
+| :ref:`sphx_glr_getting-started_tutorials_01-vector-add.py` (``01-vector-add.py``)                       | 00:07.756 | 0.0 MB |
 +---------------------------------------------------------------------------------------------------------+-----------+--------+

_sources/index.rst.txt

@@ -18,19 +18,21 @@ Getting Started
    getting-started/tutorials/index
 
 Programming Guide
---------------
+------------------
 
 Check out the following documents to learn more about Triton and how it compares against other DSLs for DNNs:
 
-- Chapter 1: :doc:`Introduction <programming-guide/introduction>`
-- Chapter 2: :doc:`Related Work <programming-guide/related-work>`
-- Chapter 3: :doc:`The Triton-C Kernel Language <programming-guide/triton-c>`
+- Chapter 1: :doc:`Introduction <programming-guide/chapter-1/introduction>`
+- Chapter 2: :doc:`Related Work <programming-guide/chapter-2/related-work>`
+- Chapter 3: :doc:`The Triton-C Language <programming-guide/chapter-3/triton-c>`
+- Chapter 4: :doc:`The Triton-IR Intermediate Representation <programming-guide/chapter-4/triton-ir>`
 
 .. toctree::
    :maxdepth: 1
    :caption: Programming Guide
    :hidden:
 
-   programming-guide/introduction
-   programming-guide/related-work
-   programming-guide/triton-c
+   programming-guide/chapter-1/introduction
+   programming-guide/chapter-2/related-work
+   programming-guide/chapter-3/triton-c
+   programming-guide/chapter-4/triton-ir

_sources/programming-guide/chapter-1/introduction.rst.txt

@@ -6,13 +6,13 @@ Introduction
 Motivations
 --------------
 
-Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing [1]_ to computer vision [2]_ to computational neuroscience [3]_. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) *layers*. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.
+Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing [SUTSKEVER2014]_ to computer vision [REDMON2016]_ to computational neuroscience [LEE2017]_. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) *layers*. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.
 
 As a consequence, Graphics Processing Units (GPUs) have become a cheap and accessible resource for exploring and/or deploying novel research ideas in the field. This trend has been accelerated by the release of several frameworks for General-Purpose GPU (GPGPU) computing, such as CUDA and OpenCL, which have made the development of high-performance programs easier. Yet, GPUs remain incredibly challenging to optimize for locality and parallelism, especially for computations that cannot be efficiently implemented using a combination of pre-existing optimized primitives. To make matters worse, GPU architectures are also rapidly evolving and specializing, as evidenced by the addition of tensor cores to NVIDIA (and more recently AMD) micro-architectures.
 
-This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial academic and industrial interest for Domain-Specific Languages (DSLs) and compilers. Regrettably, these systems -- whether they be based on  polyhedral machinery (*e.g.*, Tiramisu [4]_, Tensor Comprehensions [5]_) or scheduling languages (*e.g.*, Halide [6]_, TVM [7]_) -- remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like `cuBLAS <https://docs.nvidia.com/cuda/cublas/index.html>`_, `cuDNN <https://docs.nvidia.com/deeplearning/cudnn/api/index.html>`_ or `TensorRT <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html>`_.
+This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial academic and industrial interest for Domain-Specific Languages (DSLs) and compilers. Regrettably, these systems -- whether they be based on  polyhedral machinery (*e.g.*, Tiramisu [BAGHDADI2021]_, Tensor Comprehensions [VASILACHE2018]_) or scheduling languages (*e.g.*, Halide [JRK2013]_, TVM [CHEN2018]_) -- remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like `cuBLAS <https://docs.nvidia.com/cuda/cublas/index.html>`_, `cuDNN <https://docs.nvidia.com/deeplearning/cudnn/api/index.html>`_ or `TensorRT <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html>`_.
 
-The main premise of this project is the following: programming paradigms based on blocked algorithms [8]_ can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional "Single Program, Multiple Data" (SPMD [9]_) execution models for GPUs, and propose a variant in which programs -- rather than threads -- are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:
+The main premise of this project is the following: programming paradigms based on blocked algorithms [LAM1991]_ can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional "Single Program, Multiple Data" (SPMD [AUGUIN1983]_) execution models for GPUs, and propose a variant in which programs -- rather than threads -- are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:
 
 .. table::
     :widths: 50 50
@@ -58,12 +58,12 @@ The main challenge posed by our proposed paradigm is that of work scheduling, i.
 References
 --------------
 
-.. [1] Sutskever et al., "Sequence to Sequence Learning with Neural Networks", NIPS 2014
-.. [2] Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2016
-.. [3] Lee et al., "Superhuman Accuracy on the SNEMI3D Connectomics Challenge", ArXiV 2017
-.. [4] Baghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021
-.. [5] Vasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018
-.. [6] Ragan-Kelley et al., "Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines", PLDI 2013
-.. [7] Chen et al., "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning", OSDI 2018
-.. [8] Lam et al., "The Cache Performance and Optimizations of Blocked Algorithms", ASPLOS 1991
-.. [9] Auguin et al., "Opsila: an advanced SIMD for numerical analysis and signal processing", EUROMICRO 1983
+.. [SUTSKEVER2014] I. Sutskever et al., "Sequence to Sequence Learning with Neural Networks", NIPS 2014
+.. [REDMON2016] J. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2016
+.. [LEE2017] K. Lee et al., "Superhuman Accuracy on the SNEMI3D Connectomics Challenge", ArXiV 2017
+.. [BAGHDADI2021] R. Baghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021
+.. [VASILACHE2018] N. Vasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018
+.. [JRK2013] J. Ragan-Kelley et al., "Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines", PLDI 2013
+.. [CHEN2018] T. Chen et al., "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning", OSDI 2018
+.. [LAM1991] M. Lam et al., "The Cache Performance and Optimizations of Blocked Algorithms", ASPLOS 1991
+.. [AUGUIN1983] M. Auguin et al., "Opsila: an advanced SIMD for numerical analysis and signal processing", EUROMICRO 1983

_sources/programming-guide/chapter-2/related-work.rst.txt

@@ -8,7 +8,7 @@ At first sight, Triton may seem like just yet another DSL for DNNs. The purpose
 Polyhedral Compilation
 -----------------------
 
-Traditional compilers typically rely on intermediate representations, such as LLVM-IR [1]_, that encode control flow information using (un)conditional branches. This relatively low-level format makes it difficult to statically analyze the runtime behavior (e.g., cache misses) of input programs, and to  automatically optimize loops accordingly through the use of tiling [2]_, fusion [3]_ and interchange [4]_. To solve this issue, polyhedral compilers [5]_ rely on program representations that have statically predictable control flow, thereby enabling aggressive compile-time program transformations for data locality and parallelism. Though this strategy has been adopted by many languages and compilers for DNNs such as Tiramisu [6]_, Tensor Comprehensions [7]_, Diesel [8]_ and the Affine dialect in MLIR [9]_, it also comes with a number of limitations that will be described later.
+Traditional compilers typically rely on intermediate representations, such as LLVM-IR [LATTNER2004]_, that encode control flow information using (un)conditional branches. This relatively low-level format makes it difficult to statically analyze the runtime behavior (e.g., cache misses) of input programs, and to  automatically optimize loops accordingly through the use of tiling [WOLFE1989]_, fusion [DARTE1999]_ and interchange [ALLEN1984]_. To solve this issue, polyhedral compilers [ANCOURT1991]_ rely on program representations that have statically predictable control flow, thereby enabling aggressive compile-time program transformations for data locality and parallelism. Though this strategy has been adopted by many languages and compilers for DNNs such as Tiramisu [BAGHDADI2021]_, Tensor Comprehensions [VASILACHE2018]_, Diesel [ELANGO2018]_ and the Affine dialect in MLIR [LATTNER2019]_, it also comes with a number of limitations that will be described later in this section.
 
 +++++++++++++++++++++++
 Program Representation
@@ -111,9 +111,9 @@ Advantages
 
 Programs amenable to polyhedral compilation can be aggressively transformed and optimized. Most of these transformations actually boil down to the production of  schedules and iteration domains that enable loop transformations promoting parallelism and spatial/temporal data locality (e.g., fusion, interchange, tiling, parallelization).
 
-Polyhedral compilers can also automatically go through complex verification processes to ensure that the semantics of their input program is preserved throughout this optimization phase. Note that polyhedral optimizers are not incompatible with more standard optimization techniques. In fact, it is not uncommon for these systems to be implemented as a set of LLVM passes that can be run ahead of more traditional compilation techniques [10]_.
+Polyhedral compilers can also automatically go through complex verification processes to ensure that the semantics of their input program is preserved throughout this optimization phase. Note that polyhedral optimizers are not incompatible with more standard optimization techniques. In fact, it is not uncommon for these systems to be implemented as a set of LLVM passes that can be run ahead of more traditional compilation techniques [GROSSER2012]_.
 
-All in all, polyhedral machinery is extremely powerful, when applicable. It has been shown to support most common loop transformations, and has indeed achieved performance comparable to state-of-the-art GPU libraries for dense matrix multiplication [8]_. Additionally, it is also fully automatic and doesn't require any hint from programmers apart from source-code in a C-like format. 
+All in all, polyhedral machinery is extremely powerful, when applicable. It has been shown to support most common loop transformations, and has indeed achieved performance comparable to state-of-the-art GPU libraries for dense matrix multiplication [ELANGO2018]_. Additionally, it is also fully automatic and doesn't require any hint from programmers apart from source-code in a C-like format. 
 
 ++++++++++++
 Limitations
@@ -121,9 +121,9 @@ Limitations
 
 Unfortunately, polyhedral compilers suffer from two major limitations that have prevented its adoption as a universal method for code generation in neural networks.
 
-First, the set of possible program transformations $\Omega = \{ \Theta_S ~|~ S \in \text{program} \}$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures [11]_.
+First, the set of possible program transformations $\Omega = \{ \Theta_S ~|~ S \in \text{program} \}$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures [SATO2019]_.
 
-Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common [12]_ but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse -- or even structured-sparse -- neural networks, whose importance has been rapidly rising over the past few years.
+Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common [GIRBAL2006]_ but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse -- or even structured-sparse -- neural networks, whose importance has been rapidly rising over the past few years.
 
 On the other hand, blocked program representations advocated by this dissertation are less restricted in scope and can achieve close to peak performance using standard dataflow analysis.
 
@@ -154,7 +154,7 @@ Separation of concerns \cite{dijkstra82} is a well-known design principle in com
       .parallel(y).vectorize(xii).unroll(xi).unroll(yii);
 
 
-The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms [13]_.
+The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms [MULLAPUDI2016]_.
 
 +++++++++++
 Advantages
@@ -194,16 +194,16 @@ On the other hand, the block-based program representation that we advocate for t
 References
 --------------
 
-.. [1] Lattner et al., "LLVM: a compilation framework for lifelong program analysis transformation"
-.. [2] Wolfe, "More Iteration Space Tiling", SC 1989
-.. [3] Darte, "On the Complexity of Loop Fusion", PACT 1999
-.. [4] Allen et al., "Automatic Loop Interchange", SIGPLAN Notices 1984
-.. [5] Ancourt et al., "Scanning Polyhedra with DO Loops", PPoPP 1991
-.. [6] Baghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021
-.. [7] Vasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018
-.. [8] Elango et al. "Diesel: DSL for Linear Algebra and Neural Net Computations on GPUs", MAPL 2018
-.. [9] Lattner et al., "MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law", Arxiv 2019
-.. [10] Grosser et al., "Polly - Performing Polyhedral Optimizations on a Low-Level Intermediate Representation", Parallel Processing Letters 2012
-.. [11] Sato et al., "An Autotuning Framework for Scalable Execution of Tiled Code via Iterative Polyhedral Compilation", TACO 2019
-.. [12] Girbal et al., "Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies", International Journal of Parallel Programming 2006
-.. [13] Mullapudi et al., "Automatically scheduling halide image processing pipelines", TOG 2016
+.. [LATTNER2004] C. Lattner et al., "LLVM: a compilation framework for lifelong program analysis transformation", CGO 2004
+.. [WOLFE1989] M. Wolfe, "More Iteration Space Tiling", SC 1989
+.. [DARTE1999] A. Darte, "On the Complexity of Loop Fusion", PACT 1999
+.. [ALLEN1984] J. Allen et al., "Automatic Loop Interchange", SIGPLAN Notices 1984
+.. [ANCOURT1991] C. Ancourt et al., "Scanning Polyhedra with DO Loops", PPoPP 1991
+.. [BAGHDADI2021] R. Baghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021
+.. [VASILACHE2018] N. Vasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018
+.. [ELANGO2018] V. Elango et al. "Diesel: DSL for Linear Algebra and Neural Net Computations on GPUs", MAPL 2018
+.. [LATTNER2019] C. Lattner et al., "MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law", Arxiv 2019
+.. [GROSSER2012] T. Grosser et al., "Polly - Performing Polyhedral Optimizations on a Low-Level Intermediate Representation", Parallel Processing Letters 2012
+.. [SATO2019] Y. Sato et al., "An Autotuning Framework for Scalable Execution of Tiled Code via Iterative Polyhedral Compilation", TACO 2019
+.. [GIRBAL2006] S. Girbal et al., "Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies", International Journal of Parallel Programming 2006
+.. [MULLAPUDI2016] R. Mullapudi et al., "Automatically scheduling halide image processing pipelines", TOG 2016

_sources/programming-guide/chapter-3/triton-c.rst.txt

@@ -25,6 +25,7 @@ Extensions
 **Masked pointer dereferencement**: Block-level operations in Triton-C are "atomic", in the sense that they execute either completely or not at all. Basic element-wise control-flow for block-level operations can nonetheless be achieved using ternary operators and the *masked pointer dereferencement* operator exemplified below:
 
 .. code-block:: C
+  :force:
 
   // create mask
   bool mask[16, 16] = ...;

_sources/programming-guide/chapter-4/triton-ir.rst.txt

@@ -0,0 +1,82 @@
+==========================================
+The Triton-IR Intermediate Representation
+==========================================
+
+Triton-IR is an LLVM-based Intermediate Representation (IR) whose purpose is to provide an environment suitable for block-level program analysis, transformation and optimization.
+In our implementation, Triton-IR programs are constructed directly from Triton-C after parsing, but they could also be formed directly by higher-level DSLs in the future.
+Triton-IR and LLVM-IR programs share the same high-level structure, but the former also includes a number of extensions necessary for block-level data-flow analysis.
+These extensions are crucial for carrying out the optimizations outlined in the next chapter of this document.
+
+---------------------------------
+Structure of a Triton-IR Program
+---------------------------------
+
+++++++++
+Modules
+++++++++
+
+At the highest level, Triton-IR programs consist of one or multiple basic units of compilation known as *modules*. These modules are compiled independently from one another, and eventually aggregated by a linker whose role is to resolve forward declarations and adequately merge global definitions. Each module itself is composed of functions, global variables, constants and other miscellaneous symbols such as metadata and attributes.
+
+++++++++++
+Functions
+++++++++++
+
+Triton-IR function definitions consist of a return type, a name and a potentially empty arguments list. Additional visibility, alignment and linkage specifiers can be added if desired. Function attributes (such as inlining hints) and parameter attributes (such as "readonly", aliasing hints) can also be specified, allowing compiler backends to perform more aggressive optimizations by, for instance, making better use of non-coherent caches found on NVIDIA GPUs. This header is followed by a body composed of a list of basic blocks whose interdependencies form the Control Flow Graph (CFG) of the function.
+
++++++++++++++
+Basic Blocks
++++++++++++++
+
+Basic blocks are straight-line code sequences that may only contain so-called *terminator* instructions (i.e., branching, return) at their end. To simplify program analysis, Triton-IR uses the Static Single Assignment (SSA) form, meaning that each variable in each basic block must be (1) assigned to only once and (2) defined before being used. In so doing, each basic block implicitly defines a Data-Flow Graph (DFG). In our case, the SSA form is created directly from Triton-C's Abstract Syntax Trees (ASTs) using an algorithm from the literature [BRAUN13]_.
+
+---------------------------------
+Block-Level Dataflow Analysis
+---------------------------------
+
++++++++
+Types
++++++++
+
+Multi-dimensional blocks are at the center of data-flow analysis in Triton-JIT. They can be declared using syntax similar to vector declarations in LLVM-IR. For example, :code:`i32<8, 8>` is the type corresponding to :math:`8 \times 8` blocks of 32-bit integers. Note that there is no preprocessor in Triton-IR, hence parametric shape  values must be resolved before programs are generated. In our case, this is done by Triton-JIT's auto-tuner.
+
++++++++++++++
+Instructions
++++++++++++++
+
+Triton-IR introduces a set of *reblocking* instructions whose purpose is to support broadcasting semantics as described in the previous chapter.  The :code:`reshape` instruction creates a block of the specified shape using the raw data from its input argument. This is particularly useful to re-interpret variables as higher-dimensional arrays by padding their input shapes with ones in preparation for broadcasting. The :code:`broadcast` instruction creates a block of the specified shapes by replicating its input argument as many times as necessary along dimensions of size 1 -- as shown below for the :code:`broadcast<3,3>` instruction.
+
+|pic1| and |pic2|
+
+.. |pic1| image:: broadcast-1.png
+   :width: 40%
+
+.. |pic2| image:: broadcast-2.png
+   :width: 40%
+
+Usual scalar instructions (:code:`cmp`, :code:`getelementptr`, :code:`add`, :code:`load`...) were preserved and extended to signify element-wise operations when applicable. Finally, Triton-IR also exposes specialized arithmetic instructions for reductions (:code:`reduce`) and matrix multiplications (:code:`dot`).
+
+----------------------------------
+Block-Level Control Flow Analysis
+----------------------------------
+
+In Triton-IR, operations on block variables are atomic: they execute either in full or not at all. As a result, traditional control flow structures (e.g., conditional, loops) are not applicable to individual block elements. This is problematic, since a program may need to e.g., partially guard blocked loads against memory access violations.
+
+This could be potentially solved through the use of the Predicated SSA (PSSA) [CARTER99]_ [STOUTCHININ01]_ form for Triton-IR. However, this would create a lot of unnecessary complexity for GPUs, where the benefits of PSSA are close to none as divergent program paths  within warps are  serialized anyway. Therefore, recent versions of Triton handle intra-block control flow in a much simpler way, using conditional instructions such as  :code:`select`, :code:`masked_load` and :code:`masked_store`:
+
+.. code-block:: C
+
+  // For all indices [idx], return cond[idx] ? true_value[idx] : false_value[idx];
+  select       TYPE<TS1, ..., TSN> cond, true_value, false_value;
+  // For all indices [idx], return cond[idx] ? *true_addr[idx] : false_value[idx];
+  masked_load  TYPE<TS1, ..., TSN> cond, true_addr, false_value;
+  // For all indices [idx], execute *true_addr[idx] = true_value[idx] if cond[idx]
+  masked_store TYPE<TS1, ..., TSN> cond, true_addr, true_value;
+
+
+------------
+References
+------------
+
+.. [BRAUN13] M. Braun et al., "Simple and Efficient Construction of Static Single Assignment Form", CC 2013
+.. [CARTER99] L. Carter et al., "Predicated Static Single Assignment", PACT 1999
+.. [STOUTCHININ01] A. Stoutchinin et al., "Efficient Static Single Assignment Form for Predication", MICRO 2001

_sources/programming-guide/index.rst.txt

@@ -1,10 +0,0 @@
-Programming Guide
-==================
-
-.. toctree::
-   :maxdepth: 1
-   :caption: Getting Started
- 
-   introduction
-   related-work
-   triton-c

_static/css/custom.css

@@ -1 +0,0 @@
-table {background-color: white;}

_static/underscore-1.3.1.js

@@ -1,999 +0,0 @@
-//     Underscore.js 1.3.1
-//     (c) 2009-2012 Jeremy Ashkenas, DocumentCloud Inc.
-//     Underscore is freely distributable under the MIT license.
-//     Portions of Underscore are inspired or borrowed from Prototype,
-//     Oliver Steele's Functional, and John Resig's Micro-Templating.
-//     For all details and documentation:
-//     http://documentcloud.github.com/underscore
-
-(function() {
-
-  // Baseline setup
-  // --------------
-
-  // Establish the root object, `window` in the browser, or `global` on the server.
-  var root = this;
-
-  // Save the previous value of the `_` variable.
-  var previousUnderscore = root._;
-
-  // Establish the object that gets returned to break out of a loop iteration.
-  var breaker = {};
-
-  // Save bytes in the minified (but not gzipped) version:
-  var ArrayProto = Array.prototype, ObjProto = Object.prototype, FuncProto = Function.prototype;
-
-  // Create quick reference variables for speed access to core prototypes.
-  var slice            = ArrayProto.slice,
-      unshift          = ArrayProto.unshift,
-      toString         = ObjProto.toString,
-      hasOwnProperty   = ObjProto.hasOwnProperty;
-
-  // All **ECMAScript 5** native function implementations that we hope to use
-  // are declared here.
-  var
-    nativeForEach      = ArrayProto.forEach,
-    nativeMap          = ArrayProto.map,
-    nativeReduce       = ArrayProto.reduce,
-    nativeReduceRight  = ArrayProto.reduceRight,
-    nativeFilter       = ArrayProto.filter,
-    nativeEvery        = ArrayProto.every,
-    nativeSome         = ArrayProto.some,
-    nativeIndexOf      = ArrayProto.indexOf,
-    nativeLastIndexOf  = ArrayProto.lastIndexOf,
-    nativeIsArray      = Array.isArray,
-    nativeKeys         = Object.keys,
-    nativeBind         = FuncProto.bind;
-
-  // Create a safe reference to the Underscore object for use below.
-  var _ = function(obj) { return new wrapper(obj); };
-
-  // Export the Underscore object for **Node.js**, with
-  // backwards-compatibility for the old `require()` API. If we're in
-  // the browser, add `_` as a global object via a string identifier,
-  // for Closure Compiler "advanced" mode.
-  if (typeof exports !== 'undefined') {
-    if (typeof module !== 'undefined' && module.exports) {
-      exports = module.exports = _;
-    }
-    exports._ = _;
-  } else {
-    root['_'] = _;
-  }
-
-  // Current version.
-  _.VERSION = '1.3.1';
-
-  // Collection Functions
-  // --------------------
-
-  // The cornerstone, an `each` implementation, aka `forEach`.
-  // Handles objects with the built-in `forEach`, arrays, and raw objects.
-  // Delegates to **ECMAScript 5**'s native `forEach` if available.
-  var each = _.each = _.forEach = function(obj, iterator, context) {
-    if (obj == null) return;
-    if (nativeForEach && obj.forEach === nativeForEach) {
-      obj.forEach(iterator, context);
-    } else if (obj.length === +obj.length) {
-      for (var i = 0, l = obj.length; i < l; i++) {
-        if (i in obj && iterator.call(context, obj[i], i, obj) === breaker) return;
-      }
-    } else {
-      for (var key in obj) {
-        if (_.has(obj, key)) {
-          if (iterator.call(context, obj[key], key, obj) === breaker) return;
-        }
-      }
-    }
-  };
-
-  // Return the results of applying the iterator to each element.
-  // Delegates to **ECMAScript 5**'s native `map` if available.
-  _.map = _.collect = function(obj, iterator, context) {
-    var results = [];
-    if (obj == null) return results;
-    if (nativeMap && obj.map === nativeMap) return obj.map(iterator, context);
-    each(obj, function(value, index, list) {
-      results[results.length] = iterator.call(context, value, index, list);
-    });
-    if (obj.length === +obj.length) results.length = obj.length;
-    return results;
-  };
-
-  // **Reduce** builds up a single result from a list of values, aka `inject`,
-  // or `foldl`. Delegates to **ECMAScript 5**'s native `reduce` if available.
-  _.reduce = _.foldl = _.inject = function(obj, iterator, memo, context) {
-    var initial = arguments.length > 2;
-    if (obj == null) obj = [];
-    if (nativeReduce && obj.reduce === nativeReduce) {
-      if (context) iterator = _.bind(iterator, context);
-      return initial ? obj.reduce(iterator, memo) : obj.reduce(iterator);
-    }
-    each(obj, function(value, index, list) {
-      if (!initial) {
-        memo = value;
-        initial = true;
-      } else {
-        memo = iterator.call(context, memo, value, index, list);
-      }
-    });
-    if (!initial) throw new TypeError('Reduce of empty array with no initial value');
-    return memo;
-  };
-
-  // The right-associative version of reduce, also known as `foldr`.
-  // Delegates to **ECMAScript 5**'s native `reduceRight` if available.
-  _.reduceRight = _.foldr = function(obj, iterator, memo, context) {
-    var initial = arguments.length > 2;
-    if (obj == null) obj = [];
-    if (nativeReduceRight && obj.reduceRight === nativeReduceRight) {
-      if (context) iterator = _.bind(iterator, context);
-      return initial ? obj.reduceRight(iterator, memo) : obj.reduceRight(iterator);
-    }
-    var reversed = _.toArray(obj).reverse();
-    if (context && !initial) iterator = _.bind(iterator, context);
-    return initial ? _.reduce(reversed, iterator, memo, context) : _.reduce(reversed, iterator);
-  };
-
-  // Return the first value which passes a truth test. Aliased as `detect`.
-  _.find = _.detect = function(obj, iterator, context) {
-    var result;
-    any(obj, function(value, index, list) {
-      if (iterator.call(context, value, index, list)) {
-        result = value;
-        return true;
-      }
-    });
-    return result;
-  };
-
-  // Return all the elements that pass a truth test.
-  // Delegates to **ECMAScript 5**'s native `filter` if available.
-  // Aliased as `select`.
-  _.filter = _.select = function(obj, iterator, context) {
-    var results = [];
-    if (obj == null) return results;
-    if (nativeFilter && obj.filter === nativeFilter) return obj.filter(iterator, context);
-    each(obj, function(value, index, list) {
-      if (iterator.call(context, value, index, list)) results[results.length] = value;
-    });
-    return results;
-  };
-
-  // Return all the elements for which a truth test fails.
-  _.reject = function(obj, iterator, context) {
-    var results = [];
-    if (obj == null) return results;
-    each(obj, function(value, index, list) {
-      if (!iterator.call(context, value, index, list)) results[results.length] = value;
-    });
-    return results;
-  };
-
-  // Determine whether all of the elements match a truth test.
-  // Delegates to **ECMAScript 5**'s native `every` if available.
-  // Aliased as `all`.
-  _.every = _.all = function(obj, iterator, context) {
-    var result = true;
-    if (obj == null) return result;
-    if (nativeEvery && obj.every === nativeEvery) return obj.every(iterator, context);
-    each(obj, function(value, index, list) {
-      if (!(result = result && iterator.call(context, value, index, list))) return breaker;
-    });
-    return result;
-  };
-
-  // Determine if at least one element in the object matches a truth test.
-  // Delegates to **ECMAScript 5**'s native `some` if available.
-  // Aliased as `any`.
-  var any = _.some = _.any = function(obj, iterator, context) {
-    iterator || (iterator = _.identity);
-    var result = false;
-    if (obj == null) return result;
-    if (nativeSome && obj.some === nativeSome) return obj.some(iterator, context);
-    each(obj, function(value, index, list) {
-      if (result || (result = iterator.call(context, value, index, list))) return breaker;
-    });
-    return !!result;
-  };
-
-  // Determine if a given value is included in the array or object using `===`.
-  // Aliased as `contains`.
-  _.include = _.contains = function(obj, target) {
-    var found = false;
-    if (obj == null) return found;
-    if (nativeIndexOf && obj.indexOf === nativeIndexOf) return obj.indexOf(target) != -1;
-    found = any(obj, function(value) {
-      return value === target;
-    });
-    return found;
-  };
-
-  // Invoke a method (with arguments) on every item in a collection.
-  _.invoke = function(obj, method) {
-    var args = slice.call(arguments, 2);
-    return _.map(obj, function(value) {
-      return (_.isFunction(method) ? method || value : value[method]).apply(value, args);
-    });
-  };
-
-  // Convenience version of a common use case of `map`: fetching a property.
-  _.pluck = function(obj, key) {
-    return _.map(obj, function(value){ return value[key]; });
-  };
-
-  // Return the maximum element or (element-based computation).
-  _.max = function(obj, iterator, context) {
-    if (!iterator && _.isArray(obj)) return Math.max.apply(Math, obj);
-    if (!iterator && _.isEmpty(obj)) return -Infinity;
-    var result = {computed : -Infinity};
-    each(obj, function(value, index, list) {
-      var computed = iterator ? iterator.call(context, value, index, list) : value;
-      computed >= result.computed && (result = {value : value, computed : computed});
-    });
-    return result.value;
-  };
-
-  // Return the minimum element (or element-based computation).
-  _.min = function(obj, iterator, context) {
-    if (!iterator && _.isArray(obj)) return Math.min.apply(Math, obj);
-    if (!iterator && _.isEmpty(obj)) return Infinity;
-    var result = {computed : Infinity};
-    each(obj, function(value, index, list) {
-      var computed = iterator ? iterator.call(context, value, index, list) : value;
-      computed < result.computed && (result = {value : value, computed : computed});
-    });
-    return result.value;
-  };
-
-  // Shuffle an array.
-  _.shuffle = function(obj) {
-    var shuffled = [], rand;
-    each(obj, function(value, index, list) {
-      if (index == 0) {
-        shuffled[0] = value;
-      } else {
-        rand = Math.floor(Math.random() * (index + 1));
-        shuffled[index] = shuffled[rand];
-        shuffled[rand] = value;
-      }
-    });
-    return shuffled;
-  };
-
-  // Sort the object's values by a criterion produced by an iterator.
-  _.sortBy = function(obj, iterator, context) {
-    return _.pluck(_.map(obj, function(value, index, list) {
-      return {
-        value : value,
-        criteria : iterator.call(context, value, index, list)
-      };
-    }).sort(function(left, right) {
-      var a = left.criteria, b = right.criteria;
-      return a < b ? -1 : a > b ? 1 : 0;
-    }), 'value');
-  };
-
-  // Groups the object's values by a criterion. Pass either a string attribute
-  // to group by, or a function that returns the criterion.
-  _.groupBy = function(obj, val) {
-    var result = {};
-    var iterator = _.isFunction(val) ? val : function(obj) { return obj[val]; };
-    each(obj, function(value, index) {
-      var key = iterator(value, index);
-      (result[key] || (result[key] = [])).push(value);
-    });
-    return result;
-  };
-
-  // Use a comparator function to figure out at what index an object should
-  // be inserted so as to maintain order. Uses binary search.
-  _.sortedIndex = function(array, obj, iterator) {
-    iterator || (iterator = _.identity);
-    var low = 0, high = array.length;
-    while (low < high) {
-      var mid = (low + high) >> 1;
-      iterator(array[mid]) < iterator(obj) ? low = mid + 1 : high = mid;
-    }
-    return low;
-  };
-
-  // Safely convert anything iterable into a real, live array.
-  _.toArray = function(iterable) {
-    if (!iterable)                return [];
-    if (iterable.toArray)         return iterable.toArray();
-    if (_.isArray(iterable))      return slice.call(iterable);
-    if (_.isArguments(iterable))  return slice.call(iterable);
-    return _.values(iterable);
-  };
-
-  // Return the number of elements in an object.
-  _.size = function(obj) {
-    return _.toArray(obj).length;
-  };
-
-  // Array Functions
-  // ---------------
-
-  // Get the first element of an array. Passing **n** will return the first N
-  // values in the array. Aliased as `head`. The **guard** check allows it to work
-  // with `_.map`.
-  _.first = _.head = function(array, n, guard) {
-    return (n != null) && !guard ? slice.call(array, 0, n) : array[0];
-  };
-
-  // Returns everything but the last entry of the array. Especcialy useful on
-  // the arguments object. Passing **n** will return all the values in
-  // the array, excluding the last N. The **guard** check allows it to work with
-  // `_.map`.
-  _.initial = function(array, n, guard) {
-    return slice.call(array, 0, array.length - ((n == null) || guard ? 1 : n));
-  };
-
-  // Get the last element of an array. Passing **n** will return the last N
-  // values in the array. The **guard** check allows it to work with `_.map`.
-  _.last = function(array, n, guard) {
-    if ((n != null) && !guard) {
-      return slice.call(array, Math.max(array.length - n, 0));
-    } else {
-      return array[array.length - 1];
-    }
-  };
-
-  // Returns everything but the first entry of the array. Aliased as `tail`.
-  // Especially useful on the arguments object. Passing an **index** will return
-  // the rest of the values in the array from that index onward. The **guard**
-  // check allows it to work with `_.map`.
-  _.rest = _.tail = function(array, index, guard) {
-    return slice.call(array, (index == null) || guard ? 1 : index);
-  };
-
-  // Trim out all falsy values from an array.
-  _.compact = function(array) {
-    return _.filter(array, function(value){ return !!value; });
-  };
-
-  // Return a completely flattened version of an array.
-  _.flatten = function(array, shallow) {
-    return _.reduce(array, function(memo, value) {
-      if (_.isArray(value)) return memo.concat(shallow ? value : _.flatten(value));
-      memo[memo.length] = value;
-      return memo;
-    }, []);
-  };
-
-  // Return a version of the array that does not contain the specified value(s).
-  _.without = function(array) {
-    return _.difference(array, slice.call(arguments, 1));
-  };
-
-  // Produce a duplicate-free version of the array. If the array has already
-  // been sorted, you have the option of using a faster algorithm.
-  // Aliased as `unique`.
-  _.uniq = _.unique = function(array, isSorted, iterator) {
-    var initial = iterator ? _.map(array, iterator) : array;
-    var result = [];
-    _.reduce(initial, function(memo, el, i) {
-      if (0 == i || (isSorted === true ? _.last(memo) != el : !_.include(memo, el))) {
-        memo[memo.length] = el;
-        result[result.length] = array[i];
-      }
-      return memo;
-    }, []);
-    return result;
-  };
-
-  // Produce an array that contains the union: each distinct element from all of
-  // the passed-in arrays.
-  _.union = function() {
-    return _.uniq(_.flatten(arguments, true));
-  };
-
-  // Produce an array that contains every item shared between all the
-  // passed-in arrays. (Aliased as "intersect" for back-compat.)
-  _.intersection = _.intersect = function(array) {
-    var rest = slice.call(arguments, 1);
-    return _.filter(_.uniq(array), function(item) {
-      return _.every(rest, function(other) {
-        return _.indexOf(other, item) >= 0;
-      });
-    });
-  };
-
-  // Take the difference between one array and a number of other arrays.
-  // Only the elements present in just the first array will remain.
-  _.difference = function(array) {
-    var rest = _.flatten(slice.call(arguments, 1));
-    return _.filter(array, function(value){ return !_.include(rest, value); });
-  };
-
-  // Zip together multiple lists into a single array -- elements that share
-  // an index go together.
-  _.zip = function() {
-    var args = slice.call(arguments);
-    var length = _.max(_.pluck(args, 'length'));
-    var results = new Array(length);
-    for (var i = 0; i < length; i++) results[i] = _.pluck(args, "" + i);
-    return results;
-  };
-
-  // If the browser doesn't supply us with indexOf (I'm looking at you, **MSIE**),
-  // we need this function. Return the position of the first occurrence of an
-  // item in an array, or -1 if the item is not included in the array.
-  // Delegates to **ECMAScript 5**'s native `indexOf` if available.
-  // If the array is large and already in sort order, pass `true`
-  // for **isSorted** to use binary search.
-  _.indexOf = function(array, item, isSorted) {
-    if (array == null) return -1;
-    var i, l;
-    if (isSorted) {
-      i = _.sortedIndex(array, item);
-      return array[i] === item ? i : -1;
-    }
-    if (nativeIndexOf && array.indexOf === nativeIndexOf) return array.indexOf(item);
-    for (i = 0, l = array.length; i < l; i++) if (i in array && array[i] === item) return i;
-    return -1;
-  };
-
-  // Delegates to **ECMAScript 5**'s native `lastIndexOf` if available.
-  _.lastIndexOf = function(array, item) {
-    if (array == null) return -1;
-    if (nativeLastIndexOf && array.lastIndexOf === nativeLastIndexOf) return array.lastIndexOf(item);
-    var i = array.length;
-    while (i--) if (i in array && array[i] === item) return i;
-    return -1;
-  };
-
-  // Generate an integer Array containing an arithmetic progression. A port of
-  // the native Python `range()` function. See
-  // [the Python documentation](http://docs.python.org/library/functions.html#range).
-  _.range = function(start, stop, step) {
-    if (arguments.length <= 1) {
-      stop = start || 0;
-      start = 0;
-    }
-    step = arguments[2] || 1;
-
-    var len = Math.max(Math.ceil((stop - start) / step), 0);
-    var idx = 0;
-    var range = new Array(len);
-
-    while(idx < len) {
-      range[idx++] = start;
-      start += step;
-    }
-
-    return range;
-  };
-
-  // Function (ahem) Functions
-  // ------------------
-
-  // Reusable constructor function for prototype setting.
-  var ctor = function(){};
-
-  // Create a function bound to a given object (assigning `this`, and arguments,
-  // optionally). Binding with arguments is also known as `curry`.
-  // Delegates to **ECMAScript 5**'s native `Function.bind` if available.
-  // We check for `func.bind` first, to fail fast when `func` is undefined.
-  _.bind = function bind(func, context) {
-    var bound, args;
-    if (func.bind === nativeBind && nativeBind) return nativeBind.apply(func, slice.call(arguments, 1));
-    if (!_.isFunction(func)) throw new TypeError;
-    args = slice.call(arguments, 2);
-    return bound = function() {
-      if (!(this instanceof bound)) return func.apply(context, args.concat(slice.call(arguments)));
-      ctor.prototype = func.prototype;
-      var self = new ctor;
-      var result = func.apply(self, args.concat(slice.call(arguments)));
-      if (Object(result) === result) return result;
-      return self;
-    };
-  };
-
-  // Bind all of an object's methods to that object. Useful for ensuring that
-  // all callbacks defined on an object belong to it.
-  _.bindAll = function(obj) {
-    var funcs = slice.call(arguments, 1);
-    if (funcs.length == 0) funcs = _.functions(obj);
-    each(funcs, function(f) { obj[f] = _.bind(obj[f], obj); });
-    return obj;
-  };
-
-  // Memoize an expensive function by storing its results.
-  _.memoize = function(func, hasher) {
-    var memo = {};
-    hasher || (hasher = _.identity);
-    return function() {
-      var key = hasher.apply(this, arguments);
-      return _.has(memo, key) ? memo[key] : (memo[key] = func.apply(this, arguments));
-    };
-  };
-
-  // Delays a function for the given number of milliseconds, and then calls
-  // it with the arguments supplied.
-  _.delay = function(func, wait) {
-    var args = slice.call(arguments, 2);
-    return setTimeout(function(){ return func.apply(func, args); }, wait);
-  };
-
-  // Defers a function, scheduling it to run after the current call stack has
-  // cleared.
-  _.defer = function(func) {
-    return _.delay.apply(_, [func, 1].concat(slice.call(arguments, 1)));
-  };
-
-  // Returns a function, that, when invoked, will only be triggered at most once
-  // during a given window of time.
-  _.throttle = function(func, wait) {
-    var context, args, timeout, throttling, more;
-    var whenDone = _.debounce(function(){ more = throttling = false; }, wait);
-    return function() {
-      context = this; args = arguments;
-      var later = function() {
-        timeout = null;
-        if (more) func.apply(context, args);
-        whenDone();
-      };
-      if (!timeout) timeout = setTimeout(later, wait);
-      if (throttling) {
-        more = true;
-      } else {
-        func.apply(context, args);
-      }
-      whenDone();
-      throttling = true;
-    };
-  };
-
-  // Returns a function, that, as long as it continues to be invoked, will not
-  // be triggered. The function will be called after it stops being called for
-  // N milliseconds.
-  _.debounce = function(func, wait) {
-    var timeout;
-    return function() {
-      var context = this, args = arguments;
-      var later = function() {
-        timeout = null;
-        func.apply(context, args);
-      };
-      clearTimeout(timeout);
-      timeout = setTimeout(later, wait);
-    };
-  };
-
-  // Returns a function that will be executed at most one time, no matter how
-  // often you call it. Useful for lazy initialization.
-  _.once = function(func) {
-    var ran = false, memo;
-    return function() {
-      if (ran) return memo;
-      ran = true;
-      return memo = func.apply(this, arguments);
-    };
-  };
-
-  // Returns the first function passed as an argument to the second,
-  // allowing you to adjust arguments, run code before and after, and
-  // conditionally execute the original function.
-  _.wrap = function(func, wrapper) {
-    return function() {
-      var args = [func].concat(slice.call(arguments, 0));
-      return wrapper.apply(this, args);
-    };
-  };
-
-  // Returns a function that is the composition of a list of functions, each
-  // consuming the return value of the function that follows.
-  _.compose = function() {
-    var funcs = arguments;
-    return function() {
-      var args = arguments;
-      for (var i = funcs.length - 1; i >= 0; i--) {
-        args = [funcs[i].apply(this, args)];
-      }
-      return args[0];
-    };
-  };
-
-  // Returns a function that will only be executed after being called N times.
-  _.after = function(times, func) {
-    if (times <= 0) return func();
-    return function() {
-      if (--times < 1) { return func.apply(this, arguments); }
-    };
-  };
-
-  // Object Functions
-  // ----------------
-
-  // Retrieve the names of an object's properties.
-  // Delegates to **ECMAScript 5**'s native `Object.keys`
-  _.keys = nativeKeys || function(obj) {
-    if (obj !== Object(obj)) throw new TypeError('Invalid object');
-    var keys = [];
-    for (var key in obj) if (_.has(obj, key)) keys[keys.length] = key;
-    return keys;
-  };
-
-  // Retrieve the values of an object's properties.
-  _.values = function(obj) {
-    return _.map(obj, _.identity);
-  };
-
-  // Return a sorted list of the function names available on the object.
-  // Aliased as `methods`
-  _.functions = _.methods = function(obj) {
-    var names = [];
-    for (var key in obj) {
-      if (_.isFunction(obj[key])) names.push(key);
-    }
-    return names.sort();
-  };
-
-  // Extend a given object with all the properties in passed-in object(s).
-  _.extend = function(obj) {
-    each(slice.call(arguments, 1), function(source) {
-      for (var prop in source) {
-        obj[prop] = source[prop];
-      }
-    });
-    return obj;
-  };
-
-  // Fill in a given object with default properties.
-  _.defaults = function(obj) {
-    each(slice.call(arguments, 1), function(source) {
-      for (var prop in source) {
-        if (obj[prop] == null) obj[prop] = source[prop];
-      }
-    });
-    return obj;
-  };
-
-  // Create a (shallow-cloned) duplicate of an object.
-  _.clone = function(obj) {
-    if (!_.isObject(obj)) return obj;
-    return _.isArray(obj) ? obj.slice() : _.extend({}, obj);
-  };
-
-  // Invokes interceptor with the obj, and then returns obj.
-  // The primary purpose of this method is to "tap into" a method chain, in
-  // order to perform operations on intermediate results within the chain.
-  _.tap = function(obj, interceptor) {
-    interceptor(obj);
-    return obj;
-  };
-
-  // Internal recursive comparison function.
-  function eq(a, b, stack) {
-    // Identical objects are equal. `0 === -0`, but they aren't identical.
-    // See the Harmony `egal` proposal: http://wiki.ecmascript.org/doku.php?id=harmony:egal.
-    if (a === b) return a !== 0 || 1 / a == 1 / b;
-    // A strict comparison is necessary because `null == undefined`.
-    if (a == null || b == null) return a === b;
-    // Unwrap any wrapped objects.
-    if (a._chain) a = a._wrapped;
-    if (b._chain) b = b._wrapped;
-    // Invoke a custom `isEqual` method if one is provided.
-    if (a.isEqual && _.isFunction(a.isEqual)) return a.isEqual(b);
-    if (b.isEqual && _.isFunction(b.isEqual)) return b.isEqual(a);
-    // Compare `[[Class]]` names.
-    var className = toString.call(a);
-    if (className != toString.call(b)) return false;
-    switch (className) {
-      // Strings, numbers, dates, and booleans are compared by value.
-      case '[object String]':
-        // Primitives and their corresponding object wrappers are equivalent; thus, `"5"` is
-        // equivalent to `new String("5")`.
-        return a == String(b);
-      case '[object Number]':
-        // `NaN`s are equivalent, but non-reflexive. An `egal` comparison is performed for
-        // other numeric values.
-        return a != +a ? b != +b : (a == 0 ? 1 / a == 1 / b : a == +b);
-      case '[object Date]':
-      case '[object Boolean]':
-        // Coerce dates and booleans to numeric primitive values. Dates are compared by their
-        // millisecond representations. Note that invalid dates with millisecond representations
-        // of `NaN` are not equivalent.
-        return +a == +b;
-      // RegExps are compared by their source patterns and flags.
-      case '[object RegExp]':
-        return a.source == b.source &&
-               a.global == b.global &&
-               a.multiline == b.multiline &&
-               a.ignoreCase == b.ignoreCase;
-    }
-    if (typeof a != 'object' || typeof b != 'object') return false;
-    // Assume equality for cyclic structures. The algorithm for detecting cyclic
-    // structures is adapted from ES 5.1 section 15.12.3, abstract operation `JO`.
-    var length = stack.length;
-    while (length--) {
-      // Linear search. Performance is inversely proportional to the number of
-      // unique nested structures.
-      if (stack[length] == a) return true;
-    }
-    // Add the first object to the stack of traversed objects.
-    stack.push(a);
-    var size = 0, result = true;
-    // Recursively compare objects and arrays.
-    if (className == '[object Array]') {
-      // Compare array lengths to determine if a deep comparison is necessary.
-      size = a.length;
-      result = size == b.length;
-      if (result) {
-        // Deep compare the contents, ignoring non-numeric properties.
-        while (size--) {
-          // Ensure commutative equality for sparse arrays.
-          if (!(result = size in a == size in b && eq(a[size], b[size], stack))) break;
-        }
-      }
-    } else {
-      // Objects with different constructors are not equivalent.
-      if ('constructor' in a != 'constructor' in b || a.constructor != b.constructor) return false;
-      // Deep compare objects.
-      for (var key in a) {
-        if (_.has(a, key)) {
-          // Count the expected number of properties.
-          size++;
-          // Deep compare each member.
-          if (!(result = _.has(b, key) && eq(a[key], b[key], stack))) break;
-        }
-      }
-      // Ensure that both objects contain the same number of properties.
-      if (result) {
-        for (key in b) {
-          if (_.has(b, key) && !(size--)) break;
-        }
-        result = !size;
-      }
-    }
-    // Remove the first object from the stack of traversed objects.
-    stack.pop();
-    return result;
-  }
-
-  // Perform a deep comparison to check if two objects are equal.
-  _.isEqual = function(a, b) {
-    return eq(a, b, []);
-  };
-
-  // Is a given array, string, or object empty?
-  // An "empty" object has no enumerable own-properties.
-  _.isEmpty = function(obj) {
-    if (_.isArray(obj) || _.isString(obj)) return obj.length === 0;
-    for (var key in obj) if (_.has(obj, key)) return false;
-    return true;
-  };
-
-  // Is a given value a DOM element?
-  _.isElement = function(obj) {
-    return !!(obj && obj.nodeType == 1);
-  };
-
-  // Is a given value an array?
-  // Delegates to ECMA5's native Array.isArray
-  _.isArray = nativeIsArray || function(obj) {
-    return toString.call(obj) == '[object Array]';
-  };
-
-  // Is a given variable an object?
-  _.isObject = function(obj) {
-    return obj === Object(obj);
-  };
-
-  // Is a given variable an arguments object?
-  _.isArguments = function(obj) {
-    return toString.call(obj) == '[object Arguments]';
-  };
-  if (!_.isArguments(arguments)) {
-    _.isArguments = function(obj) {
-      return !!(obj && _.has(obj, 'callee'));
-    };
-  }
-
-  // Is a given value a function?
-  _.isFunction = function(obj) {
-    return toString.call(obj) == '[object Function]';
-  };
-
-  // Is a given value a string?
-  _.isString = function(obj) {
-    return toString.call(obj) == '[object String]';
-  };
-
-  // Is a given value a number?
-  _.isNumber = function(obj) {
-    return toString.call(obj) == '[object Number]';
-  };
-
-  // Is the given value `NaN`?
-  _.isNaN = function(obj) {
-    // `NaN` is the only value for which `===` is not reflexive.
-    return obj !== obj;
-  };
-
-  // Is a given value a boolean?
-  _.isBoolean = function(obj) {
-    return obj === true || obj === false || toString.call(obj) == '[object Boolean]';
-  };
-
-  // Is a given value a date?
-  _.isDate = function(obj) {
-    return toString.call(obj) == '[object Date]';
-  };
-
-  // Is the given value a regular expression?
-  _.isRegExp = function(obj) {
-    return toString.call(obj) == '[object RegExp]';
-  };
-
-  // Is a given value equal to null?
-  _.isNull = function(obj) {
-    return obj === null;
-  };
-
-  // Is a given variable undefined?
-  _.isUndefined = function(obj) {
-    return obj === void 0;
-  };
-
-  // Has own property?
-  _.has = function(obj, key) {
-    return hasOwnProperty.call(obj, key);
-  };
-
-  // Utility Functions
-  // -----------------
-
-  // Run Underscore.js in *noConflict* mode, returning the `_` variable to its
-  // previous owner. Returns a reference to the Underscore object.
-  _.noConflict = function() {
-    root._ = previousUnderscore;
-    return this;
-  };
-
-  // Keep the identity function around for default iterators.
-  _.identity = function(value) {
-    return value;
-  };
-
-  // Run a function **n** times.
-  _.times = function (n, iterator, context) {
-    for (var i = 0; i < n; i++) iterator.call(context, i);
-  };
-
-  // Escape a string for HTML interpolation.
-  _.escape = function(string) {
-    return (''+string).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;').replace(/'/g, '&#x27;').replace(/\//g,'&#x2F;');
-  };
-
-  // Add your own custom functions to the Underscore object, ensuring that
-  // they're correctly added to the OOP wrapper as well.
-  _.mixin = function(obj) {
-    each(_.functions(obj), function(name){
-      addToWrapper(name, _[name] = obj[name]);
-    });
-  };
-
-  // Generate a unique integer id (unique within the entire client session).
-  // Useful for temporary DOM ids.
-  var idCounter = 0;
-  _.uniqueId = function(prefix) {
-    var id = idCounter++;
-    return prefix ? prefix + id : id;
-  };
-
-  // By default, Underscore uses ERB-style template delimiters, change the
-  // following template settings to use alternative delimiters.
-  _.templateSettings = {
-    evaluate    : /<%([\s\S]+?)%>/g,
-    interpolate : /<%=([\s\S]+?)%>/g,
-    escape      : /<%-([\s\S]+?)%>/g
-  };
-
-  // When customizing `templateSettings`, if you don't want to define an
-  // interpolation, evaluation or escaping regex, we need one that is
-  // guaranteed not to match.
-  var noMatch = /.^/;
-
-  // Within an interpolation, evaluation, or escaping, remove HTML escaping
-  // that had been previously added.
-  var unescape = function(code) {
-    return code.replace(/\\\\/g, '\\').replace(/\\'/g, "'");
-  };
-
-  // JavaScript micro-templating, similar to John Resig's implementation.
-  // Underscore templating handles arbitrary delimiters, preserves whitespace,
-  // and correctly escapes quotes within interpolated code.
-  _.template = function(str, data) {
-    var c  = _.templateSettings;
-    var tmpl = 'var __p=[],print=function(){__p.push.apply(__p,arguments);};' +
-      'with(obj||{}){__p.push(\'' +
-      str.replace(/\\/g, '\\\\')
-         .replace(/'/g, "\\'")
-         .replace(c.escape || noMatch, function(match, code) {
-           return "',_.escape(" + unescape(code) + "),'";
-         })
-         .replace(c.interpolate || noMatch, function(match, code) {
-           return "'," + unescape(code) + ",'";
-         })
-         .replace(c.evaluate || noMatch, function(match, code) {
-           return "');" + unescape(code).replace(/[\r\n\t]/g, ' ') + ";__p.push('";
-         })
-         .replace(/\r/g, '\\r')
-         .replace(/\n/g, '\\n')
-         .replace(/\t/g, '\\t')
-         + "');}return __p.join('');";
-    var func = new Function('obj', '_', tmpl);
-    if (data) return func(data, _);
-    return function(data) {
-      return func.call(this, data, _);
-    };
-  };
-
-  // Add a "chain" function, which will delegate to the wrapper.
-  _.chain = function(obj) {
-    return _(obj).chain();
-  };
-
-  // The OOP Wrapper
-  // ---------------
-
-  // If Underscore is called as a function, it returns a wrapped object that
-  // can be used OO-style. This wrapper holds altered versions of all the
-  // underscore functions. Wrapped objects may be chained.
-  var wrapper = function(obj) { this._wrapped = obj; };
-
-  // Expose `wrapper.prototype` as `_.prototype`
-  _.prototype = wrapper.prototype;
-
-  // Helper function to continue chaining intermediate results.
-  var result = function(obj, chain) {
-    return chain ? _(obj).chain() : obj;
-  };
-
-  // A method to easily add functions to the OOP wrapper.
-  var addToWrapper = function(name, func) {
-    wrapper.prototype[name] = function() {
-      var args = slice.call(arguments);
-      unshift.call(args, this._wrapped);
-      return result(func.apply(_, args), this._chain);
-    };
-  };
-
-  // Add all of the Underscore functions to the wrapper object.
-  _.mixin(_);
-
-  // Add all mutator Array functions to the wrapper.
-  each(['pop', 'push', 'reverse', 'shift', 'sort', 'splice', 'unshift'], function(name) {
-    var method = ArrayProto[name];
-    wrapper.prototype[name] = function() {
-      var wrapped = this._wrapped;
-      method.apply(wrapped, arguments);
-      var length = wrapped.length;
-      if ((name == 'shift' || name == 'splice') && length === 0) delete wrapped[0];
-      return result(wrapped, this._chain);
-    };
-  });
-
-  // Add all accessor Array functions to the wrapper.
-  each(['concat', 'join', 'slice'], function(name) {
-    var method = ArrayProto[name];
-    wrapper.prototype[name] = function() {
-      return result(method.apply(this._wrapped, arguments), this._chain);
-    };
-  });
-
-  // Start chaining a wrapped Underscore object.
-  wrapper.prototype.chain = function() {
-    this._chain = true;
-    return this;
-  };
-
-  // Extracts the result from a wrapped and chained object.
-  wrapper.prototype.value = function() {
-    return this._wrapped;
-  };
-
-}).call(this);

genindex.html

@@ -94,9 +94,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             

getting-started/installation.html

@@ -92,6 +92,7 @@
               <p class="caption"><span class="caption-text">Getting Started</span></p>
 <ul class="current">
 <li class="toctree-l1 current"><a class="current reference internal" href="#">Installation</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#binary-distributions">Binary Distributions</a></li>
 <li class="toctree-l2"><a class="reference internal" href="#from-source">From Source</a><ul>
 <li class="toctree-l3"><a class="reference internal" href="#python-package">Python Package</a></li>
 <li class="toctree-l3"><a class="reference internal" href="#c-package">C++ Package</a></li>
@@ -103,9 +104,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -175,17 +177,24 @@
             
   <div class="section" id="installation">
 <h1>Installation<a class="headerlink" href="#installation" title="Permalink to this headline">¶</a></h1>
+<div class="section" id="binary-distributions">
+<h2>Binary Distributions<a class="headerlink" href="#binary-distributions" title="Permalink to this headline">¶</a></h2>
+<p>You can install the latest nightly release of Triton from pip:</p>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pip install -U --pre triton
+</pre></div>
+</div>
+</div>
 <div class="section" id="from-source">
 <h2>From Source<a class="headerlink" href="#from-source" title="Permalink to this headline">¶</a></h2>
 <div class="section" id="python-package">
 <h3>Python Package<a class="headerlink" href="#python-package" title="Permalink to this headline">¶</a></h3>
 <p>You can install the Python package from source by running the following commands:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo apt-get install llvm-10-dev
-git clone https://github.com/ptillet/triton.git<span class="p">;</span>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/ptillet/triton.git<span class="p">;</span>
 <span class="nb">cd</span> triton/python<span class="p">;</span>
 pip install -e .
 </pre></div>
 </div>
+<p>This may take a while (10-20 minutes) as it will download and compile LLVM from source.</p>
 <p>You can then test your installation by running the unit tests:</p>
 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>pytest -vs .
 </pre></div>
@@ -199,18 +208,13 @@ python -m run --with-plots --result-dir /tmp/triton-bench
 <div class="section" id="c-package">
 <h3>C++ Package<a class="headerlink" href="#c-package" title="Permalink to this headline">¶</a></h3>
 <p>Those not interested in Python integration may want to use the internals of Triton (i.e, runtime, parser, codegen, driver, intermediate representation) directly. This can be done by running the following commands:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>sudo apt-get install llvm-10-dev
-git clone https://github.com/ptillet/triton.git<span class="p">;</span>
+<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>git clone https://github.com/ptillet/triton.git<span class="p">;</span>
 mkdir build<span class="p">;</span>
 <span class="nb">cd</span> build<span class="p">;</span>
 cmake ../<span class="p">;</span>
 make -j8<span class="p">;</span>
 </pre></div>
 </div>
-<p>A custom llvm-config binary can also be provided:</p>
-<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>cmake ../ -DLLVM_CONFIG<span class="o">=</span>/path/to/llvm-config
-</pre></div>
-</div>
 <p>Note that while direct usage of the C++ API is not officially supported, a usage tutorial can be found  <a class="reference external" href="https://github.com/ptillet/triton/blob/master/tutorials/01-matmul.cc">here</a></p>
 </div>
 </div>

getting-started/tutorials/01-vector-add.html

@@ -107,9 +107,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -362,7 +363,7 @@ for different problem sizes.</p>
 </pre></div>
 </div>
 <img alt="vector-add-performance" class="sphx-glr-single-img" src="../../_images/sphx_glr_01-vector-add_001.png" />
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  8.442 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  7.756 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-01-vector-add-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/62d97d49a32414049819dd8bb8378080/01-vector-add.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">01-vector-add.py</span></code></a></p>

getting-started/tutorials/02-fused-softmax.html

@@ -109,9 +109,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -404,7 +405,7 @@ This means that – when temporary data is too large to fit entirely in the GPU
 Note that our Triton kernel is not only faster than PyTorch’s CUDA kernel, it is also <strong>easier to read, understand and maintain</strong>.</p></li>
 </ul>
 </div></blockquote>
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  20.299 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 0 minutes  19.933 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-02-fused-softmax-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/d91442ac2982c4e0cc3ab0f43534afbc/02-fused-softmax.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">02-fused-softmax.py</span></code></a></p>

getting-started/tutorials/03-matrix-multiplication.html

@@ -43,7 +43,7 @@
     
     <link rel="index" title="Index" href="../../genindex.html" />
     <link rel="search" title="Search" href="../../search.html" />
-    <link rel="next" title="Introduction" href="../../programming-guide/introduction.html" />
+    <link rel="next" title="Introduction" href="../../programming-guide/chapter-1/introduction.html" />
     <link rel="prev" title="Fused Softmax" href="02-fused-softmax.html" /> 
 </head>
 
@@ -121,9 +121,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -355,46 +356,14 @@ If <code class="code docutils literal notranslate"><span class="pre">TYPE</span>
 <span class="kn">import</span> <span class="nn">triton</span>
 
 <span class="n">autotune_configs</span> <span class="o">=</span> <span class="p">[</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s2">&quot;MB&quot;</span><span class="p">:</span> <span class="s2">&quot;128&quot;</span><span class="p">,</span>
-        <span class="s2">&quot;NB&quot;</span><span class="p">:</span> <span class="s2">&quot;128&quot;</span><span class="p">,</span>
-        <span class="s2">&quot;KB&quot;</span><span class="p">:</span> <span class="s2">&quot;32&quot;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
-    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span>
-        <span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span>
-        <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span>
-    <span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s2">&quot;MB&quot;</span><span class="p">:</span> <span class="s2">&quot;128&quot;</span><span class="p">,</span> <span class="s2">&quot;NB&quot;</span><span class="p">:</span> <span class="s2">&quot;128&quot;</span><span class="p">,</span> <span class="s2">&quot;KB&quot;</span><span class="p">:</span> <span class="s2">&quot;32&quot;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;128&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">2</span><span class="p">),</span>
+    <span class="n">triton</span><span class="o">.</span><span class="n">config</span><span class="p">(</span><span class="n">defines</span><span class="o">=</span><span class="p">{</span><span class="s1">&#39;MB&#39;</span><span class="p">:</span> <span class="s1">&#39;32&#39;</span><span class="p">,</span> <span class="s1">&#39;NB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">,</span> <span class="s1">&#39;KB&#39;</span><span class="p">:</span> <span class="s1">&#39;64&#39;</span><span class="p">},</span> <span class="n">num_warps</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
 <span class="p">]</span>
 </pre></div>
 </div>
@@ -490,21 +459,21 @@ Note that we need to modify the :code`atol` and <code class="code docutils liter
 </pre></div>
 </div>
 <p class="sphx-glr-script-out">Out:</p>
-<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-        [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-        [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+        [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+        [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
         ...,
-        [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-        [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-        [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+        [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+        [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+        [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
        device=&#39;cuda:0&#39;, dtype=torch.float16)
-tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-        [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-        [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+        [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+        [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
         ...,
-        [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-        [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-        [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+        [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+        [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+        [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
        device=&#39;cuda:0&#39;, dtype=torch.float16)
 True
 </pre></div>
@@ -518,7 +487,7 @@ True
 For this reason, we will instead compare the performance of our kernel against <a class="reference external" href="https://github.com/NVIDIA/cutlass/">CUTLASS</a> , a highly optimized CUDA library for matrix multiplication written by NVIDIA themselves._
 To install CUTLASS, you need a recent version of cmake:</p>
 <blockquote>
-<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> /tmp/
+<div><div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">cd</span> /path/to/cutlass/
 git clone https://github.com/NVIDIA/cutlass.git
 <span class="nb">cd</span> cutlass
 mkdir build
@@ -546,7 +515,7 @@ make -j8 install
 Triton comes with some basic Python bindings for benchmarking CUTLASS. These will be compiled when the environment variables <code class="code docutils literal notranslate"><span class="pre">CUTLASS_INCLUDE_DIR</span></code> and <code class="code docutils literal notranslate"><span class="pre">CUTLASS_LIBRARY_DIR</span></code> are set during the installation process.
 To re-install Triton with the updated CUTLASS bindings, run the following command:</p>
 <div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nb">export</span> <span class="nv">CUTLASS_INCLUDE_DIR</span><span class="o">=</span>/tmp/cutlass/build/install/include/
-<span class="nb">export</span> <span class="nv">CUTLASS_LIBRARY_DIR</span><span class="o">=</span>/tmp/cutlass/build/install/lib/
+<span class="nb">export</span> <span class="nv">CUTLASS_LIBRARY_DIR</span><span class="o">=</span>/tmp/cutlass/build/install/lib/a
 pip uninstall -y triton
 pip install -e <span class="s2">&quot;git+https://github.com/ptillet/triton.git#egg=triton&amp;subdirectory=python&quot;</span>
 </pre></div>
@@ -559,13 +528,13 @@ pip install -e <span class="s2">&quot;git+https://github.com/ptillet/triton.git#
 </pre></div>
 </div>
 <p class="sphx-glr-script-out">Out:</p>
-<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>tensor([[199.0000, 199.1250, 195.8750,  ..., 190.6250, 200.7500, 186.3750],
-        [196.1250, 201.6250, 197.6250,  ..., 189.6250, 197.7500, 190.0000],
-        [198.0000, 196.6250, 200.1250,  ..., 198.6250, 199.7500, 190.8750],
+<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>tensor([[199.6250, 198.0000, 195.0000,  ..., 186.0000, 193.6250, 202.1250],
+        [192.6250, 193.6250, 190.7500,  ..., 184.2500, 191.2500, 192.1250],
+        [192.3750, 196.6250, 188.8750,  ..., 185.5000, 188.7500, 191.8750],
         ...,
-        [190.3750, 192.0000, 190.5000,  ..., 187.0000, 191.7500, 180.8750],
-        [185.2500, 187.6250, 181.2500,  ..., 185.1250, 188.2500, 175.5000],
-        [191.6250, 191.6250, 194.2500,  ..., 188.2500, 192.1250, 182.0000]],
+        [196.6250, 199.8750, 196.1250,  ..., 182.6250, 194.5000, 200.8750],
+        [199.2500, 200.3750, 191.7500,  ..., 186.8750, 192.8750, 193.5000],
+        [193.5000, 195.2500, 194.1250,  ..., 188.3750, 192.6250, 198.3750]],
        device=&#39;cuda:0&#39;, dtype=torch.float16)
 True
 </pre></div>
@@ -605,7 +574,7 @@ True
 </div>
 <img alt="matmul-performance" class="sphx-glr-single-img" src="../../_images/sphx_glr_03-matrix-multiplication_001.png" />
 <p>As we can see, the performance of our kernel is pretty good. It is in fact faster than CUTLASS, and therefore probably comparable to the absolute best CUDA code an expert could write.</p>
-<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 1 minutes  10.094 seconds)</p>
+<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 1 minutes  6.502 seconds)</p>
 <div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-getting-started-tutorials-03-matrix-multiplication-py">
 <div class="sphx-glr-download sphx-glr-download-python docutils container">
 <p><a class="reference download internal" download="" href="../../_downloads/d5fee5b55a64e47f1b5724ec39adf171/03-matrix-multiplication.py"><code class="xref download docutils literal notranslate"><span class="pre">Download</span> <span class="pre">Python</span> <span class="pre">source</span> <span class="pre">code:</span> <span class="pre">03-matrix-multiplication.py</span></code></a></p>
@@ -625,7 +594,7 @@ True
           </div>
           <footer>
     <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
-        <a href="../../programming-guide/introduction.html" class="btn btn-neutral float-right" title="Introduction" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
+        <a href="../../programming-guide/chapter-1/introduction.html" class="btn btn-neutral float-right" title="Introduction" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
         <a href="02-fused-softmax.html" class="btn btn-neutral float-left" title="Fused Softmax" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
     </div>
 

getting-started/tutorials/index.html

@@ -101,9 +101,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             

getting-started/tutorials/sg_execution_times.html

@@ -94,9 +94,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -166,7 +167,7 @@
             
   <div class="section" id="computation-times">
 <span id="sphx-glr-getting-started-tutorials-sg-execution-times"></span><h1>Computation times<a class="headerlink" href="#computation-times" title="Permalink to this headline">¶</a></h1>
-<p><strong>01:10.094</strong> total execution time for <strong>getting-started_tutorials</strong> files:</p>
+<p><strong>01:34.190</strong> total execution time for <strong>getting-started_tutorials</strong> files:</p>
 <table class="docutils align-default">
 <colgroup>
 <col style="width: 85%" />
@@ -175,15 +176,15 @@
 </colgroup>
 <tbody>
 <tr class="row-odd"><td><p><a class="reference internal" href="03-matrix-multiplication.html#sphx-glr-getting-started-tutorials-03-matrix-multiplication-py"><span class="std std-ref">Matrix Multiplication</span></a> (<code class="docutils literal notranslate"><span class="pre">03-matrix-multiplication.py</span></code>)</p></td>
-<td><p>01:10.094</p></td>
+<td><p>01:06.502</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
-<tr class="row-even"><td><p><a class="reference internal" href="01-vector-add.html#sphx-glr-getting-started-tutorials-01-vector-add-py"><span class="std std-ref">Vector Addition</span></a> (<code class="docutils literal notranslate"><span class="pre">01-vector-add.py</span></code>)</p></td>
-<td><p>00:00.000</p></td>
+<tr class="row-even"><td><p><a class="reference internal" href="02-fused-softmax.html#sphx-glr-getting-started-tutorials-02-fused-softmax-py"><span class="std std-ref">Fused Softmax</span></a> (<code class="docutils literal notranslate"><span class="pre">02-fused-softmax.py</span></code>)</p></td>
+<td><p>00:19.933</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
-<tr class="row-odd"><td><p><a class="reference internal" href="02-fused-softmax.html#sphx-glr-getting-started-tutorials-02-fused-softmax-py"><span class="std std-ref">Fused Softmax</span></a> (<code class="docutils literal notranslate"><span class="pre">02-fused-softmax.py</span></code>)</p></td>
-<td><p>00:00.000</p></td>
+<tr class="row-odd"><td><p><a class="reference internal" href="01-vector-add.html#sphx-glr-getting-started-tutorials-01-vector-add-py"><span class="std std-ref">Vector Addition</span></a> (<code class="docutils literal notranslate"><span class="pre">01-vector-add.py</span></code>)</p></td>
+<td><p>00:07.756</p></td>
 <td><p>0.0 MB</p></td>
 </tr>
 </tbody>

index.html

@@ -95,9 +95,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -181,9 +182,10 @@
 <h2>Programming Guide<a class="headerlink" href="#programming-guide" title="Permalink to this headline">¶</a></h2>
 <p>Check out the following documents to learn more about Triton and how it compares against other DSLs for DNNs:</p>
 <ul class="simple">
-<li><p>Chapter 1: <a class="reference internal" href="programming-guide/introduction.html"><span class="doc">Introduction</span></a></p></li>
-<li><p>Chapter 2: <a class="reference internal" href="programming-guide/related-work.html"><span class="doc">Related Work</span></a></p></li>
-<li><p>Chapter 3: <a class="reference internal" href="programming-guide/triton-c.html"><span class="doc">The Triton-C Kernel Language</span></a></p></li>
+<li><p>Chapter 1: <a class="reference internal" href="programming-guide/chapter-1/introduction.html"><span class="doc">Introduction</span></a></p></li>
+<li><p>Chapter 2: <a class="reference internal" href="programming-guide/chapter-2/related-work.html"><span class="doc">Related Work</span></a></p></li>
+<li><p>Chapter 3: <a class="reference internal" href="programming-guide/chapter-3/triton-c.html"><span class="doc">The Triton-C Language</span></a></p></li>
+<li><p>Chapter 4: <a class="reference internal" href="programming-guide/chapter-4/triton-ir.html"><span class="doc">The Triton-IR Intermediate Representation</span></a></p></li>
 </ul>
 <div class="toctree-wrapper compound">
 </div>

programming-guide/c-language-contractions.html

@@ -1,212 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>&lt;no title&gt; &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>&lt;no title&gt;</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/c-language-contractions.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

programming-guide/c-language-extensions.html

@@ -1,212 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>&lt;no title&gt; &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>&lt;no title&gt;</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/c-language-extensions.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

programming-guide/chapter-1/introduction.html

@@ -11,13 +11,13 @@
   
 
   
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-binder.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-dataframe.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-rendered-html.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />
 
   
   
@@ -29,22 +29,22 @@
 
   
   <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
+    <script src="../../_static/js/html5shiv.min.js"></script>
   <![endif]-->
   
     
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
+      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+        <script src="../../_static/jquery.js"></script>
+        <script src="../../_static/underscore.js"></script>
+        <script src="../../_static/doctools.js"></script>
     
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
+    <script type="text/javascript" src="../../_static/js/theme.js"></script>
 
     
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" />
-    <link rel="next" title="Related Work" href="related-work.html" />
-    <link rel="prev" title="Matrix Multiplication" href="../getting-started/tutorials/03-matrix-multiplication.html" /> 
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="next" title="Related Work" href="../chapter-2/related-work.html" />
+    <link rel="prev" title="Matrix Multiplication" href="../../getting-started/tutorials/03-matrix-multiplication.html" /> 
 </head>
 
 <body class="wy-body-for-nav">
@@ -58,7 +58,7 @@
           
 
           
-            <a href="../index.html" class="icon icon-home"> Triton
+            <a href="../../index.html" class="icon icon-home"> Triton
           
 
           
@@ -71,7 +71,7 @@
 
           
 <div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+  <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
     <input type="text" name="q" placeholder="Search docs" />
     <input type="hidden" name="check_keywords" value="yes" />
     <input type="hidden" name="area" value="default" />
@@ -91,8 +91,8 @@
             
               <p class="caption"><span class="caption-text">Getting Started</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/tutorials/index.html">Tutorials</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul class="current">
@@ -102,8 +102,9 @@
 <li class="toctree-l2"><a class="reference internal" href="#references">References</a></li>
 </ul>
 </li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -119,7 +120,7 @@
       <nav class="wy-nav-top" aria-label="top navigation">
         
           <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
+          <a href="../../index.html">Triton</a>
         
       </nav>
 
@@ -150,7 +151,7 @@
 
   <ul class="wy-breadcrumbs">
     
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
+      <li><a href="../../index.html" class="icon icon-home"></a> &raquo;</li>
         
       <li>Introduction</li>
     
@@ -158,7 +159,7 @@
       <li class="wy-breadcrumbs-aside">
         
           
-            <a href="../_sources/programming-guide/introduction.rst.txt" rel="nofollow"> View page source</a>
+            <a href="../../_sources/programming-guide/chapter-1/introduction.rst.txt" rel="nofollow"> View page source</a>
           
         
       </li>
@@ -175,10 +176,10 @@
 <h1>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline">¶</a></h1>
 <div class="section" id="motivations">
 <h2>Motivations<a class="headerlink" href="#motivations" title="Permalink to this headline">¶</a></h2>
-<p>Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing <a class="footnote-reference brackets" href="#id10" id="id1">1</a> to computer vision <a class="footnote-reference brackets" href="#id11" id="id2">2</a> to computational neuroscience <a class="footnote-reference brackets" href="#id12" id="id3">3</a>. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) <em>layers</em>. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.</p>
+<p>Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing <a class="reference internal" href="#sutskever2014" id="id1"><span>[SUTSKEVER2014]</span></a> to computer vision <a class="reference internal" href="#redmon2016" id="id2"><span>[REDMON2016]</span></a> to computational neuroscience <a class="reference internal" href="#lee2017" id="id3"><span>[LEE2017]</span></a>. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) <em>layers</em>. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.</p>
 <p>As a consequence, Graphics Processing Units (GPUs) have become a cheap and accessible resource for exploring and/or deploying novel research ideas in the field. This trend has been accelerated by the release of several frameworks for General-Purpose GPU (GPGPU) computing, such as CUDA and OpenCL, which have made the development of high-performance programs easier. Yet, GPUs remain incredibly challenging to optimize for locality and parallelism, especially for computations that cannot be efficiently implemented using a combination of pre-existing optimized primitives. To make matters worse, GPU architectures are also rapidly evolving and specializing, as evidenced by the addition of tensor cores to NVIDIA (and more recently AMD) micro-architectures.</p>
-<p>This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial academic and industrial interest for Domain-Specific Languages (DSLs) and compilers. Regrettably, these systems – whether they be based on  polyhedral machinery (<em>e.g.</em>, Tiramisu <a class="footnote-reference brackets" href="#id13" id="id4">4</a>, Tensor Comprehensions <a class="footnote-reference brackets" href="#id14" id="id5">5</a>) or scheduling languages (<em>e.g.</em>, Halide <a class="footnote-reference brackets" href="#id15" id="id6">6</a>, TVM <a class="footnote-reference brackets" href="#id16" id="id7">7</a>) – remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like <a class="reference external" href="https://docs.nvidia.com/cuda/cublas/index.html">cuBLAS</a>, <a class="reference external" href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html">cuDNN</a> or <a class="reference external" href="https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html">TensorRT</a>.</p>
-<p>The main premise of this project is the following: programming paradigms based on blocked algorithms <a class="footnote-reference brackets" href="#id17" id="id8">8</a> can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional “Single Program, Multiple Data” (SPMD <a class="footnote-reference brackets" href="#id18" id="id9">9</a>) execution models for GPUs, and propose a variant in which programs – rather than threads – are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:</p>
+<p>This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial academic and industrial interest for Domain-Specific Languages (DSLs) and compilers. Regrettably, these systems – whether they be based on  polyhedral machinery (<em>e.g.</em>, Tiramisu <a class="reference internal" href="../chapter-2/related-work.html#baghdadi2021" id="id4"><span>[BAGHDADI2021]</span></a>, Tensor Comprehensions <a class="reference internal" href="../chapter-2/related-work.html#vasilache2018" id="id5"><span>[VASILACHE2018]</span></a>) or scheduling languages (<em>e.g.</em>, Halide <a class="reference internal" href="#jrk2013" id="id6"><span>[JRK2013]</span></a>, TVM <a class="reference internal" href="#chen2018" id="id7"><span>[CHEN2018]</span></a>) – remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like <a class="reference external" href="https://docs.nvidia.com/cuda/cublas/index.html">cuBLAS</a>, <a class="reference external" href="https://docs.nvidia.com/deeplearning/cudnn/api/index.html">cuDNN</a> or <a class="reference external" href="https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html">TensorRT</a>.</p>
+<p>The main premise of this project is the following: programming paradigms based on blocked algorithms <a class="reference internal" href="#lam1991" id="id8"><span>[LAM1991]</span></a> can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional “Single Program, Multiple Data” (SPMD <a class="reference internal" href="#auguin1983" id="id9"><span>[AUGUIN1983]</span></a>) execution models for GPUs, and propose a variant in which programs – rather than threads – are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:</p>
 <table class="colwidths-given docutils align-default">
 <colgroup>
 <col style="width: 50%" />
@@ -221,8 +222,8 @@
 </div>
 </td>
 </tr>
-<tr class="row-odd"><td><p><img alt="pic1" src="../_images/cuda-parallel-matmul.png" /></p></td>
-<td><p><img alt="pic2" src="../_images/triton-parallel-matmul.png" /></p></td>
+<tr class="row-odd"><td><p><img alt="pic1" src="../../_images/cuda-parallel-matmul.png" /></p></td>
+<td><p><img alt="pic2" src="../../_images/triton-parallel-matmul.png" /></p></td>
 </tr>
 </tbody>
 </table>
@@ -234,33 +235,51 @@
 </div>
 <div class="section" id="references">
 <h2>References<a class="headerlink" href="#references" title="Permalink to this headline">¶</a></h2>
-<dl class="footnote brackets">
-<dt class="label" id="id10"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
-<dd><p>Sutskever et al., “Sequence to Sequence Learning with Neural Networks”, NIPS 2014</p>
+<dl class="citation">
+<dt class="label" id="sutskever2014"><span class="brackets"><a class="fn-backref" href="#id1">SUTSKEVER2014</a></span></dt>
+<dd><ol class="upperroman simple">
+<li><p>Sutskever et al., “Sequence to Sequence Learning with Neural Networks”, NIPS 2014</p></li>
+</ol>
 </dd>
-<dt class="label" id="id11"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
-<dd><p>Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016</p>
+<dt class="label" id="redmon2016"><span class="brackets"><a class="fn-backref" href="#id2">REDMON2016</a></span></dt>
+<dd><ol class="upperalpha simple" start="10">
+<li><p>Redmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016</p></li>
+</ol>
 </dd>
-<dt class="label" id="id12"><span class="brackets"><a class="fn-backref" href="#id3">3</a></span></dt>
-<dd><p>Lee et al., “Superhuman Accuracy on the SNEMI3D Connectomics Challenge”, ArXiV 2017</p>
+<dt class="label" id="lee2017"><span class="brackets"><a class="fn-backref" href="#id3">LEE2017</a></span></dt>
+<dd><ol class="upperalpha simple" start="11">
+<li><p>Lee et al., “Superhuman Accuracy on the SNEMI3D Connectomics Challenge”, ArXiV 2017</p></li>
+</ol>
 </dd>
-<dt class="label" id="id13"><span class="brackets"><a class="fn-backref" href="#id4">4</a></span></dt>
-<dd><p>Baghdadi et al., “Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code”, CGO 2021</p>
+<dt class="label" id="baghdadi2021"><span class="brackets"><a class="fn-backref" href="#id4">BAGHDADI2021</a></span></dt>
+<dd><ol class="upperalpha simple" start="18">
+<li><p>Baghdadi et al., “Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code”, CGO 2021</p></li>
+</ol>
 </dd>
-<dt class="label" id="id14"><span class="brackets"><a class="fn-backref" href="#id5">5</a></span></dt>
-<dd><p>Vasilache et al., “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, ArXiV 2018</p>
+<dt class="label" id="vasilache2018"><span class="brackets"><a class="fn-backref" href="#id5">VASILACHE2018</a></span></dt>
+<dd><ol class="upperalpha simple" start="14">
+<li><p>Vasilache et al., “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, ArXiV 2018</p></li>
+</ol>
 </dd>
-<dt class="label" id="id15"><span class="brackets"><a class="fn-backref" href="#id6">6</a></span></dt>
-<dd><p>Ragan-Kelley et al., “Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines”, PLDI 2013</p>
+<dt class="label" id="jrk2013"><span class="brackets"><a class="fn-backref" href="#id6">JRK2013</a></span></dt>
+<dd><ol class="upperalpha simple" start="10">
+<li><p>Ragan-Kelley et al., “Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines”, PLDI 2013</p></li>
+</ol>
 </dd>
-<dt class="label" id="id16"><span class="brackets"><a class="fn-backref" href="#id7">7</a></span></dt>
-<dd><p>Chen et al., “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning”, OSDI 2018</p>
+<dt class="label" id="chen2018"><span class="brackets"><a class="fn-backref" href="#id7">CHEN2018</a></span></dt>
+<dd><ol class="upperalpha simple" start="20">
+<li><p>Chen et al., “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning”, OSDI 2018</p></li>
+</ol>
 </dd>
-<dt class="label" id="id17"><span class="brackets"><a class="fn-backref" href="#id8">8</a></span></dt>
-<dd><p>Lam et al., “The Cache Performance and Optimizations of Blocked Algorithms”, ASPLOS 1991</p>
+<dt class="label" id="lam1991"><span class="brackets"><a class="fn-backref" href="#id8">LAM1991</a></span></dt>
+<dd><ol class="upperalpha simple" start="13">
+<li><p>Lam et al., “The Cache Performance and Optimizations of Blocked Algorithms”, ASPLOS 1991</p></li>
+</ol>
 </dd>
-<dt class="label" id="id18"><span class="brackets"><a class="fn-backref" href="#id9">9</a></span></dt>
-<dd><p>Auguin et al., “Opsila: an advanced SIMD for numerical analysis and signal processing”, EUROMICRO 1983</p>
+<dt class="label" id="auguin1983"><span class="brackets"><a class="fn-backref" href="#id9">AUGUIN1983</a></span></dt>
+<dd><ol class="upperalpha simple" start="13">
+<li><p>Auguin et al., “Opsila: an advanced SIMD for numerical analysis and signal processing”, EUROMICRO 1983</p></li>
+</ol>
 </dd>
 </dl>
 </div>
@@ -272,8 +291,8 @@
           </div>
           <footer>
     <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
-        <a href="related-work.html" class="btn btn-neutral float-right" title="Related Work" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
-        <a href="../getting-started/tutorials/03-matrix-multiplication.html" class="btn btn-neutral float-left" title="Matrix Multiplication" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
+        <a href="../chapter-2/related-work.html" class="btn btn-neutral float-right" title="Related Work" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
+        <a href="../../getting-started/tutorials/03-matrix-multiplication.html" class="btn btn-neutral float-left" title="Matrix Multiplication" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
     </div>
 
   <hr/>

programming-guide/chapter-2/related-work.html

@@ -11,13 +11,13 @@
   
 
   
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-binder.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-dataframe.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-rendered-html.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />
 
   
   
@@ -29,23 +29,23 @@
 
   
   <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
+    <script src="../../_static/js/html5shiv.min.js"></script>
   <![endif]-->
   
     
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
+      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+        <script src="../../_static/jquery.js"></script>
+        <script src="../../_static/underscore.js"></script>
+        <script src="../../_static/doctools.js"></script>
         <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
     
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
+    <script type="text/javascript" src="../../_static/js/theme.js"></script>
 
     
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" />
-    <link rel="next" title="The Triton-C Language" href="triton-c.html" />
-    <link rel="prev" title="Introduction" href="introduction.html" /> 
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="next" title="The Triton-C Language" href="../chapter-3/triton-c.html" />
+    <link rel="prev" title="Introduction" href="../chapter-1/introduction.html" /> 
 </head>
 
 <body class="wy-body-for-nav">
@@ -59,7 +59,7 @@
           
 
           
-            <a href="../index.html" class="icon icon-home"> Triton
+            <a href="../../index.html" class="icon icon-home"> Triton
           
 
           
@@ -72,7 +72,7 @@
 
           
 <div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+  <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
     <input type="text" name="q" placeholder="Search docs" />
     <input type="hidden" name="check_keywords" value="yes" />
     <input type="hidden" name="area" value="default" />
@@ -92,12 +92,12 @@
             
               <p class="caption"><span class="caption-text">Getting Started</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/tutorials/index.html">Tutorials</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul class="current">
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-1/introduction.html">Introduction</a></li>
 <li class="toctree-l1 current"><a class="current reference internal" href="#">Related Work</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="#polyhedral-compilation">Polyhedral Compilation</a><ul>
 <li class="toctree-l3"><a class="reference internal" href="#program-representation">Program Representation</a></li>
@@ -113,7 +113,8 @@
 <li class="toctree-l2"><a class="reference internal" href="#references">References</a></li>
 </ul>
 </li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -129,7 +130,7 @@
       <nav class="wy-nav-top" aria-label="top navigation">
         
           <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
+          <a href="../../index.html">Triton</a>
         
       </nav>
 
@@ -160,7 +161,7 @@
 
   <ul class="wy-breadcrumbs">
     
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
+      <li><a href="../../index.html" class="icon icon-home"></a> &raquo;</li>
         
       <li>Related Work</li>
     
@@ -168,7 +169,7 @@
       <li class="wy-breadcrumbs-aside">
         
           
-            <a href="../_sources/programming-guide/related-work.rst.txt" rel="nofollow"> View page source</a>
+            <a href="../../_sources/programming-guide/chapter-2/related-work.rst.txt" rel="nofollow"> View page source</a>
           
         
       </li>
@@ -186,7 +187,7 @@
 <p>At first sight, Triton may seem like just yet another DSL for DNNs. The purpose of this section is to contextualize Triton and highlights its differences with the two leading approaches in this domain: polyhedral compilation and scheduling languages.</p>
 <div class="section" id="polyhedral-compilation">
 <h2>Polyhedral Compilation<a class="headerlink" href="#polyhedral-compilation" title="Permalink to this headline">¶</a></h2>
-<p>Traditional compilers typically rely on intermediate representations, such as LLVM-IR <a class="footnote-reference brackets" href="#id17" id="id1">1</a>, that encode control flow information using (un)conditional branches. This relatively low-level format makes it difficult to statically analyze the runtime behavior (e.g., cache misses) of input programs, and to  automatically optimize loops accordingly through the use of tiling <a class="footnote-reference brackets" href="#id18" id="id2">2</a>, fusion <a class="footnote-reference brackets" href="#id19" id="id3">3</a> and interchange <a class="footnote-reference brackets" href="#id20" id="id4">4</a>. To solve this issue, polyhedral compilers <a class="footnote-reference brackets" href="#id21" id="id5">5</a> rely on program representations that have statically predictable control flow, thereby enabling aggressive compile-time program transformations for data locality and parallelism. Though this strategy has been adopted by many languages and compilers for DNNs such as Tiramisu <a class="footnote-reference brackets" href="#id22" id="id6">6</a>, Tensor Comprehensions <a class="footnote-reference brackets" href="#id23" id="id7">7</a>, Diesel <a class="footnote-reference brackets" href="#id24" id="id8">8</a> and the Affine dialect in MLIR <a class="footnote-reference brackets" href="#id25" id="id9">9</a>, it also comes with a number of limitations that will be described later.</p>
+<p>Traditional compilers typically rely on intermediate representations, such as LLVM-IR <a class="reference internal" href="#lattner2004" id="id1"><span>[LATTNER2004]</span></a>, that encode control flow information using (un)conditional branches. This relatively low-level format makes it difficult to statically analyze the runtime behavior (e.g., cache misses) of input programs, and to  automatically optimize loops accordingly through the use of tiling <a class="reference internal" href="#wolfe1989" id="id2"><span>[WOLFE1989]</span></a>, fusion <a class="reference internal" href="#darte1999" id="id3"><span>[DARTE1999]</span></a> and interchange <a class="reference internal" href="#allen1984" id="id4"><span>[ALLEN1984]</span></a>. To solve this issue, polyhedral compilers <a class="reference internal" href="#ancourt1991" id="id5"><span>[ANCOURT1991]</span></a> rely on program representations that have statically predictable control flow, thereby enabling aggressive compile-time program transformations for data locality and parallelism. Though this strategy has been adopted by many languages and compilers for DNNs such as Tiramisu <a class="reference internal" href="#baghdadi2021" id="id6"><span>[BAGHDADI2021]</span></a>, Tensor Comprehensions <a class="reference internal" href="#vasilache2018" id="id7"><span>[VASILACHE2018]</span></a>, Diesel <a class="reference internal" href="#elango2018" id="id8"><span>[ELANGO2018]</span></a> and the Affine dialect in MLIR <a class="reference internal" href="#lattner2019" id="id9"><span>[LATTNER2019]</span></a>, it also comes with a number of limitations that will be described later in this section.</p>
 <div class="section" id="program-representation">
 <h3>Program Representation<a class="headerlink" href="#program-representation" title="Permalink to this headline">¶</a></h3>
 <p>Polyhedral compilation is a vast area of research. In this section we only outline the most basic aspects of this topic, but readers interested in the solid mathematical foundations underneath may refer to the ample litterature on linear and integer programming.</p>
@@ -202,7 +203,7 @@
 </pre></div>
 </div>
 </td>
-<td><p><a class="reference internal" href="../_images/polyhedral-iteration.png"><img alt="pic1" src="../_images/polyhedral-iteration.png" style="width: 300px;" /></a></p></td>
+<td><p><a class="reference internal" href="../../_images/polyhedral-iteration.png"><img alt="pic1" src="../../_images/polyhedral-iteration.png" style="width: 300px;" /></a></p></td>
 </tr>
 </tbody>
 </table>
@@ -269,14 +270,14 @@ i &amp; j
 <div class="section" id="advantages">
 <h3>Advantages<a class="headerlink" href="#advantages" title="Permalink to this headline">¶</a></h3>
 <p>Programs amenable to polyhedral compilation can be aggressively transformed and optimized. Most of these transformations actually boil down to the production of  schedules and iteration domains that enable loop transformations promoting parallelism and spatial/temporal data locality (e.g., fusion, interchange, tiling, parallelization).</p>
-<p>Polyhedral compilers can also automatically go through complex verification processes to ensure that the semantics of their input program is preserved throughout this optimization phase. Note that polyhedral optimizers are not incompatible with more standard optimization techniques. In fact, it is not uncommon for these systems to be implemented as a set of LLVM passes that can be run ahead of more traditional compilation techniques <a class="footnote-reference brackets" href="#id26" id="id10">10</a>.</p>
-<p>All in all, polyhedral machinery is extremely powerful, when applicable. It has been shown to support most common loop transformations, and has indeed achieved performance comparable to state-of-the-art GPU libraries for dense matrix multiplication <a class="footnote-reference brackets" href="#id24" id="id11">8</a>. Additionally, it is also fully automatic and doesn’t require any hint from programmers apart from source-code in a C-like format.</p>
+<p>Polyhedral compilers can also automatically go through complex verification processes to ensure that the semantics of their input program is preserved throughout this optimization phase. Note that polyhedral optimizers are not incompatible with more standard optimization techniques. In fact, it is not uncommon for these systems to be implemented as a set of LLVM passes that can be run ahead of more traditional compilation techniques <a class="reference internal" href="#grosser2012" id="id10"><span>[GROSSER2012]</span></a>.</p>
+<p>All in all, polyhedral machinery is extremely powerful, when applicable. It has been shown to support most common loop transformations, and has indeed achieved performance comparable to state-of-the-art GPU libraries for dense matrix multiplication <a class="reference internal" href="#elango2018" id="id11"><span>[ELANGO2018]</span></a>. Additionally, it is also fully automatic and doesn’t require any hint from programmers apart from source-code in a C-like format.</p>
 </div>
 <div class="section" id="limitations">
 <h3>Limitations<a class="headerlink" href="#limitations" title="Permalink to this headline">¶</a></h3>
 <p>Unfortunately, polyhedral compilers suffer from two major limitations that have prevented its adoption as a universal method for code generation in neural networks.</p>
-<p>First, the set of possible program transformations $Omega = { Theta_S ~|~ S in text{program} }$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures <a class="footnote-reference brackets" href="#id27" id="id12">11</a>.</p>
-<p>Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common <a class="footnote-reference brackets" href="#id28" id="id13">12</a> but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse – or even structured-sparse – neural networks, whose importance has been rapidly rising over the past few years.</p>
+<p>First, the set of possible program transformations $Omega = { Theta_S ~|~ S in text{program} }$ is large, and grows with the number of statements in the program as well as with the size of their iteration domain. Verifying the legality of each transformation can also require the resolution of complex integer linear programs, making polyhedral compilation very computationally expensive. To make matters worse, hardware properties (e.g., cache size, number of SMs) and contextual characteristics (e.g., input tensor shapes) also have to be taken into account by this framework, leading to expensive auto-tuning procedures <a class="reference internal" href="#sato2019" id="id12"><span>[SATO2019]</span></a>.</p>
+<p>Second, the polyhedral framework is not very generally applicable; SCoPs are relatively common <a class="reference internal" href="#girbal2006" id="id13"><span>[GIRBAL2006]</span></a> but require loop bounds and array subscripts to be affine functions of loop indices, which typically only occurs in regular, dense computations. For this reason, this framework still has to be successfully applied to sparse – or even structured-sparse – neural networks, whose importance has been rapidly rising over the past few years.</p>
 <p>On the other hand, blocked program representations advocated by this dissertation are less restricted in scope and can achieve close to peak performance using standard dataflow analysis.</p>
 </div>
 </div>
@@ -316,7 +317,7 @@ i &amp; j
     <span class="o">.</span><span class="n">parallel</span><span class="p">(</span><span class="n">y</span><span class="p">)</span><span class="o">.</span><span class="n">vectorize</span><span class="p">(</span><span class="n">xii</span><span class="p">)</span><span class="o">.</span><span class="n">unroll</span><span class="p">(</span><span class="n">xi</span><span class="p">)</span><span class="o">.</span><span class="n">unroll</span><span class="p">(</span><span class="n">yii</span><span class="p">);</span>
 </pre></div>
 </td></tr></table></div>
-<p>The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms <a class="footnote-reference brackets" href="#id29" id="id14">13</a>.</p>
+<p>The resulting code may however not be completely portable, as schedules can sometimes rely on execution models (e.g., SPMD) or hardware intrinsics (e.g., matrix-multiply-accumulate) that are not widely available. This issue can be mitigated by auto-scheduling mechanisms <a class="reference internal" href="#mullapudi2016" id="id14"><span>[MULLAPUDI2016]</span></a>.</p>
 <div class="section" id="id15">
 <h3>Advantages<a class="headerlink" href="#id15" title="Permalink to this headline">¶</a></h3>
 <p>The main advantage of this approach is that it allows programmers to write an algorithm <em>only once</em>, and focus on performance optimization separately. It makes it possible to manually specify optimizations that a polyhedral compiler wouldn’t be able to figure out automatically using static data-flow analysis.</p>
@@ -340,7 +341,7 @@ i &amp; j
 </pre></div>
 </div>
 </td>
-<td><p><a class="reference internal" href="../_images/halide-iteration.png"><img alt="pic2" src="../_images/halide-iteration.png" style="width: 300px;" /></a></p></td>
+<td><p><a class="reference internal" href="../../_images/halide-iteration.png"><img alt="pic2" src="../../_images/halide-iteration.png" style="width: 300px;" /></a></p></td>
 </tr>
 </tbody>
 </table>
@@ -349,45 +350,71 @@ i &amp; j
 </div>
 <div class="section" id="references">
 <h2>References<a class="headerlink" href="#references" title="Permalink to this headline">¶</a></h2>
-<dl class="footnote brackets">
-<dt class="label" id="id17"><span class="brackets"><a class="fn-backref" href="#id1">1</a></span></dt>
-<dd><p>Lattner et al., “LLVM: a compilation framework for lifelong program analysis transformation”</p>
+<dl class="citation">
+<dt class="label" id="lattner2004"><span class="brackets"><a class="fn-backref" href="#id1">LATTNER2004</a></span></dt>
+<dd><ol class="upperalpha simple" start="3">
+<li><p>Lattner et al., “LLVM: a compilation framework for lifelong program analysis transformation”, CGO 2004</p></li>
+</ol>
 </dd>
-<dt class="label" id="id18"><span class="brackets"><a class="fn-backref" href="#id2">2</a></span></dt>
-<dd><p>Wolfe, “More Iteration Space Tiling”, SC 1989</p>
+<dt class="label" id="wolfe1989"><span class="brackets"><a class="fn-backref" href="#id2">WOLFE1989</a></span></dt>
+<dd><ol class="upperalpha simple" start="13">
+<li><p>Wolfe, “More Iteration Space Tiling”, SC 1989</p></li>
+</ol>
 </dd>
-<dt class="label" id="id19"><span class="brackets"><a class="fn-backref" href="#id3">3</a></span></dt>
-<dd><p>Darte, “On the Complexity of Loop Fusion”, PACT 1999</p>
+<dt class="label" id="darte1999"><span class="brackets"><a class="fn-backref" href="#id3">DARTE1999</a></span></dt>
+<dd><ol class="upperalpha simple">
+<li><p>Darte, “On the Complexity of Loop Fusion”, PACT 1999</p></li>
+</ol>
 </dd>
-<dt class="label" id="id20"><span class="brackets"><a class="fn-backref" href="#id4">4</a></span></dt>
-<dd><p>Allen et al., “Automatic Loop Interchange”, SIGPLAN Notices 1984</p>
+<dt class="label" id="allen1984"><span class="brackets"><a class="fn-backref" href="#id4">ALLEN1984</a></span></dt>
+<dd><ol class="upperalpha simple" start="10">
+<li><p>Allen et al., “Automatic Loop Interchange”, SIGPLAN Notices 1984</p></li>
+</ol>
 </dd>
-<dt class="label" id="id21"><span class="brackets"><a class="fn-backref" href="#id5">5</a></span></dt>
-<dd><p>Ancourt et al., “Scanning Polyhedra with DO Loops”, PPoPP 1991</p>
+<dt class="label" id="ancourt1991"><span class="brackets"><a class="fn-backref" href="#id5">ANCOURT1991</a></span></dt>
+<dd><ol class="upperalpha simple" start="3">
+<li><p>Ancourt et al., “Scanning Polyhedra with DO Loops”, PPoPP 1991</p></li>
+</ol>
 </dd>
-<dt class="label" id="id22"><span class="brackets"><a class="fn-backref" href="#id6">6</a></span></dt>
-<dd><p>Baghdadi et al., “Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code”, CGO 2021</p>
+<dt class="label" id="baghdadi2021"><span class="brackets"><a class="fn-backref" href="#id6">BAGHDADI2021</a></span></dt>
+<dd><ol class="upperalpha simple" start="18">
+<li><p>Baghdadi et al., “Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code”, CGO 2021</p></li>
+</ol>
 </dd>
-<dt class="label" id="id23"><span class="brackets"><a class="fn-backref" href="#id7">7</a></span></dt>
-<dd><p>Vasilache et al., “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, ArXiV 2018</p>
+<dt class="label" id="vasilache2018"><span class="brackets"><a class="fn-backref" href="#id7">VASILACHE2018</a></span></dt>
+<dd><ol class="upperalpha simple" start="14">
+<li><p>Vasilache et al., “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, ArXiV 2018</p></li>
+</ol>
 </dd>
-<dt class="label" id="id24"><span class="brackets">8</span><span class="fn-backref">(<a href="#id8">1</a>,<a href="#id11">2</a>)</span></dt>
-<dd><p>Elango et al. “Diesel: DSL for Linear Algebra and Neural Net Computations on GPUs”, MAPL 2018</p>
+<dt class="label" id="elango2018"><span class="brackets">ELANGO2018</span><span class="fn-backref">(<a href="#id8">1</a>,<a href="#id11">2</a>)</span></dt>
+<dd><ol class="upperalpha simple" start="22">
+<li><p>Elango et al. “Diesel: DSL for Linear Algebra and Neural Net Computations on GPUs”, MAPL 2018</p></li>
+</ol>
 </dd>
-<dt class="label" id="id25"><span class="brackets"><a class="fn-backref" href="#id9">9</a></span></dt>
-<dd><p>Lattner et al., “MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law”, Arxiv 2019</p>
+<dt class="label" id="lattner2019"><span class="brackets"><a class="fn-backref" href="#id9">LATTNER2019</a></span></dt>
+<dd><ol class="upperalpha simple" start="3">
+<li><p>Lattner et al., “MLIR Primer: A Compiler Infrastructure for the End of Moore’s Law”, Arxiv 2019</p></li>
+</ol>
 </dd>
-<dt class="label" id="id26"><span class="brackets"><a class="fn-backref" href="#id10">10</a></span></dt>
-<dd><p>Grosser et al., “Polly - Performing Polyhedral Optimizations on a Low-Level Intermediate Representation”, Parallel Processing Letters 2012</p>
+<dt class="label" id="grosser2012"><span class="brackets"><a class="fn-backref" href="#id10">GROSSER2012</a></span></dt>
+<dd><ol class="upperalpha simple" start="20">
+<li><p>Grosser et al., “Polly - Performing Polyhedral Optimizations on a Low-Level Intermediate Representation”, Parallel Processing Letters 2012</p></li>
+</ol>
 </dd>
-<dt class="label" id="id27"><span class="brackets"><a class="fn-backref" href="#id12">11</a></span></dt>
-<dd><p>Sato et al., “An Autotuning Framework for Scalable Execution of Tiled Code via Iterative Polyhedral Compilation”, TACO 2019</p>
+<dt class="label" id="sato2019"><span class="brackets"><a class="fn-backref" href="#id12">SATO2019</a></span></dt>
+<dd><ol class="upperalpha simple" start="25">
+<li><p>Sato et al., “An Autotuning Framework for Scalable Execution of Tiled Code via Iterative Polyhedral Compilation”, TACO 2019</p></li>
+</ol>
 </dd>
-<dt class="label" id="id28"><span class="brackets"><a class="fn-backref" href="#id13">12</a></span></dt>
-<dd><p>Girbal et al., “Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies”, International Journal of Parallel Programming 2006</p>
+<dt class="label" id="girbal2006"><span class="brackets"><a class="fn-backref" href="#id13">GIRBAL2006</a></span></dt>
+<dd><ol class="upperalpha simple" start="19">
+<li><p>Girbal et al., “Semi-Automatic Composition of Loop Transformations for Deep Parallelism and Memory Hierarchies”, International Journal of Parallel Programming 2006</p></li>
+</ol>
 </dd>
-<dt class="label" id="id29"><span class="brackets"><a class="fn-backref" href="#id14">13</a></span></dt>
-<dd><p>Mullapudi et al., “Automatically scheduling halide image processing pipelines”, TOG 2016</p>
+<dt class="label" id="mullapudi2016"><span class="brackets"><a class="fn-backref" href="#id14">MULLAPUDI2016</a></span></dt>
+<dd><ol class="upperalpha simple" start="18">
+<li><p>Mullapudi et al., “Automatically scheduling halide image processing pipelines”, TOG 2016</p></li>
+</ol>
 </dd>
 </dl>
 </div>
@@ -399,8 +426,8 @@ i &amp; j
           </div>
           <footer>
     <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
-        <a href="triton-c.html" class="btn btn-neutral float-right" title="The Triton-C Language" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
-        <a href="introduction.html" class="btn btn-neutral float-left" title="Introduction" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
+        <a href="../chapter-3/triton-c.html" class="btn btn-neutral float-right" title="The Triton-C Language" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
+        <a href="../chapter-1/introduction.html" class="btn btn-neutral float-left" title="Introduction" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
     </div>
 
   <hr/>

programming-guide/chapter-3/triton-c.html

@@ -11,13 +11,13 @@
   
 
   
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-binder.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-dataframe.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-rendered-html.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />
 
   
   
@@ -29,21 +29,22 @@
 
   
   <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
+    <script src="../../_static/js/html5shiv.min.js"></script>
   <![endif]-->
   
     
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
+      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+        <script src="../../_static/jquery.js"></script>
+        <script src="../../_static/underscore.js"></script>
+        <script src="../../_static/doctools.js"></script>
     
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
+    <script type="text/javascript" src="../../_static/js/theme.js"></script>
 
     
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" />
-    <link rel="prev" title="Related Work" href="related-work.html" /> 
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="next" title="The Triton-IR Intermediate Representation" href="../chapter-4/triton-ir.html" />
+    <link rel="prev" title="Related Work" href="../chapter-2/related-work.html" /> 
 </head>
 
 <body class="wy-body-for-nav">
@@ -57,7 +58,7 @@
           
 
           
-            <a href="../index.html" class="icon icon-home"> Triton
+            <a href="../../index.html" class="icon icon-home"> Triton
           
 
           
@@ -70,7 +71,7 @@
 
           
 <div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
+  <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
     <input type="text" name="q" placeholder="Search docs" />
     <input type="hidden" name="check_keywords" value="yes" />
     <input type="hidden" name="area" value="default" />
@@ -90,13 +91,13 @@
             
               <p class="caption"><span class="caption-text">Getting Started</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/tutorials/index.html">Tutorials</a></li>
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul class="current">
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-2/related-work.html">Related Work</a></li>
 <li class="toctree-l1 current"><a class="current reference internal" href="#">The Triton-C Language</a><ul>
 <li class="toctree-l2"><a class="reference internal" href="#differences-with-c">Differences with C</a><ul>
 <li class="toctree-l3"><a class="reference internal" href="#extensions">Extensions</a></li>
@@ -110,6 +111,7 @@
 <li class="toctree-l2"><a class="reference internal" href="#programming-model">Programming Model</a></li>
 </ul>
 </li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>
 
             
@@ -125,7 +127,7 @@
       <nav class="wy-nav-top" aria-label="top navigation">
         
           <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
+          <a href="../../index.html">Triton</a>
         
       </nav>
 
@@ -156,7 +158,7 @@
 
   <ul class="wy-breadcrumbs">
     
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
+      <li><a href="../../index.html" class="icon icon-home"></a> &raquo;</li>
         
       <li>The Triton-C Language</li>
     
@@ -164,7 +166,7 @@
       <li class="wy-breadcrumbs-aside">
         
           
-            <a href="../_sources/programming-guide/triton-c.rst.txt" rel="nofollow"> View page source</a>
+            <a href="../../_sources/programming-guide/chapter-3/triton-c.rst.txt" rel="nofollow"> View page source</a>
           
         
       </li>
@@ -190,15 +192,15 @@
 <p><strong>Operators and built-in function</strong>: The usual C operators were extended to support element-wise array operations (<code class="code docutils literal notranslate"><span class="pre">+</span></code>, <code class="code docutils literal notranslate"><span class="pre">-</span></code>, <code class="code docutils literal notranslate"><span class="pre">&amp;&amp;</span></code>, <code class="code docutils literal notranslate"><span class="pre">*</span></code>, etc.) and complex array operations(<code class="code docutils literal notranslate"><span class="pre">&#64;</span></code> for matrix multiplication). Additionally, some built-in functions were added for concurrency (<code class="code docutils literal notranslate"><span class="pre">get_program_id</span></code>, <code class="code docutils literal notranslate"><span class="pre">atomic_add</span></code>).</p>
 <p><strong>Slicing and broadcasting</strong>: Multi-dimensional blocks can be broadcast along any particular dimension using numpy-like slicing syntax (e.g., <code class="code docutils literal notranslate"><span class="pre">int</span> <span class="pre">array[8,</span> <span class="pre">8]</span> <span class="pre">=</span> <span class="pre">range[:,</span> <span class="pre">newaxis]</span></code> for stacking columns). Note that, as of now, slicing blocks to retrieve sub-blocks (or scalars) is forbidden as it is incompatible with the automatic parallelization methods used by our JIT. Reductions can be achieved using a syntax similar to slicing (e.g., <code class="code docutils literal notranslate"><span class="pre">array[+]</span></code> for summing an array, or <code class="code docutils literal notranslate"><span class="pre">array[:,</span> <span class="pre">max]</span></code> for row-wise maximum). Currently supported reduction operators are <code class="code docutils literal notranslate"><span class="pre">+</span></code>, <code class="code docutils literal notranslate"><span class="pre">min</span></code>, <code class="code docutils literal notranslate"><span class="pre">max</span></code>.</p>
 <p><strong>Masked pointer dereferencement</strong>: Block-level operations in Triton-C are “atomic”, in the sense that they execute either completely or not at all. Basic element-wise control-flow for block-level operations can nonetheless be achieved using ternary operators and the <em>masked pointer dereferencement</em> operator exemplified below:</p>
-<div class="highlight-C notranslate"><div class="highlight"><pre><span></span>// create mask
-bool mask[16, 16] = ...;
-// conditional addition
-float x[16, 16] = mask ? a + b : 0;
-// conditional load
-float y[16] 16] = mask ? *ptr : 0;
-// conditional store
-*?(mask)ptr = y;
-\end{lstlisting}
+<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="c1">// create mask</span>
+<span class="kt">bool</span> <span class="n">mask</span><span class="p">[</span><span class="mi">16</span><span class="p">,</span> <span class="mi">16</span><span class="p">]</span> <span class="o">=</span> <span class="p">...;</span>
+<span class="c1">// conditional addition</span>
+<span class="kt">float</span> <span class="n">x</span><span class="p">[</span><span class="mi">16</span><span class="p">,</span> <span class="mi">16</span><span class="p">]</span> <span class="o">=</span> <span class="n">mask</span> <span class="o">?</span> <span class="n">a</span> <span class="o">+</span> <span class="nl">b</span> <span class="p">:</span> <span class="mi">0</span><span class="p">;</span>
+<span class="c1">// conditional load</span>
+<span class="kt">float</span> <span class="n">y</span><span class="p">[</span><span class="mi">16</span><span class="p">]</span> <span class="mi">16</span><span class="p">]</span> <span class="o">=</span> <span class="n">mask</span> <span class="o">?</span> <span class="o">*</span><span class="nl">ptr</span> <span class="p">:</span> <span class="mi">0</span><span class="p">;</span>
+<span class="c1">// conditional store</span>
+<span class="o">*?</span><span class="p">(</span><span class="n">mask</span><span class="p">)</span><span class="n">ptr</span> <span class="o">=</span> <span class="n">y</span><span class="p">;</span>
+<span class="err">\</span><span class="n">end</span><span class="p">{</span><span class="n">lstlisting</span><span class="p">}</span>
 </pre></div>
 </div>
 </div>
@@ -242,7 +244,8 @@ float y[16] 16] = mask ? *ptr : 0;
           </div>
           <footer>
     <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
-        <a href="related-work.html" class="btn btn-neutral float-left" title="Related Work" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
+        <a href="../chapter-4/triton-ir.html" class="btn btn-neutral float-right" title="The Triton-IR Intermediate Representation" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
+        <a href="../chapter-2/related-work.html" class="btn btn-neutral float-left" title="Related Work" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
     </div>
 
   <hr/>

programming-guide/chapter-4/triton-ir.html

@@ -0,0 +1,301 @@
+
+
+<!DOCTYPE html>
+<html class="writer-html5" lang="en" >
+<head>
+  <meta charset="utf-8" />
+  
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  
+  <title>The Triton-IR Intermediate Representation &mdash; Triton  documentation</title>
+  
+
+  
+  <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-binder.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-dataframe.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/gallery-rendered-html.css" type="text/css" />
+  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />
+
+  
+  
+
+  
+  
+
+  
+
+  
+  <!--[if lt IE 9]>
+    <script src="../../_static/js/html5shiv.min.js"></script>
+  <![endif]-->
+  
+    
+      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
+        <script src="../../_static/jquery.js"></script>
+        <script src="../../_static/underscore.js"></script>
+        <script src="../../_static/doctools.js"></script>
+        <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
+    
+    <script type="text/javascript" src="../../_static/js/theme.js"></script>
+
+    
+    <link rel="index" title="Index" href="../../genindex.html" />
+    <link rel="search" title="Search" href="../../search.html" />
+    <link rel="prev" title="The Triton-C Language" href="../chapter-3/triton-c.html" /> 
+</head>
+
+<body class="wy-body-for-nav">
+
+   
+  <div class="wy-grid-for-nav">
+    
+    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
+      <div class="wy-side-scroll">
+        <div class="wy-side-nav-search" >
+          
+
+          
+            <a href="../../index.html" class="icon icon-home"> Triton
+          
+
+          
+          </a>
+
+          
+            
+            
+          
+
+          
+<div role="search">
+  <form id="rtd-search-form" class="wy-form" action="../../search.html" method="get">
+    <input type="text" name="q" placeholder="Search docs" />
+    <input type="hidden" name="check_keywords" value="yes" />
+    <input type="hidden" name="area" value="default" />
+  </form>
+</div>
+
+          
+        </div>
+
+        
+        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
+          
+            
+            
+              
+            
+            
+              <p class="caption"><span class="caption-text">Getting Started</span></p>
+<ul>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/installation.html">Installation</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../../getting-started/tutorials/index.html">Tutorials</a></li>
+</ul>
+<p class="caption"><span class="caption-text">Programming Guide</span></p>
+<ul class="current">
+<li class="toctree-l1"><a class="reference internal" href="../chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="../chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1 current"><a class="current reference internal" href="#">The Triton-IR Intermediate Representation</a><ul>
+<li class="toctree-l2"><a class="reference internal" href="#structure-of-a-triton-ir-program">Structure of a Triton-IR Program</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#modules">Modules</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#functions">Functions</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#basic-blocks">Basic Blocks</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#block-level-dataflow-analysis">Block-Level Dataflow Analysis</a><ul>
+<li class="toctree-l3"><a class="reference internal" href="#types">Types</a></li>
+<li class="toctree-l3"><a class="reference internal" href="#instructions">Instructions</a></li>
+</ul>
+</li>
+<li class="toctree-l2"><a class="reference internal" href="#block-level-control-flow-analysis">Block-Level Control Flow Analysis</a></li>
+<li class="toctree-l2"><a class="reference internal" href="#references">References</a></li>
+</ul>
+</li>
+</ul>
+
+            
+          
+        </div>
+        
+      </div>
+    </nav>
+
+    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
+
+      
+      <nav class="wy-nav-top" aria-label="top navigation">
+        
+          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
+          <a href="../../index.html">Triton</a>
+        
+      </nav>
+
+
+      <div class="wy-nav-content">
+        
+        <div class="rst-content">
+        
+          
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+<div role="navigation" aria-label="breadcrumbs navigation">
+
+  <ul class="wy-breadcrumbs">
+    
+      <li><a href="../../index.html" class="icon icon-home"></a> &raquo;</li>
+        
+      <li>The Triton-IR Intermediate Representation</li>
+    
+    
+      <li class="wy-breadcrumbs-aside">
+        
+          
+            <a href="../../_sources/programming-guide/chapter-4/triton-ir.rst.txt" rel="nofollow"> View page source</a>
+          
+        
+      </li>
+    
+  </ul>
+
+  
+  <hr/>
+</div>
+          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
+           <div itemprop="articleBody">
+            
+  <div class="section" id="the-triton-ir-intermediate-representation">
+<h1>The Triton-IR Intermediate Representation<a class="headerlink" href="#the-triton-ir-intermediate-representation" title="Permalink to this headline">¶</a></h1>
+<p>Triton-IR is an LLVM-based Intermediate Representation (IR) whose purpose is to provide an environment suitable for block-level program analysis, transformation and optimization.
+In our implementation, Triton-IR programs are constructed directly from Triton-C after parsing, but they could also be formed directly by higher-level DSLs in the future.
+Triton-IR and LLVM-IR programs share the same high-level structure, but the former also includes a number of extensions necessary for block-level data-flow analysis.
+These extensions are crucial for carrying out the optimizations outlined in the next chapter of this document.</p>
+<div class="section" id="structure-of-a-triton-ir-program">
+<h2>Structure of a Triton-IR Program<a class="headerlink" href="#structure-of-a-triton-ir-program" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="modules">
+<h3>Modules<a class="headerlink" href="#modules" title="Permalink to this headline">¶</a></h3>
+<p>At the highest level, Triton-IR programs consist of one or multiple basic units of compilation known as <em>modules</em>. These modules are compiled independently from one another, and eventually aggregated by a linker whose role is to resolve forward declarations and adequately merge global definitions. Each module itself is composed of functions, global variables, constants and other miscellaneous symbols such as metadata and attributes.</p>
+</div>
+<div class="section" id="functions">
+<h3>Functions<a class="headerlink" href="#functions" title="Permalink to this headline">¶</a></h3>
+<p>Triton-IR function definitions consist of a return type, a name and a potentially empty arguments list. Additional visibility, alignment and linkage specifiers can be added if desired. Function attributes (such as inlining hints) and parameter attributes (such as “readonly”, aliasing hints) can also be specified, allowing compiler backends to perform more aggressive optimizations by, for instance, making better use of non-coherent caches found on NVIDIA GPUs. This header is followed by a body composed of a list of basic blocks whose interdependencies form the Control Flow Graph (CFG) of the function.</p>
+</div>
+<div class="section" id="basic-blocks">
+<h3>Basic Blocks<a class="headerlink" href="#basic-blocks" title="Permalink to this headline">¶</a></h3>
+<p>Basic blocks are straight-line code sequences that may only contain so-called <em>terminator</em> instructions (i.e., branching, return) at their end. To simplify program analysis, Triton-IR uses the Static Single Assignment (SSA) form, meaning that each variable in each basic block must be (1) assigned to only once and (2) defined before being used. In so doing, each basic block implicitly defines a Data-Flow Graph (DFG). In our case, the SSA form is created directly from Triton-C’s Abstract Syntax Trees (ASTs) using an algorithm from the literature <a class="reference internal" href="#braun13" id="id1"><span>[BRAUN13]</span></a>.</p>
+</div>
+</div>
+<div class="section" id="block-level-dataflow-analysis">
+<h2>Block-Level Dataflow Analysis<a class="headerlink" href="#block-level-dataflow-analysis" title="Permalink to this headline">¶</a></h2>
+<div class="section" id="types">
+<h3>Types<a class="headerlink" href="#types" title="Permalink to this headline">¶</a></h3>
+<p>Multi-dimensional blocks are at the center of data-flow analysis in Triton-JIT. They can be declared using syntax similar to vector declarations in LLVM-IR. For example, <code class="code docutils literal notranslate"><span class="pre">i32&lt;8,</span> <span class="pre">8&gt;</span></code> is the type corresponding to <span class="math notranslate nohighlight">\(8 \times 8\)</span> blocks of 32-bit integers. Note that there is no preprocessor in Triton-IR, hence parametric shape  values must be resolved before programs are generated. In our case, this is done by Triton-JIT’s auto-tuner.</p>
+</div>
+<div class="section" id="instructions">
+<h3>Instructions<a class="headerlink" href="#instructions" title="Permalink to this headline">¶</a></h3>
+<p>Triton-IR introduces a set of <em>reblocking</em> instructions whose purpose is to support broadcasting semantics as described in the previous chapter.  The <code class="code docutils literal notranslate"><span class="pre">reshape</span></code> instruction creates a block of the specified shape using the raw data from its input argument. This is particularly useful to re-interpret variables as higher-dimensional arrays by padding their input shapes with ones in preparation for broadcasting. The <code class="code docutils literal notranslate"><span class="pre">broadcast</span></code> instruction creates a block of the specified shapes by replicating its input argument as many times as necessary along dimensions of size 1 – as shown below for the <code class="code docutils literal notranslate"><span class="pre">broadcast&lt;3,3&gt;</span></code> instruction.</p>
+<p><a class="reference internal" href="../../_images/broadcast-1.png"><img alt="pic1" src="../../_images/broadcast-1.png" style="width: 40%;" /></a> and <a class="reference internal" href="../../_images/broadcast-2.png"><img alt="pic2" src="../../_images/broadcast-2.png" style="width: 40%;" /></a></p>
+<p>Usual scalar instructions (<code class="code docutils literal notranslate"><span class="pre">cmp</span></code>, <code class="code docutils literal notranslate"><span class="pre">getelementptr</span></code>, <code class="code docutils literal notranslate"><span class="pre">add</span></code>, <code class="code docutils literal notranslate"><span class="pre">load</span></code>…) were preserved and extended to signify element-wise operations when applicable. Finally, Triton-IR also exposes specialized arithmetic instructions for reductions (<code class="code docutils literal notranslate"><span class="pre">reduce</span></code>) and matrix multiplications (<code class="code docutils literal notranslate"><span class="pre">dot</span></code>).</p>
+</div>
+</div>
+<div class="section" id="block-level-control-flow-analysis">
+<h2>Block-Level Control Flow Analysis<a class="headerlink" href="#block-level-control-flow-analysis" title="Permalink to this headline">¶</a></h2>
+<p>In Triton-IR, operations on block variables are atomic: they execute either in full or not at all. As a result, traditional control flow structures (e.g., conditional, loops) are not applicable to individual block elements. This is problematic, since a program may need to e.g., partially guard blocked loads against memory access violations.</p>
+<p>This could be potentially solved through the use of the Predicated SSA (PSSA) <a class="reference internal" href="#carter99" id="id2"><span>[CARTER99]</span></a> <a class="reference internal" href="#stoutchinin01" id="id3"><span>[STOUTCHININ01]</span></a> form for Triton-IR. However, this would create a lot of unnecessary complexity for GPUs, where the benefits of PSSA are close to none as divergent program paths  within warps are  serialized anyway. Therefore, recent versions of Triton handle intra-block control flow in a much simpler way, using conditional instructions such as  <code class="code docutils literal notranslate"><span class="pre">select</span></code>, <code class="code docutils literal notranslate"><span class="pre">masked_load</span></code> and <code class="code docutils literal notranslate"><span class="pre">masked_store</span></code>:</p>
+<div class="highlight-C notranslate"><div class="highlight"><pre><span></span><span class="c1">// For all indices [idx], return cond[idx] ? true_value[idx] : false_value[idx];</span>
+<span class="n">select</span>       <span class="n">TYPE</span><span class="o">&lt;</span><span class="n">TS1</span><span class="p">,</span> <span class="p">...,</span> <span class="n">TSN</span><span class="o">&gt;</span> <span class="n">cond</span><span class="p">,</span> <span class="n">true_value</span><span class="p">,</span> <span class="n">false_value</span><span class="p">;</span>
+<span class="c1">// For all indices [idx], return cond[idx] ? *true_addr[idx] : false_value[idx];</span>
+<span class="n">masked_load</span>  <span class="n">TYPE</span><span class="o">&lt;</span><span class="n">TS1</span><span class="p">,</span> <span class="p">...,</span> <span class="n">TSN</span><span class="o">&gt;</span> <span class="n">cond</span><span class="p">,</span> <span class="n">true_addr</span><span class="p">,</span> <span class="n">false_value</span><span class="p">;</span>
+<span class="c1">// For all indices [idx], execute *true_addr[idx] = true_value[idx] if cond[idx]</span>
+<span class="n">masked_store</span> <span class="n">TYPE</span><span class="o">&lt;</span><span class="n">TS1</span><span class="p">,</span> <span class="p">...,</span> <span class="n">TSN</span><span class="o">&gt;</span> <span class="n">cond</span><span class="p">,</span> <span class="n">true_addr</span><span class="p">,</span> <span class="n">true_value</span><span class="p">;</span>
+</pre></div>
+</div>
+</div>
+<div class="section" id="references">
+<h2>References<a class="headerlink" href="#references" title="Permalink to this headline">¶</a></h2>
+<dl class="citation">
+<dt class="label" id="braun13"><span class="brackets"><a class="fn-backref" href="#id1">BRAUN13</a></span></dt>
+<dd><ol class="upperalpha simple" start="13">
+<li><p>Braun et al., “Simple and Efficient Construction of Static Single Assignment Form”, CC 2013</p></li>
+</ol>
+</dd>
+<dt class="label" id="carter99"><span class="brackets"><a class="fn-backref" href="#id2">CARTER99</a></span></dt>
+<dd><ol class="upperalpha simple" start="12">
+<li><p>Carter et al., “Predicated Static Single Assignment”, PACT 1999</p></li>
+</ol>
+</dd>
+<dt class="label" id="stoutchinin01"><span class="brackets"><a class="fn-backref" href="#id3">STOUTCHININ01</a></span></dt>
+<dd><ol class="upperalpha simple">
+<li><p>Stoutchinin et al., “Efficient Static Single Assignment Form for Predication”, MICRO 2001</p></li>
+</ol>
+</dd>
+</dl>
+</div>
+</div>
+
+
+           </div>
+           
+          </div>
+          <footer>
+    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
+        <a href="../chapter-3/triton-c.html" class="btn btn-neutral float-left" title="The Triton-C Language" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
+    </div>
+
+  <hr/>
+
+  <div role="contentinfo">
+    <p>
+        &#169; Copyright 2020, Philippe Tillet.
+
+    </p>
+  </div>
+    
+    
+    
+    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
+    
+    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
+    
+    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
+
+</footer>
+        </div>
+      </div>
+
+    </section>
+
+  </div>
+  
+
+  <script type="text/javascript">
+      jQuery(function () {
+          SphinxRtdTheme.Navigation.enable(true);
+      });
+  </script>
+
+  
+  
+    
+   
+
+</body>
+</html>

programming-guide/index.html

@@ -1,219 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>Programming Guide &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-        <script async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-AMS-MML_HTMLorMML"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>Programming Guide</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/index.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  <div class="section" id="programming-guide">
-<h1>Programming Guide<a class="headerlink" href="#programming-guide" title="Permalink to this headline">¶</a></h1>
-<div class="toctree-wrapper compound">
-<p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-</div>
-</div>
-
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

programming-guide/performance-optimization.html

@@ -1,212 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>&lt;no title&gt; &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>&lt;no title&gt;</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/performance-optimization.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

programming-guide/programming-interface.html

@@ -1,212 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>&lt;no title&gt; &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>&lt;no title&gt;</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/programming-interface.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

programming-guide/programming-model.html

@@ -1,212 +0,0 @@
-
-
-<!DOCTYPE html>
-<html class="writer-html5" lang="en" >
-<head>
-  <meta charset="utf-8" />
-  
-  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-  
-  <title>&lt;no title&gt; &mdash; Triton  documentation</title>
-  
-
-  
-  <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-binder.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-dataframe.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/gallery-rendered-html.css" type="text/css" />
-  <link rel="stylesheet" href="../_static/css/custom.css" type="text/css" />
-
-  
-  
-
-  
-  
-
-  
-
-  
-  <!--[if lt IE 9]>
-    <script src="../_static/js/html5shiv.min.js"></script>
-  <![endif]-->
-  
-    
-      <script type="text/javascript" id="documentation_options" data-url_root="../" src="../_static/documentation_options.js"></script>
-        <script src="../_static/jquery.js"></script>
-        <script src="../_static/underscore.js"></script>
-        <script src="../_static/doctools.js"></script>
-    
-    <script type="text/javascript" src="../_static/js/theme.js"></script>
-
-    
-    <link rel="index" title="Index" href="../genindex.html" />
-    <link rel="search" title="Search" href="../search.html" /> 
-</head>
-
-<body class="wy-body-for-nav">
-
-   
-  <div class="wy-grid-for-nav">
-    
-    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
-      <div class="wy-side-scroll">
-        <div class="wy-side-nav-search" >
-          
-
-          
-            <a href="../index.html" class="icon icon-home"> Triton
-          
-
-          
-          </a>
-
-          
-            
-            
-          
-
-          
-<div role="search">
-  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
-    <input type="text" name="q" placeholder="Search docs" />
-    <input type="hidden" name="check_keywords" value="yes" />
-    <input type="hidden" name="area" value="default" />
-  </form>
-</div>
-
-          
-        </div>
-
-        
-        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
-          
-            
-            
-              
-            
-            
-              <p class="caption"><span class="caption-text">Getting Started</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/installation.html">Installation</a></li>
-<li class="toctree-l1"><a class="reference internal" href="../getting-started/tutorials/index.html">Tutorials</a></li>
-</ul>
-<p class="caption"><span class="caption-text">Programming Guide</span></p>
-<ul>
-<li class="toctree-l1"><a class="reference internal" href="introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="triton-c.html">The Triton-C Language</a></li>
-</ul>
-
-            
-          
-        </div>
-        
-      </div>
-    </nav>
-
-    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
-
-      
-      <nav class="wy-nav-top" aria-label="top navigation">
-        
-          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
-          <a href="../index.html">Triton</a>
-        
-      </nav>
-
-
-      <div class="wy-nav-content">
-        
-        <div class="rst-content">
-        
-          
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-<div role="navigation" aria-label="breadcrumbs navigation">
-
-  <ul class="wy-breadcrumbs">
-    
-      <li><a href="../index.html" class="icon icon-home"></a> &raquo;</li>
-        
-      <li>&lt;no title&gt;</li>
-    
-    
-      <li class="wy-breadcrumbs-aside">
-        
-          
-            <a href="../_sources/programming-guide/programming-model.rst.txt" rel="nofollow"> View page source</a>
-          
-        
-      </li>
-    
-  </ul>
-
-  
-  <hr/>
-</div>
-          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
-           <div itemprop="articleBody">
-            
-  
-
-           </div>
-           
-          </div>
-          <footer>
-
-  <hr/>
-
-  <div role="contentinfo">
-    <p>
-        &#169; Copyright 2020, Philippe Tillet.
-
-    </p>
-  </div>
-    
-    
-    
-    Built with <a href="https://www.sphinx-doc.org/">Sphinx</a> using a
-    
-    <a href="https://github.com/readthedocs/sphinx_rtd_theme">theme</a>
-    
-    provided by <a href="https://readthedocs.org">Read the Docs</a>. 
-
-</footer>
-        </div>
-      </div>
-
-    </section>
-
-  </div>
-  
-
-  <script type="text/javascript">
-      jQuery(function () {
-          SphinxRtdTheme.Navigation.enable(true);
-      });
-  </script>
-
-  
-  
-    
-   
-
-</body>
-</html>

search.html

@@ -97,9 +97,10 @@
 </ul>
 <p class="caption"><span class="caption-text">Programming Guide</span></p>
 <ul>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/introduction.html">Introduction</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/related-work.html">Related Work</a></li>
-<li class="toctree-l1"><a class="reference internal" href="programming-guide/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-1/introduction.html">Introduction</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-2/related-work.html">Related Work</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-3/triton-c.html">The Triton-C Language</a></li>
+<li class="toctree-l1"><a class="reference internal" href="programming-guide/chapter-4/triton-ir.html">The Triton-IR Intermediate Representation</a></li>
 </ul>