triton/master/.doctrees/programming-guide/chapter-1/introduction.doctree

<EFBFBD><05><><EFBFBD><00>sphinx.addnodes<65><73>document<6E><74><EFBFBD>)<29><>}<7D>(<28>	rawsource<63><65><00><>children<65>]<5D><>docutils.nodes<65><73>section<6F><6E><EFBFBD>)<29><>}<7D>(hhh]<5D>(h	<09>title<6C><65><EFBFBD>)<29><>}<7D>(h<05>Introduction<6F>h]<5D>h	<09>Text<78><74><EFBFBD><EFBFBD>Introduction<6F><6E><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<13>parent<6E>hh
h<03>source<63>N<EFBFBD>line<6E>Nuba<62>
attributes<EFBFBD>}<7D>(<28>ids<64>]<5D><>classes<65>]<5D><>names<65>]<5D><>dupnames<65>]<5D><>backrefs<66>]<5D>u<EFBFBD>tagname<6D>hhhh
hh<1C>k/tmp/tmp0jkcdajs/ae2a1ab225335b054e04f6baaf537184bb337a15/docs/programming-guide/chapter-1/introduction.rst<73>hKubh)<29><>}<7D>(hhh]<5D>(h)<29><>}<7D>(h<05>Motivations<6E>h]<5D>h<16>Motivations<6E><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh1hh/h
hhNhNubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hhh,h
hhh+hKubh	<09>	paragraph<70><68><EFBFBD>)<29><>}<7D>(hX<>Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing [SUTSKEVER2014]_ to computer vision [REDMON2016]_ to computational neuroscience [LEE2017]_. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) *layers*. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.<2E>h]<5D>(h<16><>Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>Over the past decade, Deep Neural Networks (DNNs) have emerged as an important class of Machine Learning (ML) models, capable of  achieving state-of-the-art performance across many domains ranging from natural language processing <20>hh?h
hhNhNubh<00>pending_xref<65><66><EFBFBD>)<29><>}<7D>(h<05>
SUTSKEVER2014<31>h]<5D>h	<09>inline<6E><65><EFBFBD>)<29><>}<7D>(hhLh]<5D>h<16>[SUTSKEVER2014]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhhPubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhhJubah}<7D>(h ]<5D><>id1<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69><6E>citation<6F><6E>reftype<70><65>ref<65><66>	reftarget<65>hL<68>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hK	hh?h
hubh<16> to computer vision <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> to computer vision <20>hh?h
hhNhNubhI)<29><>}<7D>(h<05>
REDMON2016<EFBFBD>h]<5D>hO)<29><>}<7D>(hhrh]<5D>h<16>[REDMON2016]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhhtubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhhpubah}<7D>(h ]<5D><>id2<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69>he<68>reftype<70>hg<68>	reftarget<65>hr<68>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hK	hh?h
hubh<16> to computational neuroscience <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> to computational neuroscience <20>hh?h
hhNhNubhI)<29><>}<7D>(h<05>LEE2017<31>h]<5D>hO)<29><>}<7D>(hh<>h]<5D>h<16>	[LEE2017]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhh<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhh<>ubah}<7D>(h ]<5D><>id3<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69>he<68>reftype<70>hg<68>	reftarget<65>h<EFBFBD><68>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hK	hh?h
hubh<16><>. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>. The strength of these models lies in their hierarchical structure, composed of a sequence of parametric (e.g., convolutional) and non-parametric (e.g., rectified linearity) <20>hh?h
hhNhNubh	<09>emphasis<69><73><EFBFBD>)<29><>}<7D>(h<05>*layers*<2A>h]<5D>h<16>layers<72><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhh<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h<>hh?ubh<16><>. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>. This pattern, though notoriously computationally expensive, also generates a large amount of highly parallelizable work particularly well suited for multi- and many- core processors.<2E>hh?h
hhNhNubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK	hh,h
hubh>)<29><>}<7D>(hX<>As a consequence, Graphics Processing Units (GPUs) have become a cheap and accessible resource for exploring and/or deploying novel research ideas in the field. This trend has been accelerated by the release of several frameworks for General-Purpose GPU (GPGPU) computing, such as CUDA and OpenCL, which have made the development of high-performance programs easier. Yet, GPUs remain incredibly challenging to optimize for locality and parallelism, especially for computations that cannot be efficiently implemented using a combination of pre-existing optimized primitives. To make matters worse, GPU architectures are also rapidly evolving and specializing, as evidenced by the addition of tensor cores to NVIDIA (and more recently AMD) micro-architectures.<2E>h]<5D>hX<>As a consequence, Graphics Processing Units (GPUs) have become a cheap and accessible resource for exploring and/or deploying novel research ideas in the field. This trend has been accelerated by the release of several frameworks for General-Purpose GPU (GPGPU) computing, such as CUDA and OpenCL, which have made the development of high-performance programs easier. Yet, GPUs remain incredibly challenging to optimize for locality and parallelism, especially for computations that cannot be efficiently implemented using a combination of pre-existing optimized primitives. To make matters worse, GPU architectures are also rapidly evolving and specializing, as evidenced by the addition of tensor cores to NVIDIA (and more recently AMD) micro-architectures.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hh<>hh<>h
hhNhNubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhh,h
hubh>)<29><>}<7D>(hX%This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial academic and industrial interest for Domain-Specific Languages (DSLs) and compilers. Regrettably, these systems -- whether they be based on  polyhedral machinery (*e.g.*, Tiramisu [BAGHDADI2021]_, Tensor Comprehensions [VASILACHE2018]_) or scheduling languages (*e.g.*, Halide [JRK2013]_, TVM [CHEN2018]_) -- remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like `cuBLAS <https://docs.nvidia.com/cuda/cublas/index.html>`_, `cuDNN <https://docs.nvidia.com/deeplearning/cudnn/api/index.html>`_ or `TensorRT <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html>`_.<2E>h]<5D>(hX1
This tension between the computational opportunities offered by DNNs and the practical difficulty of GPU programming has created substantial 
[CHEN2018]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjv
ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhjr
ubah}<7D>(h ]<5D><>id7<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69>he<68>reftype<70>hg<68>	reftarget<65>jt
<00>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hK
hh<>h
hubh<16><>) – remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>) -- remain less flexible and (for the same algorithm) markedly slower than the best handwritten compute kernels available in libraries like <20>hh<>h
hhNhNubh	<09>	reference<63><65><EFBFBD>)<29><>}<7D>(h<05>:`cuBLAS <https://docs.nvidia.com/cuda/cublas/index.html>`_<>h]<5D>h<16>cuBLAS<41><53><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>cuBLAS<41>hj<>
ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>name<6D>j<EFBFBD>
<00>refuri<72><69>.https://docs.nvidia.com/cuda/cublas/index.html<6D>uh*j<>
hh<>ubh	<09>target<65><74><EFBFBD>)<29><>}<7D>(h<05>1 <https://docs.nvidia.com/cuda/cublas/index.html><3E>h]<5D>h}<7D>(h ]<5D><>cublas<61>ah"]<5D>h$]<5D><>cublas<61>ah&]<5D>h(]<5D><>refuri<72>j<EFBFBD>
uh*j<>
<00>
referenced<EFBFBD>K
hh<>ubh<16>, <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>, <20>hh<>h
hhNhNubj<62>
)<29><>}<7D>(h<05>D`cuDNN <https://docs.nvidia.com/deeplearning/cudnn/api/index.html>`_<>h]<5D>h<16>cuDNN<4E><4E><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>cuDNN<4E>hj<>
ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>name<6D>j<EFBFBD>
j<>
<00>9https://docs.nvidia.com/deeplearning/cudnn/api/index.html<6D>uh*j<>
hh<>ubj<62>
)<29><>}<7D>(h<05>< <https://docs.nvidia.com/deeplearning/cudnn/api/index.html><3E>h]<5D>h}<7D>(h ]<5D><>cudnn<6E>ah"]<5D>h$]<5D><>cudnn<6E>ah&]<5D>h(]<5D><>refuri<72>j<EFBFBD>
uh*j<>
j<>
K
hh<>ubh<16> or <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05> or <20>hh<>h
hhNhNubj<62>
)<29><>}<7D>(h<05>V`TensorRT <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html>`_<>h]<5D>h<16>TensorRT<52><54><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>TensorRT<52>hj<>
ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>name<6D>j<EFBFBD>
j<>
<00>Hhttps://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html<6D>uh*j<>
hh<>ubj<62>
)<29><>}<7D>(h<05>K <https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html><3E>h]<5D>h}<7D>(h ]<5D><>tensorrt<72>ah"]<5D>h$]<5D><>tensorrt<72>ah&]<5D>h(]<5D><>refuri<72>j<EFBFBD>
uh*j<>
j<>
K
hh<>ubh<16>
.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>
.<2E>hh<>h
hhNhNubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK
hh,h
hubh>)<29><>}<7D>(hX<>
The main premise of this project is the following: programming paradigms based on blocked algorithms [LAM1991]_ can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional "Single Program, Multiple Data" (SPMD [AUGUIN1983]_) execution models for GPUs, and propose a variant in which programs -- rather than threads -- are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:<3A>h]<5D>(h<16>eThe main premise of this project is the following: programming paradigms based on blocked algorithms <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05>eThe main premise of this project is the following: programming paradigms based on blocked algorithms <20>hj	h
hhNhNubhI)<29><>}<7D>(h<05>LAM1991<39>h]<5D>hO)<29><>}<7D>(hjh]<5D>h<16>	[LAM1991]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhjubah}<7D>(h ]<5D><>id8<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69>he<68>reftype<70>hg<68>	reftarget<65>j<00>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hKhj	h
hubh<16><> can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional “Single Program, Multiple Data” (SPMD <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><> can facilitate the construction of high-performance compute kernels for neural networks.  We specifically revisit traditional "Single Program, Multiple Data" (SPMD <20>hj	h
hhNhNubhI)<29><>}<7D>(h<05>
AUGUIN1983<EFBFBD>h]<5D>hO)<29><>}<7D>(hj6h]<5D>h<16>[AUGUIN1983]<5D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj8ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hNhj4ubah}<7D>(h ]<5D><>id9<64>ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>	refdomain<69>he<68>reftype<70>hg<68>	reftarget<65>j6<00>refwarn<72><6E><EFBFBD>support_smartquotes<65><73>uh*hHhh+hKhj	h
hubh<16><>) execution models for GPUs, and propose a variant in which programs – rather than threads – are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:<3A><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>) execution models for GPUs, and propose a variant in which programs -- rather than threads -- are blocked. For example, in the case of matrix multiplication, CUDA and Triton differ as follows:<3A>hj	h
hhNhNubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhh,h
hubh	<09>table<6C><65><EFBFBD>)<29><>}<7D>(hhh]<5D>h	<09>tgroup<75><70><EFBFBD>)<29><>}<7D>(hhh]<5D>(h	<09>colspec<65><63><EFBFBD>)<29><>}<7D>(hhh]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>colwidth<74>K2uh*jfhjcubjg)<29><>}<7D>(hhh]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>colwidth<74>K2uh*jfhjcubh	<09>thead<61><64><EFBFBD>)<29><>}<7D>(hhh]<5D>h	<09>row<6F><77><EFBFBD>)<29><>}<7D>(hhh]<5D>(h	<09>entry<72><79><EFBFBD>)<29><>}<7D>(hhh]<5D>(h>)<29><>}<7D>(h<05>CUDA Programming Model<65>h]<5D>h<16>CUDA Programming Model<65><6C><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhj<>ubh>)<29><>}<7D>(h<05>!(Scalar Program, Blocked Threads)<29>h]<5D>h<16>!(Scalar Program, Blocked Threads)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhj<>ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubj<62>)<29><>}<7D>(hhh]<5D>(h>)<29><>}<7D>(h<05>Triton Programming Model<65>h]<5D>h<16>Triton Programming Model<65><6C><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhj<>ubh>)<29><>}<7D>(h<05>!(Blocked Program, Scalar Threads)<29>h]<5D>h<16>!(Blocked Program, Scalar Threads)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKhj<>ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj~ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j|hjcubh	<09>tbody<64><79><EFBFBD>)<29><>}<7D>(hhh]<5D>(j<>)<29><>}<7D>(hhh]<5D>(j<>)<29><>}<7D>(hhh]<5D>h	<09>
literal_block<63><6B><EFBFBD>)<29><>}<7D>(h<05><>#pragma parallel
for(int m = 0; i < M; m++)
#pragma parallel
for(int n = 0; j < N; n++){
  float acc = 0;
  for(int k = 0; k < K;k ++)
    acc += A[i, k]* B[k, j];

  C[i, j] = acc;
}<7D>h]<5D>h<16><>#pragma parallel
for(int m = 0; i < M; m++)
#pragma parallel
for(int n = 0; j < N; n++){
  float acc = 0;
  for(int k = 0; k < K;k ++)
    acc += A[i, k]* B[k, j];

  C[i, j] = acc;
}<7D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>	xml:space<63><65>preserve<76><65>force<63><65><EFBFBD>language<67><65>
C<><43>highlight_args<67>}<7D>uh*j<>hh+hKhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubj<62>)<29><>}<7D>(hhh]<5D>j<EFBFBD>)<29><>}<7D>(h<05><>#pragma parallel
for(int m = 0; m < M; m += MB)
#pragma parallel
for(int n = 0; n < N; n += NB){
  float acc[MB, NB] = 0;
  for(int k = 0; k < K; k += KB)
    acc +=  A[m:m+MB, k:k+KB]
          @ B[k:k+KB, n:n+NB];
  C[m:m+MB, n:n+NB] = acc;
}<7D>h]<5D>h<16><>#pragma parallel
for(int m = 0; m < M; m += MB)
#pragma parallel
for(int n = 0; n < N; n += NB){
  float acc[MB, NB] = 0;
  for(int k = 0; k < K; k += KB)
    acc +=  A[m:m+MB, k:k+KB]
          @ B[k:k+KB, n:n+NB];
  C[m:m+MB, n:n+NB] = acc;
}<7D><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj	ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>j<><00>j<EFBFBD>j<>j<>}<7D>uh*j<>hh+hKhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubj<62>)<29><>}<7D>(hhh]<5D>(j<>)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>|pic1|<7C>h]<5D>h	<09>image<67><65><EFBFBD>)<29><>}<7D>(h<05> image:: cuda-parallel-matmul.png<6E>h]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>alt<6C><74>pic1<63><31>uri<72><69>4programming-guide/chapter-1/cuda-parallel-matmul.png<6E><67>
candidates<EFBFBD>}<7D><>
*<2A>j=suh*j.hh+hK-hj*ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK)hj'ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj$ubj<62>)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>|pic2|<7C>h]<5D>j/)<29><>}<7D>(h<05>"image:: triton-parallel-matmul.png<6E>h]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>alt<6C><74>pic2<63><32>uri<72><69>6programming-guide/chapter-1/triton-parallel-matmul.png<6E>j>}<7D>j@jasuh*j.hh+hK/hjPubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK)hjMubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj$ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hj<>ubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j<>hjcubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>cols<6C>Kuh*jahj^ubah}<7D>(h ]<5D>h"]<5D><>colwidths-given<65>ah$]<5D>h&]<5D>h(]<5D>uh*j\hh,h
hhh+hNubh	<09>substitution_definition<6F><6E><EFBFBD>)<29><>}<7D>(h<05>+.. |pic1| image:: cuda-parallel-matmul.png
<EFBFBD>h]<5D>j/)<29><>}<7D>(hj2h]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>alt<6C>j;<00>uri<72><69>4programming-guide/chapter-1/cuda-parallel-matmul.png<6E>j>}<7D>j@j<>suh*j.hj<>hh+hK-ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>j;ah&]<5D>h(]<5D>uh*j<>hh+hK-hh,h
hubj<62>)<29><>}<7D>(h<05>-.. |pic2| image:: triton-parallel-matmul.png
<EFBFBD>h]<5D>j/)<29><>}<7D>(hjVh]<5D>h}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>alt<6C>j_<00>uri<72><69>6programming-guide/chapter-1/triton-parallel-matmul.png<6E>j>}<7D>j@j<>suh*j.hj<>hh+hK/ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>j_ah&]<5D>h(]<5D>uh*j<>hh+hK/hh,h
hubh>)<29><>}<7D>(hX
A key benefit of this approach is that it leads to block-structured iteration spaces that offer programmers more flexibility than existing DSLs when implementing sparse operations, all while allowing compilers to aggressively optimize programs for data locality and parallelism.<2E>h]<5D>hX
A key benefit of this approach is that it leads to block-structured iteration spaces that offer programmers more flexibility than existing DSLs when implementing sparse operations, all while allowing compilers to aggressively optimize programs for data locality and parallelism.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>h
hhNhNubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK1hh,h
hubeh}<7D>(h ]<5D><>motivations<6E>ah"]<5D>h$]<5D><>motivations<6E>ah&]<5D>h(]<5D>uh*h
hhh
hhh+hKubh)<29><>}<7D>(hhh]<5D>(h)<29><>}<7D>(h<05>
Challenges<EFBFBD>h]<5D>h<16>
Challenges<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>h
hhNhNubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hhj<>h
hhh+hK5ubh>)<29><>}<7D>(hX`The main challenge posed by our proposed paradigm is that of work scheduling, i.e., how the work done by each program instance should be partitioned for efficient execution on modern GPUs. To address this issue, the Triton compiler makes heavy use of *block-level data-flow analysis*, a technique for scheduling iteration blocks statically based on the control- and data-flow structure of the target program. The resulting system actually works surprisingly well: our compiler manages to apply a broad range of interesting optimization automatically (e.g., automatic coalescing, thread swizzling, pre-fetching, automatic vectorization, tensor core-aware instruction selection, shared memory allocation/synchronization, asynchronous copy scheduling). Of course doing all this is not trivial; one of the purposes of this guide is to give you a sense of how it works.<2E>h]<5D>(h<16><>The main challenge posed by our proposed paradigm is that of work scheduling, i.e., how the work done by each program instance should be partitioned for efficient execution on modern GPUs. To address this issue, the Triton compiler makes heavy use of <20><><EFBFBD><EFBFBD><EFBFBD>}<7D>(h<05><>The main challenge posed by our proposed paradigm is that of work scheduling, i.e., how the work done by each program instance should be partitioned for efficient execution on modern GPUs. To address this issue, the Triton compiler makes heavy use of <20>hj<>h
hhNhNubh<62>)<29><>}<7D>(h<05> *block-level data-flow analysis*<2A>h]<5D>h<16>block-level data-flow analysis<69><73><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h<>hj<>ubhXE, a technique for scheduling iteration blocks statically based on the control- and data-flow structure of the target program. The resulting system actually works surprisingly well: our compiler manages to apply a broad range of interesting optimization automatically (e.g., automatic coalescing, thread swizzling, pre-fetching, automatic vectorization, tensor core-aware instruction selection, shared memory allocation/synchronization, asynchronous copy scheduling). Of course doing all this is not trivial; one of the purposes of this guide is to give you a sense of how it works.<2E><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hXE, a technique for scheduling iteration blocks statically based on the control- and data-flow structure of the target program. The resulting system actually works surprisingly well: our compiler manages to apply a broad range of interesting optimization automatically (e.g., automatic coalescing, thread swizzling, pre-fetching, automatic vectorization, tensor core-aware instruction selection, shared memory allocation/synchronization, asynchronous copy scheduling). Of course doing all this is not trivial; one of the purposes of this guide is to give you a sense of how it works.<2E>hj<>h
hhNhNubeh}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK7hj<>h
hubeh}<7D>(h ]<5D><>
challenges<EFBFBD>ah"]<5D>h$]<5D><>
challenges<EFBFBD>ah&]<5D>h(]<5D>uh*h
hhh
hhh+hK5ubh)<29><>}<7D>(hhh]<5D>(h)<29><>}<7D>(h<05>
References<EFBFBD>h]<5D>h<16>
References<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjhj
h
hhNhNubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*hhj
h
hhh+hK;ubh	he<68><65>)<29><>}<7D>(h<05>TI. Sutskever et al., "Sequence to Sequence Learning with Neural Networks", NIPS 2014<31>h]<5D>(h	<09>label<65><6C><EFBFBD>)<29><>}<7D>(h<05>
SUTSKEVER2014<31>h]<5D>h<16>
SUTSKEVER2014<31><34><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj"ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>support_smartquotes<65><73>uh*j hjubh	<09>enumerated_list<73><74><EFBFBD>)<29><>}<7D>(hhh]<5D>h	<09>	list_item<65><6D><EFBFBD>)<29><>}<7D>(h<05>QSutskever et al., "Sequence to Sequence Learning with Neural Networks", NIPS 2014<31>h]<5D>h>)<29><>}<7D>(hj:h]<5D>h<16>USutskever et al., “Sequence to Sequence Learning with Neural Networks”, NIPS 2014<31><34><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj:hj<ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK=hj8ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj3ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>enumtype<70><65>
upperroman<EFBFBD><EFBFBD>prefix<69>h<06>suffix<69>juh*j1hjubeh}<7D>(h ]<5D><>
sutskever2014<31>ah"]<5D>h$]<5D><>
sutskever2014<31>ah&]<5D>h(]<5D>h_a<5F>docname<6D><65>(programming-guide/chapter-1/introduction<6F>uh*hehh+hK=hj
h
h<03>resolved<65>K
ubj)<29><>}<7D>(h<05>VJ. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2016<31>h]<5D>(j!)<29><>}<7D>(h<05>
REDMON2016<EFBFBD>h]<5D>h<16>
REDMON2016<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjhubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hjdubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>SRedmon et al., "You Only Look Once: Unified, Real-Time Object Detection", CVPR 2016<31>h]<5D>h>)<29><>}<7D>(hj{h]<5D>h<16>WRedmon et al., “You Only Look Once: Unified, Real-Time Object Detection”, CVPR 2016<31><36><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj{hj}ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK>hjyubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hjvubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jU<00>
upperalpha<EFBFBD>jWhjXj<00>start<72>K
uh*j1hjdubeh}<7D>(h ]<5D><>
redmon2016<EFBFBD>ah"]<5D>h$]<5D><>
redmon2016<EFBFBD>ah&]<5D>h(]<5D>h<EFBFBD>ajajbuh*hehh+hK>hj
h
hjcK
ubj)<29><>}<7D>(h<05>VK. Lee et al., "Superhuman Accuracy on the SNEMI3D Connectomics Challenge", ArXiV 2017<31>h]<5D>(j!)<29><>}<7D>(h<05>LEE2017<31>h]<5D>h<16>LEE2017<31><37><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hj<>ubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>SLee et al., "Superhuman Accuracy on the SNEMI3D Connectomics Challenge", ArXiV 2017<31>h]<5D>h>)<29><>}<7D>(hj<>h]<5D>h<16>WLee et al., “Superhuman Accuracy on the SNEMI3D Connectomics Challenge”, ArXiV 2017<31><37><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK?hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>Kuh*j1hj<>ubeh}<7D>(h ]<5D><>lee2017<31>ah"]<5D>h$]<5D><>lee2017<31>ah&]<5D>h(]<5D>h<EFBFBD>ajajbuh*hehh+hK?hj
h
hjcK
ubj)<29><>}<7D>(h<05>eR. Baghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021<32>h]<5D>(j!)<29><>}<7D>(h<05>BAGHDADI2021<32>h]<5D>h<16>BAGHDADI2021<32><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hj<>ubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>bBaghdadi et al., "Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code", CGO 2021<32>h]<5D>h>)<29><>}<7D>(hj<>h]<5D>h<16>fBaghdadi et al., “Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code”, CGO 2021<32><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hK@hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>Kuh*j1hj<>ubeh}<7D>(h ]<5D><>baghdadi2021<32>ah"]<5D>h$]<5D><>baghdadi2021<32>ah&]<5D>h(]<5D>j
ajajbuh*hehh+hK@hj
h
hjcK
ubj)<29><>}<7D>(h<05>{N. Vasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018<31>h]<5D>(j!)<29><>}<7D>(h<05>
VASILACHE2018<31>h]<5D>h<16>
VASILACHE2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hjubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>xVasilache et al., "Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions", ArXiV 2018<31>h]<5D>h>)<29><>}<7D>(hj+h]<5D>h<16>|Vasilache et al., “Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, ArXiV 2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj+hj-ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKAhj)ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj&ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>Kuh*j1hjubeh}<7D>(h ]<5D><>
vasilache2018<31>ah"]<5D>h$]<5D><>
vasilache2018<31>ah&]<5D>h(]<5D>j.
ajajbuh*hehh+hKAhj
h
hjcK
ubj)<29><>}<7D>(h<05><>J. Ragan-Kelley et al., "Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines", PLDI 2013<31>h]<5D>(j!)<29><>}<7D>(h<05>JRK2013<31>h]<5D>h<16>JRK2013<31><33><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjRubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hjNubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05><>Ragan-Kelley et al., "Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines", PLDI 2013<31>h]<5D>h>)<29><>}<7D>(hjeh]<5D>h<16><>Ragan-Kelley et al., “Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines”, PLDI 2013<31><33><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjehjgubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKBhjcubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj`ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>K
uh*j1hjNubeh}<7D>(h ]<5D><>jrk2013<31>ah"]<5D>h$]<5D><>jrk2013<31>ah&]<5D>h(]<5D>jc
ajajbuh*hehh+hKBhj
h
hjcK
ubj)<29><>}<7D>(h<05>_T. Chen et al., "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning", OSDI 2018<31>h]<5D>(j!)<29><>}<7D>(h<05>CHEN2018<31>h]<5D>h<16>CHEN2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hj<>ubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>\Chen et al., "TVM: An Automated End-to-End Optimizing Compiler for Deep Learning", OSDI 2018<31>h]<5D>h>)<29><>}<7D>(hj<>h]<5D>h<16>`Chen et al., “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning”, OSDI 2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKChj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>Kuh*j1hj<>ubeh}<7D>(h ]<5D><>chen2018<31>ah"]<5D>h$]<5D><>chen2018<31>ah&]<5D>h(]<5D>j<EFBFBD>
ajajbuh*hehh+hKChj
h
hjcK
ubj)<29><>}<7D>(h<05>[M. Lam et al., "The Cache Performance and Optimizations of Blocked Algorithms", ASPLOS 1991<39>h]<5D>(j!)<29><>}<7D>(h<05>LAM1991<39>h]<5D>h<16>LAM1991<39><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hj<>ubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>XLam et al., "The Cache Performance and Optimizations of Blocked Algorithms", ASPLOS 1991<39>h]<5D>h>)<29><>}<7D>(hj<>h]<5D>h<16>\Lam et al., “The Cache Performance and Optimizations of Blocked Algorithms”, ASPLOS 1991<39><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hj<>hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKDhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>K
uh*j1hj<>ubeh}<7D>(h ]<5D><>lam1991<39>ah"]<5D>h$]<5D><>lam1991<39>ah&]<5D>h(]<5D>j%ajajbuh*hehh+hKDhj
h
hjcK
ubj)<29><>}<7D>(h<05>iM. Auguin et al., "Opsila: an advanced SIMD for numerical analysis and signal processing", EUROMICRO 1983<38>h]<5D>(j!)<29><>}<7D>(h<05>
AUGUIN1983<EFBFBD>h]<5D>h<16>
AUGUIN1983<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>j0<00>uh*j hj<>ubj2)<29><>}<7D>(hhh]<5D>j7)<29><>}<7D>(h<05>fAuguin et al., "Opsila: an advanced SIMD for numerical analysis and signal processing", EUROMICRO 1983<38>h]<5D>h>)<29><>}<7D>(hjh]<5D>h<16>jAuguin et al., “Opsila: an advanced SIMD for numerical analysis and signal processing”, EUROMICRO 1983<38><33><EFBFBD><EFBFBD><EFBFBD>}<7D>(hjhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hh+hKEhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*j6hjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>jUj<>jWhjXjj<>K
uh*j1hj<>ubeh}<7D>(h ]<5D><>
auguin1983<EFBFBD>ah"]<5D>h$]<5D><>
auguin1983<EFBFBD>ah&]<5D>h(]<5D>jGajajbuh*hehh+hKEhj
h
hjcK
ubeh}<7D>(h ]<5D><>
references<EFBFBD>ah"]<5D>h$]<5D><>
references<EFBFBD>ah&]<5D>h(]<5D>uh*h
hhh
hhh+hK;ubeh}<7D>(h ]<5D><>introduction<6F>ah"]<5D>h$]<5D><>introduction<6F>ah&]<5D>h(]<5D>uh*h
hhh
hhh+hKubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>source<63>h+uh*h
<01>current_source<63>N<EFBFBD>current_line<6E>N<EFBFBD>settings<67><73>docutils.frontend<6E><64>Values<65><73><EFBFBD>)<29><>}<7D>(hN<>	generator<6F>N<EFBFBD>	datestamp<6D>N<EFBFBD>source_link<6E>N<EFBFBD>
source_url<EFBFBD>N<EFBFBD>
toc_backlinks<6B>j<EFBFBD><00>footnote_backlinks<6B>K
<01>
sectnum_xform<72>K
<01>strip_comments<74>N<EFBFBD>strip_elements_with_classes<65>N<EFBFBD>
strip_classes<65>N<EFBFBD>report_level<65>K<02>
halt_level<EFBFBD>K<05>exit_status_level<65>K<05>debug<75>N<EFBFBD>warning_stream<61>N<EFBFBD>	traceback<63><6B><EFBFBD>input_encoding<6E><67>	utf-8-sig<69><67>input_encoding_error_handler<65><72>strict<63><74>output_encoding<6E><67>utf-8<><38>output_encoding_error_handler<65>jh<00>error_encoding<6E><67>utf-8<><38>error_encoding_error_handler<65><72>backslashreplace<63><65>
language_code<64><65>en<65><6E>record_dependencies<65>N<EFBFBD>config<69>N<EFBFBD>	id_prefix<69>h<06>auto_id_prefix<69><78>id<69><64>
dump_settings<67>N<EFBFBD>dump_internals<6C>N<EFBFBD>dump_transforms<6D>N<EFBFBD>dump_pseudo_xml<6D>N<EFBFBD>expose_internals<6C>N<EFBFBD>strict_visitor<6F>N<EFBFBD>_disable_config<69>N<EFBFBD>_source<63>h+<2B>_destination<6F>N<EFBFBD>
_config_files<65>]<5D><>pep_references<65>N<EFBFBD>pep_base_url<72><6C> https://www.python.org/dev/peps/<2F><>pep_file_url_template<74><65>pep-%04d<34><64>rfc_references<65>N<EFBFBD>rfc_base_url<72><6C>https://tools.ietf.org/html/<2F><>	tab_width<74>K<08>trim_footnote_reference_space<63><65><EFBFBD>file_insertion_enabled<65><64><EFBFBD>raw_enabled<65>K
<01>syntax_highlight<68><74>long<6E><67>smart_quotes<65><73><EFBFBD>smartquotes_locales<65>]<5D><>character_level_inline_markup<75><70><EFBFBD>doctitle_xform<72><6D><EFBFBD>
docinfo_xform<72>K
<01>sectsubtitle_xform<72><6D><EFBFBD>embed_stylesheet<65><74><EFBFBD>cloak_email_addresses<65><73><EFBFBD>env<6E>Nub<75>reporter<65>N<EFBFBD>indirect_targets<74>]<5D><>substitution_defs<66>}<7D>(j;j<>j_j<>u<>substitution_names<65>}<7D>(<28>pic1<63>j;<00>pic2<63>j_u<>refnames<65>}<7D>(<28>
sutskever2014<31>]<5D>h	<09>citation_reference<63><65><EFBFBD>)<29><>}<7D>(h<05>[SUTSKEVER2014]_<>h]<5D>h<16>
SUTSKEVER2014<31><34><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h_ah"]<5D>h$]<5D>h&]<5D>h(]<5D><>refid<69>j[uh*j<>hh?jcK
uba<62>
redmon2016<EFBFBD>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>
[REDMON2016]_<>h]<5D>h<16>
REDMON2016<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h<EFBFBD>ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>uh*j<>hh?jcK
uba<62>lee2017<31>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>
[LEE2017]_<>h]<5D>h<16>LEE2017<31><37><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h<EFBFBD>ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>uh*j<>hh?jcK
uba<62>baghdadi2021<32>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>[BAGHDADI2021]_<>h]<5D>h<16>BAGHDADI2021<32><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>j
ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>juh*j<>hh<>jcK
uba<62>
vasilache2018<31>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>[VASILACHE2018]_<>h]<5D>h<16>
VASILACHE2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>j.
ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>jHuh*j<>hh<>jcK
uba<62>jrk2013<31>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>
[JRK2013]_<>h]<5D>h<16>JRK2013<31><33><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>jc
ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>uh*j<>hh<>jcK
uba<62>chen2018<31>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>[CHEN2018]_<>h]<5D>h<16>CHEN2018<31><38><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj
ubah}<7D>(h ]<5D>j<EFBFBD>
ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>uh*j<>hh<>jcK
uba<62>lam1991<39>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>
[LAM1991]_<>h]<5D>h<16>LAM1991<39><31><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h ]<5D>j%ah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j<>uh*j<>hj	jcK
uba<62>
auguin1983<EFBFBD>]<5D>j<EFBFBD>)<29><>}<7D>(h<05>
[AUGUIN1983]_<>h]<5D>h<16>
AUGUIN1983<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj*ubah}<7D>(h ]<5D>jGah"]<5D>h$]<5D>h&]<5D>h(]<5D>j<EFBFBD>j0uh*j<>hj	jcK
ubau<61>refids<64>}<7D><>nameids<64>}<7D>(jCj@j<>j<>j<>
j<>
j<>
j<>
j<>
j<>
jjj;j8j^j[j<>j<>j<>j<>jjjKjHj<>j<>j<>j<>j<>j<>j3j0u<>	nametypes<65>}<7D>(jCNj<4E>Nj<4E>
<00>j<EFBFBD>
<00>j<EFBFBD>
<00>jNj;Nj^<00>j<EFBFBD><00>j<EFBFBD><00>j<00>jK<00>j<EFBFBD><00>j<EFBFBD><00>j<EFBFBD><00>j3<00>uh }<7D>(j@hj<>h,h_j<5F>h<>j<EFBFBD>h<>j<EFBFBD>j
j<>j.
j<>jc
j<>j<>
j
j<>
j<>
j<>
j<>
j<>
j<>
j%jjGj*jj<>j8j
j[jj<>jdj<>j<>jj<>jHjj<>jNj<>j<>j<>j<>j0j<>u<>
footnote_refs<66>}<7D><>
citation_refs<66>}<7D>(j<>]<5D>j<EFBFBD>aj<61>]<5D>j<EFBFBD>aj<61>]<5D>j<EFBFBD>aj<61>]<5D>j<EFBFBD>aj<61>]<5D>j<EFBFBD>aj<61>]<5D>j<EFBFBD>aj]<5D>j
aj]<5D>jaj(]<5D>j*au<61>
autofootnotes<65>]<5D><>autofootnote_refs<66>]<5D><>symbol_footnotes<65>]<5D><>symbol_footnote_refs<66>]<5D><>	footnotes<65>]<5D><>	citations<6E>]<5D>(jjdj<>j<>jjNj<>j<>j<>e<>autofootnote_start<72>K
<01>symbol_footnote_start<72>K<00>
id_counter<EFBFBD><EFBFBD>collections<6E><73>Counter<65><72><EFBFBD>}<7D>jvK	s<><73>R<EFBFBD><52>parse_messages<65>]<5D>(h	<09>system_message<67><65><EFBFBD>)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "J" (ordinal 10)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “J” (ordinal 10)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjhubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hjeubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70><65>INFO<46><4F>source<63>h+<2B>line<6E>Kuh*jchjdubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "K" (ordinal 11)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “K” (ordinal 11)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchj<>ubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "R" (ordinal 18)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “R” (ordinal 18)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchj<>ubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "N" (ordinal 14)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “N” (ordinal 14)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchjubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "J" (ordinal 10)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “J” (ordinal 10)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchjNubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "T" (ordinal 20)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “T” (ordinal 20)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj<>ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchj<>ubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "M" (ordinal 13)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “M” (ordinal 13)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hjubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchj<>ubjd)<29><>}<7D>(hhh]<5D>h>)<29><>}<7D>(h<05>;Enumerated list start value not ordinal-1: "M" (ordinal 13)<29>h]<5D>h<16>?Enumerated list start value not ordinal-1: “M” (ordinal 13)<29><><EFBFBD><EFBFBD><EFBFBD>}<7D>(hhhj&ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D>uh*h=hj#ubah}<7D>(h ]<5D>h"]<5D>h$]<5D>h&]<5D>h(]<5D><>level<65>K
<01>type<70>j~<00>source<63>h+<2B>line<6E>Kuh*jchj<>ube<62>transform_messages<65>]<5D><>transformer<65>N<EFBFBD>
decoration<EFBFBD>Nh
hub.