2018-11-07 17:19:42 -08:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "Ynb-laSwmpac"
},
"source": [
2018-11-07 17:20:52 -08:00
"# Loading and visualizing results ([open in colab](https://colab.research.google.com/github/openai/baselines/blob/master/docs/viz/viz.ipynb))\n",
2018-11-07 17:19:42 -08:00
"In order to compare performance of algorithms, we often would like to visualize learning curves (reward as a function of time steps), or some other auxiliary information about learning aggregated into a plot. Baselines repo provides tools for doing so in several different ways, depending on the goal."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "yreoV7OClzYG"
},
"source": [
"## Preliminaries / TensorBoard\n",
"First, let us install baselines repo from github"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "r4Aul2Qujlg9"
},
"outputs": [],
"source": [
"!pip install git+https://github.com/openai/baselines > ~/pip_install_baselines.log"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1n7XAyVWniRp"
},
"source": [
"For all algorithms in baselines summary data is saved into a folder defined by logger. By default, a folder $TMPDIR/openai-<date>-<time> is used; you can see the location of logger directory at the beginning of the training in the message like this:\n",
"\n",
"Logging to /var/folders/mq/tgrn7bs17s1fnhlwt314b2fm0000gn/T/openai-2018-10-29-15-03-13-537078\n",
"The location can be changed by changing OPENAI_LOGDIR environment variable. For instance, "
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 68
},
"colab_type": "code",
"executionInfo": {
"elapsed": 32433,
"status": "ok",
"timestamp": 1541626389411,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "xzqFBYiZjtUr",
"outputId": "92dc8be5-c18c-4399-b2d0-0f444b321a13"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n"
]
}
],
"source": [
"!OPENAI_LOGDIR=$HOME/logs/cartpole-ppo OPENAI_LOG_FORMAT=csv python -m baselines.run --alg=ppo2 --env=CartPole-v0 --num_timesteps=30000 --nsteps=128"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "WYwTBFMCn95w"
},
"source": [
"Notice also the usage of `OPENAI_LOG_FORMAT` environment variable - this allows us to suppress printing to stdout. \n",
"Permissible values for `OPENAI_LOG_FORMAT` environment variables are `stdout`, `log`, `csv`, and `tensorboard` (multiple values can be comma-separated). \n",
"The latter one (`tensorboard`) dumps values in tensorboard format, which allows for their viewing in [TensorBoard](https://www.tensorflow.org/guide/summaries_and_tensorboard). TensorBoard provides nice visualization and in many cases is the easiest way to look at the summary of the results. However, if tensorboard visualization is not sufficient, read on...\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "QcWDWCfIojHy"
},
"source": [
"## Loading results and plotting using matplotlib\n",
"Bа
"For instance, the following:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 71
},
"colab_type": "code",
"executionInfo": {
"elapsed": 361,
"status": "ok",
"timestamp": 1541626398698,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "cDdZjFR2kFG2",
"outputId": "09d91a7a-f851-459b-903a-f8c1d32ccb8e"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.6/dist-packages/baselines/bench/monitor.py:164: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access\n",
" df.headers = headers # HACK to preserve backwards compatibility\n"
]
}
],
"source": [
"from baselines.common import plot_util as pu\n",
"results = pu.load_results('~/logs/cartpole-ppo') "
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "sj3jfhbBpGPq"
},
"source": [
"will search for all folders with baselines-compatible results in ~/logs/cartpole-ppo and subfolders and return a list of `Result` objects. Each `Result` object is a named tuple with the following fields:\n",
"\n",
" * dirname: str - name of the folder from which data was loaded\n",
" \n",
" * metadata: dict) - dictionary with various metadata (read from metadata.json file)\n",
"\n",
" * progress: pandas.DataFrame - tabular data saved by logger as a pandas dataframe. Available if csv is in logger formats.\n",
"\n",
" * monitor: pandas.DataFrame - raw episode data (length, episode reward, timestamp). Available if environment wrapped with Monitor wrapper\n",
" \n",
"Thus, a learning curve from a single run can be plotted as follows (note the cumulative sum trick to get convert lengths of the episodes into number of time steps taken so far)\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 364
},
"colab_type": "code",
"executionInfo": {
"elapsed": 635,
"status": "ok",
"timestamp": 1541626401733,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "jEBxUzvVsNbP",
"outputId": "7b7016a5-6f94-4384-821d-a9e197211315"
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7f4d4bba0518>]"
]
},
"execution_count": 21,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAFKCAYAAADMuCxnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvWmUJUd1LvplnnNq6q7urm5VY1he\n8HhcsLFbgLVkX8QyfovJkv0e0wWBzZNY+GktcxeyBbauBXjZD97T8wUBCzPpXoFsEIhZ8iSbQcIY\nDAZNqEFIaGgJgYaW1F3VVdU11zmZGe9HZkTuiNyRwzmZ51S14vtTdXKIjIzMjB1772/v7QkhBBwc\nHBwcHBxGCn/UHXBwcHBwcHBwAtnBwcHBwWFbwAlkBwcHBweHbQAnkB0cHBwcHLYBnEB2cHBwcHDY\nBnAC2cHBwcHBYRugPcqLz82t1NrezMwUFhfXa21zJ8ONhw43HjrceOhw46HDjYeOusZjdnbauu+U\n0pDb7daou7Ct4MZDhxsPHW48dLjx0OHGQ8cwxuOUEsgODg4ODg47FU4gOzg4ODg4bAM4gezg4ODg\n4LAN4ASyg4ODg4PDNoATyA4ODg4ODtsATiA7ODg4ODhsAziB7ODg4ODgsA3gBLKDg4ODg8M2QKlM\nXe973/tw2223IQgCvPnNb8bpp5+OSy65BGEYYnZ2Fu9///sxNjaG6667Dp/+9Kfh+z5e97rX4dxz\nz226/w4ODg4ODqcECgXyTTfdhPvuuw9f+tKXsLi4iFe/+tU466yz8IY3vAG/8zu/gw9+8IO49tpr\n8apXvQqXX345rr32WnQ6Hbz2ta/Fy172Muzbt28Y9+Hg4ODg4LCjUWiy/vVf/3V8+MMfBgDs2bMH\nGxsbuPnmm/GSl7wEAPCiF70IN954I26//XacfvrpmJ6exsTEBM444wwcPny42d47OGwTCCFwy93H\nsLiyVWlfFZxc3cLNdx2DEGKgdih6QYjv3/kYNrtBbW061IefPbaMIw8vjbobT0jc98gSfvbY8lCv\nWaght1otTE1NAQCuvfZa/NZv/Rb+4z/+A2NjYwCAAwcOYG5uDvPz89i/f786b//+/Zibm8tte2Zm\nqvb8oHmJu5+IcOOho6nxeHRuFVf800/wqv/tGbjgFYe0fUdz9lXBV25+CH/3rfvxvGc/Cb9Y0308\nOL+Bv/mXu/HW14/jpb/xtFra3MnYbt/L//eZ27Cy0cUn/+K3a287igQ8D/A8z3rMdhuPYeIdn7gJ\nU+NtfPS/vUhta3o8Sld7+td//Vdce+21+OQnP4nf/u305bCt1sus4uuuJDI7O117BamdDDceOpoc\nj8eOxe0ur2xmrnE0WWWfZPZVwYmlDQDAseMrGLfPoaUxOzuNuROrAID5hfUn/LuyHb+X1Y0uNreC\nRvr1V1f/AE85sAt/8LvPZvdvx/EYJtY3evCEUGNQ13gMXO3pu9/9Lq644gpceeWVmJ6extTUFDY3\nNwEAx44dw8GDB3Hw4EHMz8+rc44fP46DBw8O2HUHh52BKFmAcsvQXhDBurMCgjAarAEGURR3KqrR\nDO5QH6JIoKlH87NHV/Dw8dVmGj8FIL+NYaJQIK+srOB973sfPv7xjyuC1gte8AJcf/31AIAbbrgB\nL3zhC/Hc5z4Xd9xxB5aXl7G2tobDhw/jzDPPbLb3Dg7bBJGUlcw33EsE6aCfdxgmQr/GeUItJJw8\n3pYQQtTKGZAIwsgtwgoQjmB8Ck3WX/3qV7G4uIi3ve1tatt73/te/MVf/AW+9KUv4SlPeQpe9apX\nodPp4OKLL8YFF1wAz/Nw4YUXYnr6iet/cHhiIW9yC5SGPNgHHkb1a8iyS6PQBhyKEYlmFkvKauNg\nhYjEwIvoqigUyK9//evx+te/PrP9U5/6VGbbOeecg3POOaeenjk47CBIgcZ9wEHNGnKdSE3tTiBv\nR0SimSfTdQK5EOF2NFk7ODgUI8+sqEzW29CHLKQP2WnI2xIiasZk3euFcfu1t3zqYFv6kB0cHIqh\nPl5m8kzNg4N94EEDE0ROtx22ASLRjNB0GnI+lGViyN+FE8gODjVACTZmX1CTqTl0LOsnHKKmNGQn\nkHMxKouRE8gODjWgDKlrYJN1IxqyY1lvZwg0oyJ3g7D+Rk8hjMJ/DDiB7OBQCxSpq9GwJ9lOfZOF\nY1lvb0RRan2pE85knQ+nITs47GCUC3sa7BqOZf3EQ/x8miB11fNOnqoIo9F8F6VTZzo4ONiRhggz\npK6aNNtGTNaKZV170w41oClNzZms8+E0ZAeHHQyR44utK3VmI6Qu1W+nKm1HRKKZ1JmO1JWPUZEc\nnUB2cKgBuSbrmnzIQROpMx3LettCJMK4CbOp8yHnw2nIDg47GKUydQ3Msm5CQ47/Onm8/SAfSSMa\nskoM4h48hzCHpNkknEB2cKgBeRpmL1BT60DXaILUJU3VjmW9/UCfSd0uBach58NpyA4OOxhDqfbU\niIbs4pC3K6gQrvvxOIGcDxeH7OCwg5EXPrSdw55E0jXnQ95+0NZfNT+enmNZ58JpyA4OOxjDJHXV\nCcey3r6g71TdCyanIefDacgODjsYIieZdU+lzuz/I4+EaESLdSzr7YsmF0kuMUg+XNiTg8MORt6C\nuo6yidRcXedcoVJnuol524E+k/pJXc5knQenITs47GDkhT31ahDITdRCBojJ2knkbQedZV1v2y4x\nSD7yctM3CSeQHRxqQB5buVdDtaemVuyy304ebz9Qs2ndgsH5kPPhSF0ODjsYTZO6aNrMOpM5pJqA\nk8jbDZqGXLOzVyYGceAROh+yg8PORTp5MmFP0v87wEfeBMMaoBqyE8jbDfSRNKUhu6fOw2nIDg47\nGHnfb6+Gya+JpCBAGus6KpP13NIG7v75wmguvs3RpMna+ZDzEeYssJuEE8gODjVAlCF1DfBtN6Uh\nS1PoqEzWX/7W/fjgl293AoKBJpBrFgyOZZ0PpyE7OOxg5PqQa9CQNZZ1E9WeRjQBbWwFCCPhcmkz\naJJl7Uhd+XAC2cFhB0MJZJHdHtZAnGqOZR3/HZULWcZXu6pDWTQZhywTgzjqAA8Xh+zgsIMRWazS\nQU2aSBN5rIHU1D4qUlcTJSVPFQiNZV0vnIacjzQ3/XDhBLKDQw2wCTRqah5E5mnt9N9MBqOu9tTU\nQuNUgPZO1emmEKKxRDOnCkZlsm6XOejIkSN4y1vegje96U0477zzcNFFF2FxcREAsLS0hOc973l4\n85vfjJe//OU4dOgQAGBmZgYf+chHmuu5g8M2gi2et1eTwGnKhJamzhzNBDRIIfhb7j6GR+ZW8V9+\n6xk192p7QDRksnYEumKMymRdKJDX19dx6aWX4qyzzlLbqKB95zvfiXPPPRcA8PSnPx1XX311A910\ncNjesAk0WuZukEm18dSZIxbI/eBbh4/i3oeX8Krf/F/h+16Nvdoe0FnW9cEJ5GJs29SZY2NjuPLK\nK3Hw4MHMvgceeAArKyt4znOe00jnHBx2CoRljqsrXKkxUteIWdbhAAuNcMT+76bRFMu6q2XpOjXH\nblBsW1JXu93GxMQEu+8zn/kMzjvvPPV7fn4eF110EX7v934P1113XX29dHDY5rD5Yimpa1uGPY3a\nhzzAxCf7PqrJs2noiUGcyXqY2NY+ZA7dbhe33XYb3v3udwMA9u3bh7e+9a14xStegZWVFZx77rl4\n/vOfz2rWEjMzU2i3W/12gcXs7HSt7e10uPHQ0dR4jI3Hn9L4eFu7xtJmkB4z1u77+lMPLan/9+2b\nqu0+5PfXavsjeVfktHfgwG7smuxUOleaqffv34WpiWrn2rCdvpfjK131/8z+XZidmaql3dVeKpDb\n7VbuPW+n8RgmJqfGAAAt39PGoOnx6Fsg33rrrZqpevfu3XjNa14DANi/fz8OHTqEBx54IFcgLy6u\n93t5FrOz05ibW6m1zZ0
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4bbe1b00>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"r = results[0]\n",
"plt.plot(np.cumsum(r.monitor.l), r.monitor.r)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "lLGJgLFjqLDB"
},
"source": [
"The raw learning curve from a single run can be very noisy. To smoothen it and analyze if any learning actually takes place, we can use `plot_util.smooth()` function:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 364
},
"colab_type": "code",
"executionInfo": {
"elapsed": 514,
"status": "ok",
"timestamp": 1541626405684,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "Rz4vAnURuURX",
"outputId": "f5e38cd7-c63c-4b9c-aaa0-b24592b636ba"
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7f4d4bb85550>]"
]
},
"execution_count": 22,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAFKCAYAAABcq1WoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvXl8HHd9//+anb0PSStpJUuyJd9X\nbOdyHGLiJE7CESCQBCipgUBb2uYLpLRfaFLy5Qht+T3gS0shhUIpafgRahISIARoiGnIHceOY8eO\nr/iQZVn36tz7nPn+MfOZnb1nV7Or3dX7+Y/Xe8x+9qPdec375kRRFEEQBEEQRNUwLPQCCIIgCGKx\nQeJLEARBEFWGxJcgCIIgqgyJL0EQBEFUGRJfgiAIgqgyJL4EQRAEUWWM1XgTr9ev6/HcbjtmZkK6\nHrOeof1Ih/YjHdqPdGg/0qH9SEfP/fB4XHkfq0vL12jkF3oJNQXtRzq0H+nQfqRD+5EO7Uc61dqP\nuhRfgiAIgqhnSHwJgiAIosqQ+BIEQRBElSHxJQiCIIgqQ+JLEARBEFWGxJcgCIIgqgyJL0EQBEFU\nGRJfgiAIgqgyJL4EQRAEUWVIfAmCIAiiypD4EgRBEJoJRxPYf2Icoigu9FLqmqoMViAIgiAag2df\nH8ajz5yF3WrEphVtC72cuqWo+AqCgC9/+cs4ffo0TCYT7rvvPtjtdtx9991IJpPweDz4xje+AbPZ\nXI31EgRBEAvI5GwEADA6FVLE97U3vXjxyAg+eetmmIzkUNVCUfF9+umn4ff78fDDD2NwcBBf/epX\n0drail27duGmm27CN7/5TTz22GPYtWtXNdZLEARBLCBzwRgAwDsbVu574cgIjpydwuhUEL2d+cfo\nESmKXqIMDAxgy5YtAIDe3l6MjIxg3759uOGGGwAAO3fuxN69eyu7SoIgCKImmAtEAQDemZT4jkwG\nAQDBcHxB1lSPFBXftWvX4sUXX0QymUR/fz8uXLiA4eFhxc3c1tYGr9db8YUSBEEQC49i+c5J7udo\nLIkp+bafxFczRd3O1157LQ4ePIgPf/jDWLduHVauXIlTp04pj2vJeHO77boPKPZ4yLWhhvYjHdqP\ndGg/0qH9SEfrfoiiqIjv5GwY7e1OnB2eg6ICPN8Qe1uNz6Ap2/lv/uZvlNs33ngjOjs7EYlEYLVa\nMT4+jo6OjoKvn5kJzW+VGXg8Lni9fl2PWc/QfqRD+5EO7Uc6tB/plLIfoUgc8YQAAIglBJwZmMKJ\ngRnl8TGvv+73Vs/vRyERL+p2PnnyJD7/+c8DAJ5//nls3LgR27dvx1NPPQUA2LNnD3bs2KHLQgmC\nIIjaZTYQS/v/xEwYI1NB5f+BELmdtVLU8l27di1EUcQHPvABWCwW/NM//RN4nsc999yDRx55BN3d\n3bjllluqsVaCIAhiAWEu5yaHGb5gDN7ZsJJsBQABivlqpqj4GgwGfO1rX8u6/8EHH6zIggiCIIja\nhGU6r+lpxmunvIr4Ws08IrEkiW8JUDU0QRAEoQlm+a5e2gwAGJkKYWI2jN4OJ8xGA2U7lwCJL0EQ\nBKGJOTnmu6KrCQaOwxv9UxBFoLvdAafdRHW+JUDiSxAEQWhiNii5nVubLOhstSEaSwIAutodcNpM\nZPmWAIkvQRAEoQlm+TY7LFjZ3aTc3y2LbzSWVEqRiMKQ+BIEQRCamAvG4LAaYTIasLqnWbm/u00S\nX4AynrVC4ksQBEEURRRFzAWiaHZaAACrVOLb4jTDZZNaDpP4aoPElyAIgijKyGQQwUgC3e0OAFD+\nXepxgOM4OGxS5WogFMt7DCKFpvaSBEEQxOJmWG6msVYuMzJwHL7z1zvAGyQbzmWXLd9IYmEWWGeQ\n+BIEQRBF8cutI5scZuU+u9Wk3FZivmT5aoLczgRBEERRfKy1pN2c83EmvlRupA0SX4IgCKIoftmi\ndTkKiy8lXGmDxJcgCIIoio+5ne2mnI+T+JYGiS9BEARRFF8oBo4DHLY84msn8S0FEl+CIAiiKMFw\nHA6rCQaOy/m4xcTDbDTQTF+NkPgSBEEQRYknBJiMhSXDYTOR5asREl+CIAiiKElBhJHPbfUyXCS+\nmiHxJQiCIIoSTwgw8oUlw2k3IULDFTRB4ksQBEEUJSloEF85GWtOHj1I5IfElyAIokpEY0n8/Lmz\nmPZFFnopJRNPiEXFd0WXNGbwyNmpaiypriHxJQiCqBInBmfw273nsffY2EIvpSREUUQyKRSN+W7b\n0AkOwCvHxquzsDqGxJcgCKJKRGNJAPVXC5sURIhAUcvX7bJgfZ8bZ4bn4J0NV2dxdQqJL0EQRJWI\nJepUfJMigOLiCwBv2dgJANh3nKzfQpD4EgRBVAmWBRwM19fYvXhSWncxtzMAXL6uA0begFeOj0MU\nJdGOxZM42j+F2QAlYjFopCBBEESViMUlEQtE6s3yZeJb3F6zW424eHUbXnvTiwsTAYxNh/DgkycR\njSVx0XI3Pnv7pZVebl1Ali9BEESViMtu52CduZ3jJYgvALxl4xIAwItvjOLhp09DFEW0NllwfGCG\nrF8ZEl+CIIgqEVPczvUlvqmYb3G3MwBsWdUKm8WIp18bwmwghmsu7sZNV/ZBBPDqyYkKrrR+IPEl\nCIKoEkrMN5JQ4qH1QIJZvkV6OzNMRh5b13kgigAH4MbLl2Lr+g5wHLD/BCViASS+BEEQVYNZvklB\nxCN/OIPBcf8Cr0gbCWb5GrRLxvZNkuv5kjXt6HDb0ewwY0OfG2eHfZikMqTiCVfBYBD33HMP5ubm\nEI/H8alPfQoejwf33XcfAGDdunX4yle+Uul1EgRB1D3xeFK5vefVC9jz6gX8y11Xo9lhXsBVFSdl\n+WpzOwPAul43Pn3bZqzuaVbu27SiDccHZnB+3I/2Fpvu66wniorvL3/5S6xYsQKf/exnMT4+jo99\n7GPweDy49957sWXLFnz2s5/Fc889h2uvvbYa6yUIgqhbYjkGDkzMhOpHfEuwfAHgsrWetP+3OKXP\n6QvG9FlYHVN0J91uN2ZnZwEAPp8PLS0tGB4expYtWwAAO3fuxN69eyu7SoIgiAYg17QflsxUyyhu\nZ40x33w0yRcZcyS+xcX33e9+N0ZGRvC2t70NH/nIR3D33XejqalJebytrQ1er7eiiyQIgqhXQqqa\nXtbhSk1CqP3xe6U02SgEE19/qL6yvStBUbfzr371K3R3d+OBBx7AyZMn8alPfQoul0t5XEvGnttt\nh9HIz2+lGXg8ruJPWkTQfqRD+5EO7Uc61dqPY/1T+Lvvvogv/Mk2XLmpCyKyxcvhtC7436fY+ztG\npcSwlmbbvNZqskriG0kIC/6ZC1GNtRUV34MHD+Lqq68GAKxfvx7RaBSJRKo12vj4ODo6OgoeY2Ym\nNM9lpuPxuOD11keWYDWg/UiH9iMd2o90StmPGX8UZpMBDquprPd6s38SALDnlQGs7HQilKO+d3o6\ntKB/Hy37MTUTBABEwvF5rVUQRHAc4J1Z2M9cCD1/L4VEvKjbua+vD4cPHwYADA8Pw+FwYNWqVThw\n4AAAYM+ePdixY4cuCyUIgqgVAuE47vn+XvzVt1/AP/74AM4Mz5V8DJaodOzcNJKCkNPtnKwDt3Op\nTTbyYTBwcNnNlHAFDZbvhz70Idx77734yEc+gkQigfvuuw8ejwdf+tKXIAgCLr74Ymzfvr0aayUI\ngqgas4EoEkkBdosR/SM+PPnKedz1/i0lHSMui1YwksC5ET/iCQEcB6ijdfWQcFVqe8lCNNnNmJyj\nOt+i4utwOPDtb3876/7du3dXZEEEQRC1AMtMvubibrx2agInB2eRFATwJZTbMMsXAI70TyGWENDs\nMGM2EMv5nFrlwngAAGA1zT93p9lhwpA3gFg8CbMOx6tXqMMVQRBEDmJyQwyT0YCNy1sRjiZK7kuc\nUJUWHe2fQjyRhNOWXtO
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4bc1be48>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(np.cumsum(r.monitor.l), pu.smooth(r.monitor.r, radius=10))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "_e-NRHW8qoMF"
},
"source": [
"Similar curve can be obtained by using logger summaries (instead of raw episode data in monitor.csv):\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 364
},
"colab_type": "code",
"executionInfo": {
"elapsed": 619,
"status": "ok",
"timestamp": 1541626409332,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "E2PawoKEua1N",
"outputId": "02c4b58e-b665-4534-d02c-24c5862e473f"
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7f4d4bc35240>]"
]
},
"execution_count": 23,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAeAAAAFKCAYAAADFU4wdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3XdUlGfeP/73FIYmnaGI0kGRoqCx\noCRiS+wlsRHjk2KaP43P7ibqZnP2Sb57nmQ3JtlsysY0fXajRiMx9oIlJhbEYEMURZp0GHobyszc\nvz+IrAlGQGfmnvJ+nZNzYBhnPvPJPbyZ+7qv65IIgiCAiIiIjEoqdgFERETWiAFMREQkAgYwERGR\nCBjAREREImAAExERiYABTEREJAK5MZ5EpWrU22O5uTmgtrZFb49nCdiT7tiTO2NfumNPumNPuruX\nniiVTnf9udl9ApbLZWKXYHLYk+7YkztjX7pjT7pjT7ozRE/MLoCJiIgsAQOYiIhIBAxgIiIiETCA\niYiIRMAAJiIiEgEDmIiISAQMYCIiIhEwgImIiETAACYiIhJBj0tRbt++Hbt37+76PjMzE19//TVe\nf/11AMCgQYPwxhtvGKxAIiIiS9RjAM+fPx/z588HAJw9exYHDhzA//7v/+LVV19FTEwM/vCHP+CH\nH37AQw89ZPBiiYjIMpRVN6OusQ0Rge5ilyKaPp2C/vjjj/Hss8+ipKQEMTExAIDExESkpqYapDgi\nIrI89U1teGvTeazbehGX86rFLkc0vd4NKSMjA76+vpDJZHB2du663cPDAyqV6q7/1s3NQa8LWfe0\nw4Q1Yk+6Y0/ujH3pjj3pzlA9EQQB/9x1BU3qDgDAl/uy8I/fj4enq71Bnk+f9N2TXgdwcnIy5s6d\n2+12QRB6/Lf63NZKqXTS6/aGloA96Y49uTP2pTv2pDtD9uT4hRKkZ1UgMtANw8KU2Hw4G29uTMPq\npFjIpKZ7XfC99ERv2xGmpaUhNjYW7u7uqKur67q9oqICXl5efSqKiIisT0VNC7YeuwFHOzmenj4E\nE+L8MGKwF24U12PniXyxyzO6XgVwRUUFHB0doVAoYGNjg+DgYKSnpwMAUlJSkJCQYNAiiYjIvGl1\nOny+9yraO3R44uFBcHOyhUQiwZOPDIaXqz32pd60uvHgXgWwSqWCu/t/rlR79dVX8d5772HRokXw\n9/dHfHy8wQokIiLzty/1JvJKGzB6iDdGRnh33e5gJ8eLc6Igl0nw+Z6rqGloFbFK4+rVGHBUVBS+\n+OKLru9DQ0OxZcsWgxVFRET/0dKqgY1cAhs9XsxqTPllDdh9sgBuTrZ4fEp4t58H+Dhh0cQwbErJ\nxqe7r5j8eLC+9PoiLCIiMg5BEFBY0YSMvGpk5FYhr7QBjnY2mD0uCA8N6w+5zHzCqa1Di8/3XIVO\nEPDM9Ag42tnc8X6JsX64VliH9GuV2HkiH48+FGLkSo2PAUxEZALUbRpcLajF5bwqZORWo66pHQAg\nlUgQ0t8FxaombD6cjaPnijE/MQTDQj0hkUhErrpnyd/norymBZNHDMSQuyy6cWs8uLC8EftSb2LQ\nQFdEBXsYsVLjYwATEYmkvKYFGTlVyMirxvXCOmh1ndM6+9nbYEykD2JCPBAV7A5HOxs0NLdj18l8\nHL9Ygg+/vYzB/q5YOCEMAT6mO4c5M68aR88Xo7+nIx4bH9zj/W+NB//vV+n4bM9VvPH0SLg52Rqh\nUnEwgImIjKRDo8P1olpk5FQjI68albXqrp8FeDshJsQDMSEeCPJ1hlT6y0+3zo4KPPHwIEwYPgDb\nv89BRm41/t///YT4aB/MezDE5IKqSd2BL/dnQSaV4NkZQ3o9fh3g44SFE8Kw+XA2Pt2ViVcseDyY\nAUxEZEA1Da2dY7k51ci6WYu2Di0AwE4hw/Bw5c+fcj16HaB+no747/lDcSW/BtuO3cCpy+X46Vol\nHhnpj0dG+cNOIf6vdUEQ8O9D11Hf1I5HHwru86f0CXF+uF5Yi/TrKoseDxb//xQRkQXq0Ojw74PX\ncCqzvOs2H3cHxIR4YGiIB8IGut7XxVSRQe54/amROHm5DN/9mIfdpwrww6VSzEsIxtho326foI3p\nzNUKpF+rROgAF0wdFdDnfy+RSPDk1AjcrLDs8WAGMBGRnjWpO/DRjsvILqqDv1c/jIvxRUyIB7zc\nHPT6PFKpBA8O7Y+REV44cKYQh84WYuOBazicXoyFE0MRKcJOQ9X1rdiUkg1bhQzLZgy55z8Ebo0H\nv/nVOYsdD7bME+tERCKprFPjza/OIbuoDiMGKfHqE8MxacRAvYfv7ewUcsx9MBhvPjcaY6N8UKJq\nwrtbL+L97ZdQWtVssOf9NZ0g4Mt9V6Fu0yBpYhi87nODhUAfZyycEIYmdQc+3ZUJrU6np0pNAz8B\nExHpSW5JPT74NgONLR2YOsofj44PgdSIU4Xcne3wzIwhmDRiILYdu4GM3Gpk5tXgoWH9MXtcEJwd\nFQZ9/iM/FeFaYR1iwzwxLsZXL49pyePB/ARMRKQH6dcq8fbXF9Cs1mDpw4MwPzHUqOF7uwAfJ7yy\nOBYrH42G0s0e318owR8/S8X+MzfR/vNFYPpWrGpC8g95cHawwX89Mlhvc5RvjQcrXe2wL/UmMi1o\nvWgGMBHRfRAEAQfTCvHJzkxIpRK89FgMxsf6iV0WJBIJYsOU+MszI5E0KQxSiQTJx3Ox+pPT2HO6\nAM2tHXp7rg6NDp/vuQqNVocnp0bo/ZP27etFf7bnKmob2/TyuDpBQE5xPTYfzsYbG3/CzXLjbkvJ\nU9BERPdIq9Nhy+Eb+P5CCdycbLHqsRj4e5vWwhhymRSTRgxEfJQPDqQV4tj5Enz3Yx72n7mJ8cP6\nY8oD/vd9cdOuk/koqmzCg0P7Y1iYp54q/6Vb48GbD3euF/3K4mH3ND9YEAQUlDfibFYFfrpWiZqG\nzjDvZ28DY5+wYAATEd0DdZsGn+6+gozcagz06odVj8XA3dlO7LJ+k4OdDR59KATTRgfg+MUSpPxU\nhENni3AkvRhjonwwdZQ/fD0c+/y42UV1OHDmJpSudlg0MdQAlf/HhDg/XCusxbnrKuw6mY95D/Zu\nPFgQBBSrmjtDN6sSlXWdC6DY28owNsoHI4d4IyLAzehrbDOAiYj6qLaxDf/YfgmFlU2ICnbHi7Oj\nYG9rHr9O7W3lmDoqAJOGD0TqlXIcSCvEyYwynMoow7AwT0wbHQClsnef4tVtGnyx9yogAZ6dEWnw\nRUAkEgmemhqBwopG7Dt9E+ED7j4/uKy6GWezKnE2qwJl1S0AAFsbGUYN8cbIwV6ICnYXdYcp8zhi\niIhMRGFFI/6RnIHaxjaMj/XD45PDzHKpRBu5FA8O7Y9x0b64cEOF/WcKceFGFS7cqELU6QJMihuA\n6GD3u15M9fXRG6iqb8WM+ACEDnAxSt09zQ9W1alxNqsCZ7MqUVTZBKDzNPzwcCVGDvFGTIgHbG1M\nY1tHBjARUS9dzqvGP3dmoq1di/mJIXhkpL9Z7Eh0N1KpBMMHeSEuXIlrhZ2nkzNzq5H586n1qaP8\n8UCEV7c/Ms5nq3AyowwB3k6YNTbIqDX/ejx42YwInLuuwtmsSuSXNQAAZFIJhoZ4YGSEN4aFeZrk\nGQrTq4iIyAQdv1iCTYeyIZNJsHxOFEYM9hK7JL2SSCSICHBDRIAbGtt12HIwC2ezKvDZnqvY8WMe\nHh7pj3ExvrC1kaG+uR3/d+AabORSLJs5RJT9iW8fD179SSqAzq0bIwPdMDLCG3GDlL+597CpYAAT\nEd2FThDw7fFcHEgrRD97G7z0WAxC/YxzulUswX4ueH5WJOY+GIxDZzvHiDcfzsauk/mYNGIA8kob\n0KTuwOJJYfDz7PuFW/pwazy4vrkdUokEoyK8MHyQl8EXG9EnBjAR0W9o79Dii31ZSL9WCW93B/xu\nfoxBl5Q0NV6u9nhiyiD
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d513dcbe0>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"plt.plot(r.progress.total_timesteps, r.progress.eprewmean)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "2iI5DRKArEVl"
},
"source": [
"Note, however, that raw episode data is stored by the Monitor wrapper, and hence looks similar for all algorithms, whereas progress data is handled by the algorithm itself, and hence can vary (column names, type of data available) between algorithms."
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ETICR8zsrKg2"
},
"source": [
"## Plotting: many curves\n",
"While the loading and plotting functions described above in principle give you access to any slice of the training summaries, sometimes it is necessary to plot and compare many training runs (multiple algorithms, multiple seeds for random number generator), and usage of the functions above can get tedious and messy. For that case, `baselines.common.plot_util` provides convenience function plot_results that handles multiple `Result` objects that need to be routed in multiple plots. Consider the following bash snippet that runs ppo2 with cartpole with 6 different seeds for 30k time steps, first with rollout batch size 32, and then with batch size 128 (note that the next cell will take a little while to run):"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 629
},
"colab_type": "code",
"executionInfo": {
"elapsed": 470993,
"status": "ok",
"timestamp": 1541626885128,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "OLEYIE95ue2e",
"outputId": "d1f2760b-1433-4b73-c5ce-ceace66e07cc"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 32, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n",
"env_type: classic_control\n",
"Training ppo2 on classic_control:CartPole-v0 with arguments \n",
"{'nsteps': 128, 'network': 'mlp'}\n"
]
}
],
"source": [
"!for seed in $(seq 0 5); do OPENAI_LOG_FORMAT=csv OPENAI_LOGDIR=$HOME/logs/cartpole-ppo-hp/b32-$seed python -m baselines.run --alg=ppo2 --env=CartPole-v0 --num_timesteps=3e4 --seed=$seed --nsteps=32; done\n",
"!for seed in $(seq 0 5); do OPENAI_LOG_FORMAT=csv OPENAI_LOGDIR=$HOME/logs/cartpole-ppo-hp/b128-$seed python -m baselines.run --alg=ppo2 --env=CartPole-v0 --num_timesteps=3e4 --seed=$seed --nsteps=128; done"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XzG8DDNwrwbG"
},
"source": [
"The results of 12 runs from the cell above can be loaded just as before, via (we discard first result that is actually from the very first run in the previous section):"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 88
},
"colab_type": "code",
"executionInfo": {
"elapsed": 319,
"status": "ok",
"timestamp": 1541629440197,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "cNKemLHPv03E",
"outputId": "8ff4813e-23f3-4696-a531-6ef3bd12c569"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"12\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/usr/local/lib/python3.6/dist-packages/baselines/bench/monitor.py:164: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access\n",
" df.headers = headers # HACK to preserve backwards compatibility\n"
]
}
],
"source": [
"results = pu.load_results('~/logs/cartpole-ppo-hp'); print(len(results))"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "xy388w_tssIL"
},
"source": [
"But how do we plot all 12 of them in a sensible manner? `baselines.common.plot_util` module provides plot_results function to do just that:"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 937
},
"colab_type": "code",
"executionInfo": {
"elapsed": 858,
"status": "ok",
"timestamp": 1541629448579,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "1yemaAkKweB2",
"outputId": "612d1da5-e2a1-4836-b223-aebfaa3feac6"
},
"outputs": [
{
"data": {
"text/plain": [
"(<matplotlib.figure.Figure at 0x7f4d4bc1bc88>,\n",
" array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4bad9358>],\n",
" [<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4bab7f28>]],\n",
" dtype=object))"
]
},
"execution_count": 26,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAANUCAYAAAD8QVkFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXdgHNXV9p+d7U19JVuWCzYYG2xT\nDZhuSjChJoEXYkwJECDUjxBqQnnjEGpCCyG8VFNMh1AM2FTbgLEx7r1b1dKqbm8z8/0xc+/cKati\nJEs298cfaGdnd++s5HvmnPOcc2yyLMvgcDgcDqefEfp7ARwOh8PhANwgcTgcDmeAwA0Sh8PhcAYE\n3CBxOBwOZ0DADRKHw+FwBgTcIHE4HA5nQMANEofTCyxcuBAnn3yy5XPV1dX41a9+hUsuuUR3vLGx\nEVdddRVOPfVUTJkyBTNnzqTPff311zjrrLMwZcoUnH/++VixYkVfLp/DGRA4+nsBHM6ezJYtW3DN\nNddg4sSJqK6u1j131113Ydy4cfjPf/6DxsZGnH766TjiiCNQVlaGm266Ca+++irGjBmDefPm4brr\nrsPcuXP76So4nF0D95A4nF7kgQcewCmnnIIpU6ZgyZIlcLvdmDFjBg488EDTueeddx4uuugiAEBF\nRQWqqqqwZcsW1NTUwOv1YsyYMQCAI444Ajt27EAkEtml18Lh7Gq4QeJweom6ujqMGzcOs2fPxqWX\nXoq//vWvGDJkCMrLyy3PP+GEE1BYWAgAqK+vx7Zt27Dffvth1KhREAQBCxYsAADMnj0b48aNQ0FB\nwS67Fg6nP+AhOw6nl3C73Tj11FMBAKeeeiruvPNOpNNpuN3uTl8XiURw3XXX4corr0RlZSUAYPr0\n6bjyyivh8XggSRKeffbZPl8/h9PfcA+Jw+klioqKIAjKP6lAIAAA6Ojo6PQ14XAYF110EY477jhc\nddVVABSxw5///Ge89dZbWLRoEZ588klce+21iMfjfXsBHE4/ww0Sh9NLsMaH5HuKiorynh+LxXDZ\nZZfh7LPPxvXXX0+PL126FFVVVdh3330BAIcffjgEQcDmzZv7aOUczsCAGyQOp5dIpVL47LPPACh5\nn/Hjx8PlcuU9/9FHH8URRxxhkoOPGDECmzZtQm1tLQBg9erViEajGDZsWJ+tncMZCNj4+AkO56ez\ncOFCTJ8+Hcceeyy+/PJLCIKA+++/H6tXr8aMGTMQi8UQi8UwaNAgTJgwAQ8++CDGjRuH8vJyndG6\n+OKL8dvf/havvfYaXnrpJUiSBJfLhRtuuAEnnXRSP14hh9P3cIPE4XA4nAEBD9lxOBwOZ0DADRKH\nw+FwBgTcIHE4HA5nQMANEofD4XAGBNwgcTgcDmdA0K+tg8Lh6E9+j+JiH9raEr2wmoENv849C36d\nexY/l+sEfvq1hkLBvM/t9h6Sw2Hv7yXsEvh17lnw69yz+LlcJ9C317rbGyQOh8Ph7Blwg8ThcDic\nAQE3SBwOh8MZEHCDxOFwOJwBATdIHA6HwxkQdMsgbdiwASeddBJeeeUVAEBDQwMuvPBCTJ06FTfc\ncAMymQwA4IMPPsBvfvMbnHvuuXjrrbf6btUcDofD2ePo0iAlEglMnz4dkyZNoscef/xxTJ06FTNn\nzsTw4cPx9ttvI5FI4Mknn8SLL76Il19+GTNmzEB7e3ufLp7D4XA4ew5dGiSXy4VnnnkG5eXl9NjC\nhQtx4oknAgAmT56MBQsWYPny5Rg/fjyCwSA8Hg8OPvhgLFmypO9W3od8/PGH+Ne/HtUdkyQJTz31\nBE4/XT+T5s03X8Pvf38RLr/8Irz7ruIVbtmyGddddyWuu+5K3HDD1aivr9tla+dwOJzdlS4NksPh\ngMfj0R1LJpN0qFhpaSnC4TCam5tRUlJCzykpKUE4HO7l5fYfr7zyIioqBoEdH1VXV4uPP/4QTz31\nPJ566jnMnPkSYrEYnn/+aUybdgmeeOJpnHbamXj11Rn9uHIOh8PZPfjJrYPyzffrzty/4mJfr1T9\ndtaKYmcIBj1obW3CHXf8ETt27MDFF1+MK6+8DIFAAM8//zT9vNLSffHmm6/D7/cDAPx+HzweoKIi\nBFlOq+dlMHhwea+ssbevc6DCr3PPgl/nnkdfXetOGSSfz4dUKgWPx4PGxkaUl5ejvLwczc3N9Jym\npiYceOCBnb5PV/2Q7vnuL/hw8387PUcQbJCk7g+9PWPU2bjnyL91ek40msKmTZvx/POvIh6P4ZJL\npuLYY3+BZDIKWZZNPfgSiSgWLfoegUABHI4Apk27DJdffhEef/wJSJKEZ5556Sf37QuFgr3S+2+g\nw69zz4Jf557HT73WXu9ld+SRR2L27NkAgDlz5uCYY47BAQccgJUrVyISiSAej2PJkiU49NBDd27F\nA4AJEw6Ew+FAYWER/H4/Ojo68p67atVKPPnko7jrrukAgKeffhJXXnkNZs58B+eeez5efPGZXbVs\nDofD2W3p0kNatWoVHnjgAdTV1cHhcGD27Nl4+OGHcdttt+GNN95AZWUlzj77bDidTtx000247LLL\nYLPZcM011yAY/Glu3T1H/q1Lb6bv7kxs+kc267M2btyABx6YjgcffBQVFYMAACtXLscf/nAdAGDi\nxMPxwAP39sH6OBwOZ8+iS4M0btw4vPzyy6bjL7zwgunYlClTMGXKlN5ZWT+zevUKiKKISCSCZDKJ\ngoJC0zmiKOK++/6Ke+99EIMHV9LjQ4YMxZo1qzBp0tFYu3YNqqqG7sqlczgczm5Jv85DGsgMGzYC\nd955G+rqanDFFVfj0UcfwubNmxCLxXDttVfg6KOPxciRe6OhoR4PPvh3+rqrr74e11xzAx5++D68\n+upLcLlcuOWWv/TjlXA4nD2FtlQrPt7yEX47dhoE257XaMcmd0cO10f0Rqjt55JM5Ne5Z8Gvc89i\nV13nI4sfwn2LpuO1097GicN/0eefZ8WAEzVwOBwOZ9dTH68HAKxpXdPPK+kbuEHicDic3YRwogkA\nsLFtPbJiFh3pnWvP1ppqwcKG73tzab0CN0gcDoezmxBOKgZpQ+s6XDp7GvZ5bhhaUy09fp8z3j0F\nZ7z3C6xrXZv3nN9+9BsMfqq4W00OegtukDgcDmc3oTmptGPb0LYBs7d9AgBoUr2mnrCxfQMAoD6W\nv8/mF9WfQZRFJHKdNzDoTbhB4nA4nN2EcEIxSLGsJirIStmdfj9RynV5TjQT2en37yncIHE4HM5u\nQDKX1BkiQiqX3On3bEu3dXlORzp/l5rehhskDofD2Q0g4Tqvw6s7nsqldvo921KtlseTjJGLZLhB\n4nA4HA4DMTz7lY4zHO+Zh5QW0/TnfB5Se0o7HuEeEofD4XBYcmq+Z9/iMbouDSmxZx4S6xWxhoel\nlTknwnNIHA6Hw2HJqeKFgCuA00aeSY8ne+ghNSe1MUH5BBFtac0g8RwSh8PhcHQQD8luc+C5U17C\nUyc9C6DnOaQWxiBJsmR5Thv3kDgcDoeTj5ysGCSn4AQAeFRxQ09zSC2prg2SLmTHPSQOh8PhsIiS\nCABwCHYAgEs1TNlu1BKxtCa1zg7d85C4QeJwOJwBQVuqFX/86jr873d35hUB7AqIh2QXHLr/i3LP\nDFIz4yGJsmh5Tms/GSQ+D4nD4XDysKp5JS78+DzUxWoBAK+texn3HHkvzh9zwS5fC8khOWzKtu1Q\nDVKuhx5SS089pDTPIXE4HE6/89iP/0BdrBY3T7wdd076K5K5JG6ff3O/rIW0+SGeETFMPTdImock\n5zFI7Ux90sb2DT3+jJ2FGyQOh8PJA+mkfeMhN+O6g/4f9ivdH1kx0y9ryanhNSJqENRcUk9Ddmx3\ncLETUYPdZscFYy/C9sg2vLzmxZ1Ycc/hBonD4XDyEM1E4HP4aHjMBgEy+mfINg3ZCXbIsoz/bnxb\nPW6dB8pHd2XfxZ5i3Hb
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4bc1bc88>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"pu.plot_results(results)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "uUNu5fEfsvot"
},
"source": [
"The results are split into two groups based on batch size and are plotted on a separate graph. More specifically, by default plot_results considers digits after dash at the end of the directory name to be seed id and groups the runs that differ only by those together.\n",
"\n",
"Showing all seeds on the same plot may be somewhat hard to comprehend and analyse. We can instead average over all seeds via the following command:"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 937
},
"colab_type": "code",
"executionInfo": {
"elapsed": 848,
"status": "ok",
"timestamp": 1541629456405,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "7d_aeRYXacFP",
"outputId": "22554c64-b5a5-4892-81d0-8a7d0d3569a4"
},
"outputs": [
{
"data": {
"text/plain": [
"(<matplotlib.figure.Figure at 0x7f4d4bcd8240>,\n",
" array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4b9f55c0>],\n",
" [<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4b9bda20>]],\n",
" dtype=object))"
]
},
"execution_count": 27,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAANUCAYAAAD8QVkFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvWeYZGd1rn3vULFz7p7pCZphZqRh\nNApIIISQECAjmWzDwWA+wUW2Rfh8cQ58xjZHmKND8uUDl+EAJhiBkUg2BhMsgpCEQFkjjWakydPT\nOVeu2nl/P6pqd1XXrtAzHYf3/iN1hV3vru7Za6+1nvdZkuu6LgKBQCAQrDHyWi9AIBAIBAIQAUkg\nEAgE6wQRkAQCgUCwLhABSSAQCATrAhGQBAKBQLAuEAFJIBAIBOsCEZAEgrPkoYce4oYbbvB9bnh4\nmNe+9rW89a1vLXt8amqK97znPdx0003ceOON3HHHHd5z99xzD69+9au58cYb+bM/+zMOHjy4kssX\nCNYd6lovQCA43zh16hS33HILV155JcPDw2XPffSjH2Xfvn186UtfYmpqile84hVcddVVdHd388EP\nfpBvf/vbXHjhhdx33328733v4957712jsxAIVh+RIQkE58inPvUpXvayl3HjjTfy+OOPEwqFuP32\n27n00ksrXvuGN7yBm2++GYC+vj4GBwc5deoUIyMjRCIRLrzwQgCuuuoqJicnSSaTq3ouAsFaIgKS\nQHAOjI2NsW/fPu666y7e9ra38fd///ds3ryZ3t5e39e/+MUvpq2tDYDx8XGGhobYu3cvO3fuRJZl\nHnjgAQDuuusu9u3bR2tr66qdi0Cw1oiSnUBwDoRCIW666SYAbrrpJv7u7/4OXdcJhUI135dMJnnf\n+97Hu9/9bjZt2gTAxz/+cd797ncTDodxHIevfvWrK75+gWA9ITIkgeAcaG9vR5bz/4yam5sBSCQS\nNd8zMzPDzTffzHXXXcd73vMeIC92+Ju/+Ru+//3v8/DDD/OFL3yB9773vWQymZU9AYFgHSECkkBw\nDpQGn2K/p729verr0+k0b3/723nNa17D+9//fu/xAwcOMDg4yJ49ewB43vOehyzLnDx5coVWLhCs\nP0RAEgjOAU3T+OUvfwnk+z4XX3wxwWCw6us/+9nPctVVV1XIwbdv386JEycYHR0F4PDhw6RSKbZu\n3bpiaxcI1huSGD8hEJwdDz30EB//+Me59tprufvuu5FlmU9+8pMcPnyY22+/nXQ6TTqdpr+/n/37\n9/PpT3+affv20dvbWxa03vKWt/DGN76RO++8k29+85s4jkMwGOQDH/gAL33pS9fwDAWC1UUEJIFA\nIBCsC0TJTiAQCATrAhGQBAKBQLAuEAFJIBAIBOsCEZAEAoFAsC4QAUkgEAgE64JVsQ6amUkty3E6\nOqLEYtllOdZ6R5zr+Yk41/MTca6N09PTUvW5DZUhqaqy1ktYNcS5np+Icz0/Eee6PGyogCQQCASC\n8xcRkAQCgUCwLhABSSAQCATrAhGQBAKBQLAuEAFJIBAIBOsCEZAEAoFAsC4QAUkgEAgE6wIRkAQC\ngWAD8rOf/Sef//xnKx53HIcvfvGfeMUrymdpfe97d/LOd97MO95xM//+798H4NSpk7zvfe/mfe97\nNx/4wF8yPj5WcbxsNsv73/8ekskkU1OTvPGNb+Sd77yZz3zmfwPw+c9/lvvuu2dZzkkEJIFAIDiP\n+Nd//QZ9ff2UjrobGxvlZz/7T774xa/zxS9+jTvu+CbpdJqvf/3LvPnNb+Wf/unLvPzlr+Lb3769\n4nhf//o/88pXvobW1lY+//nP8ra3vY2vfOWbyLLC5OQk73znX/C1r30ZTdPOee0iIAkEAsEGZWJi\njP/+39/PzTe/gZ/85EcAvO51b+BP/uT1Za8bGNjE//2/X0VVVQKBAOFwmEwmTVtbO4lEAoBUKklb\nW3vZ+3Rd5557fs2LX3wDjuNw8OABXvziFwPwwQ9+mP7+fkKhEC94wQv55S//65zPZ1W87AQCgeB8\n5n/84n/w3UPfW9ZjvnLna7j16v9V8zUjI8N8/evfJpNJ89a3vomXv/xVRKNNFa+TZZloNArAww8/\nSFtbO319/bzjHe/hHe+4mW984ys4jsNXvvLNsvc988xhdu58FoqiMD8/RyTSxCc+8QmeeOIgl1xy\nGe95z3sBuPTSy/n5z3/CK1/5mnM6Z5EhCQQCwQZl//5LUVWVtrZ2mpqavGynGocOPcUXvvBZPvrR\njwPw5S9/gXe/+xbuuOPfeP3r/4xvfOMrZa+fnZ2hp6cPANd1mZ2d5uabb+bzn/9njh07yu9/fz8A\nPT29TE9PnfP5iAxJIBAIzpHP/NFn+NBlH12DT5bKf5KqvAw4fvwYn/rUx/n0pz9LX18/AE899SR/\n8RfvA+DKK5/Hpz51W+UnFA7a1tZOf/8AW7duZWYmxRVXXMnp0ye5+uprlulcRIYkEAgEG5bDhw9i\n2zaxWIxcLkdra5vv62zb5hOf+Htuu+3TDAxs8h7fvHkLTz99CIBnnnmawcEtZe/r7u5hZiaf+aiq\nyqZNmxkaGgLg6NFn2Lp1GwCzs9P09PSe8/mIDEkgEAg2KFu3bufv/u7/Y2xshHe96y+RJIn/838+\nzcmTJ0in07z3ve/immuuZceOZzExMc6nP/2/vff+5V++n1tu+QD/8A+f4Nvf/ibBYJAPfehvy45/\n0UXP5sSJ49i2jaIovP/9H+Sv//qvMQyLHTt28oIXXAvAE08c4PLLrzjn85HcUm3gCrFcA/p6elqW\n7VjrHXGu5yfiXM9P1upcLcdClVc2r/inf/pH9u7dx0te8kdA5bnqus673vVWvvSlrxOJROoe77wZ\n0CcQCASCPLql8/Tc4RX/nLe//d38+Mc/JJlM+j7/1a9+ibe97Z0NBaN6iJKdQCAQbEDmtFlsx17x\nz4lGm/jc575Y9flbbvnAsn2WyJAEAoFgA2LYBpZjrkpQWi1EQBIIBIINiG7rSJJEzsqt9VKWDRGQ\nBAKBYAOi2xoBOUDGyqz1UpYNEZAEAoFgg+G6LqZtAWDZxhqvZvkQAUkgEAg2GKZj4rh24f+tNV7N\n8iECkkAgEGwwMkYaRVaA/F6k8wURkAQCgWCDkbNzKFI+IJmOucarWT5EQBIIBIINhlHSN7JEQBII\nBALBWmE4pQFJ7EMSCASCPyjGUiOk9PXhzVeaFdmuzSpYkq4KIiAJBAJBHVzXZTIzRcqoPQBvtTDt\n0jKdg2Zpa7aW5aSul53jOPzP//k/OX78OIFAgFtvvZVoNMqHPvQhbNump6eHz3zmMwSDwdVYr0Ag\nEKw6WSsLkkvGyq71UoB8mU4tqOxUKUDSTBIJnLu56VpTNyD9+te/JpVK8Z3vfIfh4WFuu+02Ojs7\nedOb3sRNN93EP/7jP/KDH/yAN73pTauxXoFAIFh1EloMVVLJrYOAZDlWYQ9SPiBJkoRmrv26loO6\nJbuhoSH2798PwNatWxkfH+ehhx7iJS95CQDXX389DzzwwMquUiAQCNaQbKEkVnRHWEuKHnal5Ow/\nkJLd7t27uf3223nLW97CmTNnGBkZIZfLeSW6rq4uZmZmah6joyOKqirLsuBaw53ON8S5np+Ic914\nTDoqQbsJwzbo7Ip6m1JLWbVzzWj0uG1lQUmRlFX9rlfqs+oGpOuuu47HH3+cP//zP2fPnj3s2LGD\nY8eOec83ou6IxZYnnRQTKM9PxLmen5xP5zoxM48sSRi2wZgyV9GvWc1zHc/ME8+UX1Ntx2FGXZ3P\nP9dzrRXMGhrQ91d/9Vfe/7/0pS+lr68PTdMIh8NMTU3R29t71osTCASC9Yxpm5iuSUgKosoqWSu7\npgIC09YrHrNdC9uxfTO3jUTdHtKRI0f467/+awDuu+8+9u7dy9VXX81dd90FwC9+8Qte+MIXruwq\nBQKBYI2Y0+ZQCzY9siS
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4bcd8240>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"pu.plot_results(results, average_group=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "iA89jxdgtElg"
},
"source": [
"The lighter shade shows the standard deviation of data, and darker shade - error in estimate of the mean (that is, standard deviation divided by square root of number of seeds). Note that averaging over seeds requires resampling to a common grid, which, in turn, requires smoothing (using language of signal processing, we need to do low-pass filtering before resampling to avoid aliasing effects). You can change the amount of smoothing by adjusting `resample` and `smooth_step` arguments to achieve desired smoothing effect See the docstring of plot_util function for more info.\n",
"\n",
"To plot both groups on the same graph, we can use the following:"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 488
},
"colab_type": "code",
"executionInfo": {
"elapsed": 581,
"status": "ok",
"timestamp": 1541629463814,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "_5IMVF3EGBMD",
"outputId": "b29e81b6-8a57-4007-fe31-86652b7ee1e5"
},
"outputs": [
{
"data": {
"text/plain": [
"(<matplotlib.figure.Figure at 0x7f4d4b919cc0>,\n",
" array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4b95f198>]],\n",
" dtype=object))"
]
},
"execution_count": 28,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAGkCAYAAAB+TFE1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsvXeAJGd5r/t8VdVxct6gDVqlVUAC\nYZKQwAiBbUA21wdfDoYLNkZgLIdr+yAu3Gss+1wdTLANB3wwJoPBgG1ANiZIYCPAEkFZ2tXmMDl0\nT8fK6fxR3TXd090TdmdnA9/zj3a6qqu+r2dUb7/p94owDEMkEolEIjnLKGd7ARKJRCKRgDRIEolE\nIjlHkAZJIpFIJOcE0iBJJBKJ5JxAGiSJRCKRnBNIgySRSCSScwJtM26ysFDZkOsMDGQpFIwNuda5\njtzrhYnc64WJ3OvaGRnp6XjsvPKQNE0920vYNOReL0zkXi9M5F43hvPKIEkkEonkwkUaJIlEIpGc\nE0iDJJFIJJJzAmmQJBKJRHJOIA2SRCKRSM4JpEGSSCQSyTmBNEgSiUQiOSeQBkkikUjOQ77xjX/l\nwx/+QMvrQRDwkY98iFe84pam17/85X/gtttez5ve9Hq+8pV/BODYsaP83u+9hd/7vbfwB3/wO0xP\nT7VczzAMfv/3f5tyuczc3Cyvec1ruO221/O+9/0PAD784Q/w/e9/b0P2JA2SRCKRXED8/d9/mrGx\nLTTOXp2amuQb3/hXPvKRT/KRj3yCL3zhs1SrVT75yY/yutf9Bh/60Ed5+ct/mc9//jMt1/vkJ/+O\nW299Jb29vXz4wx/gjW98Ix/72GdRFJXZ2Vluu+2tfOITH8WyrNNeuzRIEolEcp4yMzPFf/tvv8/r\nX/9qvv71uwF41ateza/+6q81nbd16zb+1//6OJqmkUgkSKfT6HqVvr5+SqUSAJVKmb6+/qb32bbN\n9773XW6++SUEQcDjjz/CzTffDMAf//Hb2bJlC6lUiuc//ybuvfdbp72fTdGyk0gkkguZt93zNr70\n5Jc39Jq3XvJK7rzh/1/xnImJcT75yc+j61V+4zd+nZe//JfJZrtazlMUhWw2C8BPfvIj+vr6GRvb\nwpve9Nu86U2v59Of/hhBEPCxj3226X1PPbWPSy65FFVVWVzMk8l08e53v5tHH32c6657Br/9278L\nwNOffj3f/ObXufXWV57WnqWHJJFIJOcp1177dDRNo6+vn66urtjb6cSTTz7B3/zNB3jXu/47AB/9\n6N/wlrfczhe+8M/82q/9Vz796Y81nZ/LLTAyMgZAGIbkcvO8/vWv58Mf/jsOHTrI/ff/EICRkVHm\n5+dOez/SQ5JIJJLT5H0vfR93PONdZ+HOovkn0eE04PDhQ7znPf+d9773A4yNbQHgiSce461v/T0A\nnvWs5/Ce99zVeofaRfv6+tmyZSs7d+5kYaHCz/3cszh+/Cg33HDjBu1FekgSiURy3rJv3+P4vk+h\nUMA0TXp7+9qe5/s+7373n3PXXe9l69Zt8evbt+9g//4nAXjqqf1cdNGOpvcND4+wsBB5PpqmsW3b\ndk6cOAHAwYNPsXPnLgByuXlGRkZPez/SQ5JIJJLzlJ07d/Mnf/L/MDU1wZvf/DsIIfjrv34vR48e\noVqt8ru/+2ZuvPEF7NlzKTMz07z3vf8jfu/v/M7vc/vtf8D73/9uPv/5z5JMJrnjjv+v6fpXXnk1\nR44cxvd9VFXl93//j3nHO96B43js2XMJz3/+CwB49NFHuP76nzvt/YiwsTbwDLFRA/pGRno27Frn\nOnKvFyZyrxcmZ2uvfuCjKmd2FtOHPvRXXHXVNbz4xS8FWvdq2zZvfvNv8Ld/+0kymcyq17tgBvRJ\nJBKJJMLxHPbn953x+/zWb72Ff/mXr1Iul9se//jH/5Y3vvG2NRmj1ZAhO4lEIjkPyVs5vMA74/fJ\nZrv44Ac/0vH47bf/wYbdS3pIEolEch5i+zZe4OIH/tleyoYhDZJEIpGch9i+jRAC0zPP9lI2DGmQ\nJBKJ5DzE8W00oaF7+tleyoYhDZJEIpGchzi+ixACz3fO9lI2DFnUIJFIJOcZju/wo3//IfOT8/zm\nW25rOvbDH97H5z73aRKJBP39A/zJn/w5qVSKL3/5H7j33m8ShvCyl93aIsAK8LWv/TO6XuW1r30D\nX/jC57jnnm+iaRp//MdvZ2homHe/+8/51Kc+ccb2JT0kiUQiOc/QnSqKiB7fyyvt/vEfv8hf/uWH\n+PCH/45sNst99/1Hx/ETjRQKi/zLv3yV17zm/+LYsaN897v38PGPf5a3ve2d3H//DxkdHeO5z72B\nz362WYB1I5EGSSKRSM4zTN9ECIX8fI733/nupvETH/zgR+ju7sbzPPL5PCMjIx3HTzRy991f4Rd/\n8WUoisL99/+Am2++BU3TuOKKvfzWb70FgF/+5V/li1/84hnblwzZSSQSyWnytrfBl77UOvbhdLj1\nVo8777TbHnNqeaP56Tn+37/8U3Zn98TjJ4QQfOMb/8rHP/633HjjC3jGM54J0Hb8RCMPP/wgt9/+\nfwMwOzuDoij80R/9Hr7v8bu/+4dcdtnlZDIZhoaGmJgYZ8eOnRu6X5AekkQikZx3OEFkkPZceSmh\nQsv4iZe97Fa+/OW7qVQq3HPP0uC85eMnGsnlFhgdjQRSwzAkCAL+8i//J29841t4z3uW5jKNjY1t\nyKiJdkgPSSKRSNbAdHWKvlQ/XYlWT+h974M77ti88msvcAEQCLzQj8eVu67Dj350P8997g1omsaN\nN76QRx55iJe+9Bfbjp9oJRo1MTg4xM6duxBCcN11T2d2dnoztiU9JIlEIlmNMAyZqc5QsgpneykA\nuH5kkI4fOkrge8zlZzFNk4GBQd773rvI5RYA2L//SXbu3NVx/EQjjaMmnvOcG/jJT34EwMmTJxgd\nHYvPm5ub25BRE+1Y1UMKgoA//dM/5fDhwyQSCe68806y2Sx33HEHvu8zMjLC+973PpLJ5BlZoEQi\nkZxtDM8gJEA/R1QRvJpc0Nj2LXzmrz5JYb7Am9/8O2iaxtve9k7e8Y4/JpFIMjg4yG23vZWHHvpp\n2/ETV111Tfzz9df/HI899iiXX76Xa655Gj/+8f285S2/CcAf/dHbAbAsi1wuF89B2mhWHT9x7733\n8m//9m984AMfYHx8nLvuuovBwUFe8IIX8Eu/9Ev81V/9FVu2bOHXf/3XO15Djp9YP3KvFyZyr+cn\n05VJFswFhBBcO/L0luObuVcv8Hhk7iGS6pIT0J/qZ1ffxad13Xw+xx13/CEf//hn4ymxy/nyl/+B\nZFLwylf+11O+z2mNnzhx4gTXXnstADt37mR6epof//jHvPjFLwbgRS96EQ888MApL04ikUjOdQzP\nAsDzz76QqeM7LQbD9K3Tvu7Q0DC33vor/MM/fK7t8fn5Oe6//we84Q1vOO17dWLVkN3ll1/OZz7z\nGd7whjdw8uRJJiYmME0zDtENDQ2xsLCw4jUGBrJo2sYMkVrJul5oyL1emMi9nn/MBhpJvwvHdxgc\nyrYdirdZe13QLUbCviajpAp1Q+5/222/2fHYyEgPn//852r/PjMpmlUN0gtf+EIefvhhXvva13LF\nFVewZ88eDh06FB9fy8DZQsE4vVXWuJBCAKsh93phIvd6fjKzsIgiBI7vMKXmySSah9Ft5l6nq3mK\nRvMz1Q8CFrTNuf/p7nUlw7mmsu8//MM/jP99yy23MDY2hmVZpNNp5ubm4tp1iUQiudDwAg83dEmJ\nJJqiYXhGi0HaTJw2Yqp+6G3KOPMzzao5pAMHDvCOd7wDgO9///tcddVV3HDDDXz7298G4J577uGm\nm246s6uUSCSSs0TOzKGJ6EGvCCVuSj1buLUepOXYfntVh/OJNeWQwjDkVa96FalUive///2oqsrb\n3/52vvSlL7Ft2zZe+cp
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4b919cc0>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"pu.plot_results(results, average_group=True, split_fn=lambda _: '')"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ABZmMmCmtQ1X"
},
"source": [
"Option `split_fn=labmda _:''` effectively disables splitting, so that all curves end up on the same panel.\n",
"\n",
"Now, with many groups the overlapping shaded regions may start looking messy. We can disable either light shaded region (corresponding to standard deviation of the curves in the group) or darker shaded region (corresponding to the error in mean estimate) by using `shaded_std=False` or `shaded_err=False` options respectively. For instance,"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 488
},
"colab_type": "code",
"executionInfo": {
"elapsed": 546,
"status": "ok",
"timestamp": 1541629471846,
"user": {
"displayName": "Peter Zhokhov",
"photoUrl": "",
"userId": "10254602425711636265"
},
"user_tz": 480
},
"id": "x4rVG6RGI31B",
"outputId": "a036e51d-b6c6-4855-ce5a-a42c149a59ee"
},
"outputs": [
{
"data": {
"text/plain": [
"(<matplotlib.figure.Figure at 0x7f4d4b8577b8>,\n",
" array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f4d4b8bc4e0>]],\n",
" dtype=object))"
]
},
"execution_count": 29,
"metadata": {
"tags": []
},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAaQAAAGkCAYAAAB+TFE1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXd4ZGd97z/nnOlFo1Evu9pebK/X\nbd3WDdNCAIMhEIxDyQWME+oNBBJ8bwwh1+ECNwkEgwkYYnAwxjTbEAM2Dm644bK9SyutujRF09sp\n948zczSjmZFGu9Lu2nk/z8ODdebMmfeM7PPVr7zfn2QYhoFAIBAIBKcY+VQvQCAQCAQCEIIkEAgE\ngtMEIUgCgUAgOC0QgiQQCASC0wIhSAKBQCA4LRCCJBAIBILTAtvJ+JDp6cSSXCcY9BCNppfkWqc7\n4l5fnoh7fXki7rVx2tv9dV97SUVINptyqpdw0hD3+vJE3OvLE3GvS8NLSpAEAoFA8PJFCJJAIBAI\nTguEIAkEAoHgtEAIkkAgEAhOC4QgCQQCgeC0QAiSQCAQCE4LhCAJBAKB4LRACJJAIBC8BHnggV9w\n661fqTqu6zq33fY13vjGV1ccv+eeH3LDDe/hAx94Dz/72Y8BGBjo56MfvZGPfvRGPv7xDzE2Nlp1\nvXQ6zcc+9hfE43EmJyd45zvfyQ03vIcvf/kfAbj11q/w2GOPLMk9CUESCASClxH/8R930NnZRfns\n1dHRER544Bfcdtt3ue2273DXXd8nmUzy3e/+G+9615/zta/9G294w5v4wQ++V3W97373W1xzzbU0\nNTVx661f4X3vex/f/vb3kWWFiYkJbrjhL/nOd/6NbDZ7wmsXgiQQCAQvUcbHR/nrv/4Y73nPO/jl\nL+8D4G1vewdvfevbK87r7u7hG9+4HZvNht1ux+VykUolCQSaicViACQScQKB5or35XI5HnnkYV75\nyteg6zq7dr3IK1/5SgA++cm/oaurC6fTyWWXXcFDD/36hO/npHjZCQQCwcuZTz34KX60554lveY1\n667lc9v/z7znDA8f47vf/QGpVJI///PrecMb3oTH4606T5ZlPB4PAM8++zSBQDOdnV184AN/wQc+\n8B7uuOPb6LrOt7/9/Yr37d+/l3Xr1qMoCpFIGLfbyxe+8AV27NjFOeecx1/8xUcAOPfc8/nVr37J\nNddce0L3LCIkgUAgeImydeu52Gw2AoFmvF6vFe3UY8+e3Xz961/h5pv/AYB/+7evc+ONH+auu37K\n299+HXfc8e2K80OhadrbOwEwDINQaIr3vOc93Hrrtzh06CBPPvkEAO3tHUxNTZ7w/YgISSAQCE6Q\nL7/2y3z6vJtPwSdLlT9JdU4DDh8+xBe/+A986UtfobOzC4Ddu3fyl3/5UQAuvPBivvjFW6o/oXjR\nQKCZrq5u+vr6mJ5OsG3bhRw92s/27Zcv0b2ICEkgEAhesuzduwtN04hGo2QyGZqaAjXP0zSNL3zh\n89xyy5fo7u6xjvf2rmTfvj0A7N+/jxUrVla8r62tnelpM/Kx2Wz09PQyODgIwMGD++nrWwVAKDRF\ne3vHCd+PiJAEAoHgJUpf32r+7u/+ltHRYT74wQ8hSRL/8i9for//CMlkko985INcfvmVrF27nvHx\nMb70pX+03vuhD32MD3/44/y///cFfvCD7+NwOPj0p/93xfXPOOMsjhw5jKZpKIrCxz72ST7zmc+Q\nz6usXbuOyy67EoAdO17k/PO3nfD9SEZ5b+AysVQD+trb/Ut2rdMdca8vT8S9vnQpaAVkSUaRq+cB\nvdzutZyvfe2fOfPMLbzqVa8Fqu81l8vxwQ/+Od/85ndxu90LXu9lM6BPIBAIThX39f+MfeG9p3oZ\nJ533v/9G7r//58Tj8Zqv3377N3nf+25oSIwWQqTsBAKBYAHCmTAz2RmOxgY4u33rqV7OScXj8fLV\nr95W9/UPf/jjS/ZZIkISCASCBeifOYxDcRDLz5zqpVjk1ByTqRNvtT6dEIIkEAgECzCTjQKQVbPk\ntfwpXo3J7tBOHhv53alexpIiBEkgEAgWIJ6frZ9EM5FTuJJZItkwyXwS3dBP9VKWDCFIAoFAMA+G\nYZAsmF1lDtnBVPb0SJNFMmFUQyWWO33SiCeKaGoQCASCeUjk4+S1Ai6bgiRJJPPJU70ksmqWFx99\nnvhEnO0fuYygq8V67YknHuXOO+/AbrfT3Bzk7/7u8zidTu6554c89NCvMAx4/euvqTJgBbj33p+S\nSiX5sz97L3fddScPPvgrbDYbn/zk39Da2sYXvvB5/v3fv7Ns9yUiJIFAIJiHseQYDsVh/RzJhE/h\nakzGkqPIsimQ8VxlO/aPf3w3//RPX+PWW7+Fx+Ph0Ud/V3f8RDnRaIT77/8573znuxkY6Ofhhx/k\n9tu/z6c+dRNPPvkEHR2dXHLJdr7//UoD1qVECJJAIBDMQygbQpZmH5WhTOiU121CGXNN8VCcf/vH\nr1eMn/jqV2/D5/OhqirhcJj29va64yfKue++n/G6170eWZZ58snHeeUrX43NZmPTps28//03AvCm\nN72Vu+++e9nuS6TsBAKBYB5mspVNDHk9z3Rqik5fl3XsU5+CH/2oeuzDiXDNNSqf+1yu5mvxvOnq\nHZuc4X2f/yCv7H61NX5CkiQeeOAX3H77N7n88is577wLAGqOnyjnhRee48Mf/p8ATEyMI8syn/jE\nR9E0lY985K/YsGEjbreb1tZWhoePsXJl35LeL4gISSAQCOZlbtOAQ3YQyk6fotWYpAspALrWd5M1\nslXjJ17/+mu45577SCQSPPjg7OC8ueMnygmFpunoMA1SDcNA13X+6Z/+lfe970a++MXZuUydnZ1L\nMmqiFiJCEggEgjrohk5Wy2GX7dYxSZJIF9IV5335y/DpT6dO2royanp2LWraSiEWCnmefvpJLrlk\nOzabjcsvv4oXX3ye1772dTXHT1RjjppoaWmlr28VkiRxzjnnMjExdjJuS0RIAoFAUI+0mkbTtarj\nWS17ClYzS0Y1U3mTAxPous6BkX1kMhmCwRa+9KVbCIXMCG7fvj309a2qO36inPJRExdfvJ1nn30a\ngKGhQTo6Oq3zJicnl2TURC1EhCQQCAR1iOdiFQ0NJUops1NBQStQ0E23iOauII9++7/4TeQ/+eAH\nP4TNZuNTn7qJz3zmk9jtDlpaWrjhhr/k+ef/UHP8xJlnbrF+Pv/8bezcuYONGzezZcvZPPPMk9x4\n4/8A4BOf+BsAstksoVDImoO01IjxE6cp4l5fnoh7fWmxP7yXZ8efsaamlvA7mnjLhj+xfj6Z9zqT\njXLPobtxKS7rmNvm5u2brjuh64bDIT796b/i9tu/X3W/Je6554c4HBLXXnv8nyXGTwgEAsFxkFEz\nNR/OmVMYIUWzUeySveJYLDdDPGc2NOS0HKOJkUVft7W1jWuueTM//OGdNV+fmprkyScf573vfe/i\nF90gQpAEAoGgDhk1U/u4lkHV1ZO8GpN4IVE1JNChOHly7AkKWoHhxDGeGX963mvU20d17bVv4/rr\n31PztY6OTr7ylW/gcDhqvr4UCEESCASCOuS0nPX/ByP7rQYH3dArDFdPJsk6nzuZmuR3ww8TzoSI\nZEOk8rWjuKya5d7DP13OJR43QpAEAoGgDhk1Q0bN8P/+8H/59u5v8oMD38cwDOyyg8nU+ClZU6qO\nl54kSQzHjzEwM4BNtnMgsp/fHXu4alzG7umdRLNRnh1/+pQ7TsxFCJJAIBDUIVNI88z4U0RzplvD\nrukdRHNRZEkmcYpMVhOF+p9rV+yk1RSyJPPC1HMMxQfZE9pVcc5MLopdsbM3tIeHBn9DQSss95Ib\nRgiSQCAQ1CGjZdgx9QKKpHD1ylcBMJoYBiA1jzAsJ4n8/N18pU28TsWJLMmMJUcrXi+lGhVZYSw5\nwi8H7uOFyeeWZ7GLRAi
"text/plain": [
"<matplotlib.figure.Figure at 0x7f4d4b8577b8>"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"pu.plot_results(results, average_group=True, split_fn=lambda _: '', shaded_std=False)"
]
},
{
"cell_type": "code",
"execution_count": 0,
"metadata": {
"colab": {},
"colab_type": "code",
"id": "YMA86VtfI8d5"
},
"outputs": [],
"source": []
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"name": "baselines_viz.ipynb",
"provenance": [],
"version": "0.3.2"
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
