diff --git a/is/UB3/ExerciseSheet3JP.ipynp b/is/UB3/ExerciseSheet3JP.ipynp new file mode 100644 index 0000000..d6eaabe --- /dev/null +++ b/is/UB3/ExerciseSheet3JP.ipynp @@ -0,0 +1,722 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "preamble": true + }, + "source": [ + "(Defining latex commands: not to be shown...)\n", + "$$\n", + "\\newcommand{\\norm}[1]{\\left \\| #1 \\right \\|}\n", + "\\DeclareMathOperator{\\minimize}{minimize}\n", + "\\newcommand{\\real}{\\mathbb{R}}\n", + "\\newcommand{\\normal}{\\mathcal{N}}\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#Intelligent Systems Sheet 3\n", + "Maximus Mutschler & Jan-Peter Hohloch" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Gaussian Algebra (25 Points)\n", + "\n", + "Prove that the product of two univariate (scalar) Gaussian distributions is a Gaussian again, i.e. show, by explicitly performing the required arithmetic transformations, that\n", + "\n", + "\\begin{equation}\n", + " \\normal(x;\\mu,\\sigma^2)\\normal(x;m,s^2) = \\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right].\n", + "\\end{equation}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Proof\n", + "\\begin{align*}\n", + " f(x) &= \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\cdot e^{\\frac{-(x-\\mu)^2}{2\\sigma^2}}\\cdot \\frac{1}{\\sqrt{2\\pi s^2}}\\cdot e^{\\frac{-(x-m)^2}{2s^2}}\\\\\n", + " &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\alpha}\\\\\n", + " \\text{where }\\alpha &= \\frac{s^2(x-\\mu)^2+\\sigma^2(x-m)^2}{2\\sigma^2 s^2}\\\\\n", + " &=\\frac{(s^2+\\sigma^2)x^2-2(s^2\\mu+\\sigma^2 m)x+s^2\\mu^2+\\sigma^2m^2}{2\\sigma^2s^2}\\\\\n", + " &=\\frac{x^2-2\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}x+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n", + " &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n", + " &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}+\\underbrace{\\frac{\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}_\\beta\\\\\n", + " \\beta&= \\frac{\\frac{s^4\\mu^2+s^2\\sigma^2\\mu^2+s^2\\sigma^2m^2+\\sigma^4m^2-\\left(s^4\\mu^2+2s^2\\sigma^2\\mu m+\\sigma^4m^2\\right)}{\\left(s^2+\\sigma^2\\right)^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n", + " &=\\frac{s^2\\sigma^2\\left(\\mu^2-2\\mu m+m^2\\right)}{2\\sigma^2s^2\\left(s^2+\\sigma^2\\right)}\\\\\n", + " &=\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}\\\\\n", + " \\Rightarrow f(x)&=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-(\\alpha-\\beta)-\\beta}\\\\\n", + " &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}\\cdot e^{-\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}}\\\\\n", + " &=\\frac{\\sqrt{2\\pi(s^2+\\sigma^2)}\\sqrt{2\\pi\\frac{\\sigma^2s^2}{s^2+\\sigma^2}}}{2\\pi s\\sigma}\\normal\\left[x;\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2},\\frac{\\sigma^2s^2}{s^2+\\sigma^2}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n", + " &=\\frac{\\sqrt{2\\pi\\cdot 2\\pi\\sigma^2s^2}}{2\\pi s\\sigma}\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n", + " &=\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\hspace{2cm} q.e.d.\n", + "\\end{align*}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Maximum Likelihood Estimator of Simple Linear Regression (25 Points)\n", + "\n", + "Derive the formula $\\mathbf{w}_{MLE} = (X^TX)^{-1}X^T\\mathbf{y}$ from the lecture, by calculating the derivative of $p(\\mathbf{y}\\,|X,\\mathbf{w}) = \\normal(\\mathbf{y}\\,|X\\mathbf{w}, \\sigma^2I)$ with respect to $\\mathbf{w}$, setting it to zero and solving it for $\\mathbf{w}$.\n", + "\n", + "\n", + "Note: _To refresh your linear algebra you might find it useful to have a look in [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/pdfs/Murray_cribsheet.pdf)._" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##wtf?\n", + "\n", + "$\\vec{w}=X^{-1}\\vec{y}$, zumindest wenn ich das ausrechne..." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Linear regression (50 Points)\n", + "\n", + "In this exercise you will perform a regression analysis on a toy dataset. You will implement ridge regression and learn how to find a good model through a comparative performance analysis.\n", + "\n", + "1) Download the [training set](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_train.csv)!
\n", + "2) Implement $\\mathbf{w}_{RIDGE}$ as a function of a given $X, \\mathbf{y}$ array and a regularization parameter $\\lambda$!" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + ":0: FutureWarning: IPython widgets are experimental and may change in the future.\n" + ] + } + ], + "source": [ + "# Loading the required packages\n", + "%matplotlib inline\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt\n", + "from IPython.html.widgets import interact" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "metadata": { + "collapsed": false, + "scrolled": false + }, + "outputs": [], + "source": [ + "def wRidge(X,y,lamb):\n", + " # Change the following line and implement the ridge regression estimator wRidge\n", + " a = (lamb*np.eye(X.shape[0])+X.dot(X.T))\n", + " b=np.linalg.inv(a)\n", + " c=b.T.dot(X)\n", + " return c.T.dot(y) # TODO pbyl wrong like this" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "3) Load \"ex1_train.csv\" into a numpy array! The first column in the csv file is $X$ and the second column is $\\mathbf{y}$, assign them to each variable!" + ] + }, + { + "cell_type": "code", + "execution_count": 73, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "# Read ex1_train.csv and assign the first column and \n", + "# second column to variables x and y respectively.\n", + "from numpy import genfromtxt\n", + "a=np.genfromtxt('ex1_train.csv',delimiter=' ')\n", + "x=a[:,0].T\n", + "y=a[:,1].T" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "4) Plot the training data with appropriate labels on each axes!" + ] + }, + { + "cell_type": "code", + "execution_count": 74, + "metadata": { + "collapsed": false, + "scrolled": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 74, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": [ + "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n", + "AAALEgAACxIB0t1+/AAAFmxJREFUeJzt3XuQZGd93vHvo1tWWIsEBmNZ7CIsQYJYQICREZcwQBzE\n", + "xoHIGDCOMRi05Spgl4oJZQQJkkM53BLC7lJ2EgHiZm4yBiOQZGSsNUpUXAQISxFClsKyKwwSV7Eb\n", + "dhNdfvnj9LJ9RnPpnpnu0z3z/VRNTZ/3dJ/+zZZ0nn7f97ynU1VIknTIEV0XIEmaLAaDJKnFYJAk\n", + "tRgMkqQWg0GS1GIwSJJaDAapA0nek+QNXdchzcVg0KqVZHeSp4/hfc5P8v4hX1a9n0GOvyvJS4ev\n", + "TFoag0Gr2cAn345kwOdN8t+gVchg0JqQ5MVJ/keStyb5YZL/neSsvv27krwxyReS3J7kE0nu09s3\n", + "k2TvrOPtTvL03jHOBZ6fZF+Sr87z/o9O8pUkP0nyYWBd3777JPlUktt6tV2c5KTevj8Gngy8o3f8\n", + "Hb327Un29Gq9OsmTVvifTGuYwaC15AzgBuDngbcA75q1/4XA7wEnAncCOxY4VgFVVZcB/xH4cFWt\n", + "r6pHz35ikmOATwDvBe4DXAQ8h8M9gfRq2dj7OQC8g+YNXgdcCby8d/xtvdd8EXhU73gfBC7qvY+0\n", + "bAaD1pJvVdW7qrlB2PuAE5P8Qm9fAe+rquur6qfAvweel2SQ4Z6w8LDQ44Gjqmp7Vd1VVR8DvnRo\n", + "Z1X9sKo+XlUHq2o/TdA8ZY73oO81f1ZVP6qqu6vqbcA/Av7xALVKizIYtJZ899CD3skf4Li+/f3D\n", + "RXuAo4H7rcD7/hLw7Vlt36J3sk9yryT/rTc8dTvwt8Dxs0KpNc+Q5N8muT7Jj5P8CDh+hWqVDAap\n", + "z8ZZj+8Avg/8H+Beh3YkORK4f99zF5sc/g5w0qy2B/W97lXAQ4Ezqup4mt5Cfy9kdig8GXg18Nyq\n", + "OqGq7gPczuCT2dKCDAapEeB3kjwsyb2A/wBc1Bt2uhFYl2RzkqOBf0czdHPId4GTFxh2ugq4M8m2\n", + "JEcn+Q3gcX37j6OZV7g9yX2B82a9/lbglL7t9TRzIN9PckyS1wP3XsofLc3FYNBaMdelqzXr8fuB\n", + "99B8wj8G2AZQVbcDLwPeCdwC7Kc97HRR7/cPklx9jzeuugP4DeDFwA+A5wEf63vK24FjaXonVwGX\n", + "zqptO/CbvSuW3g5c1vu5EdhNEyp7FvzrpSGkyy/qSfJu4F8At1XVI3pt5wPnAN/rPe3c3pUf0sgk\n", + "uQJ4f1W9u+tapK513WO4EDhrVlsBb6uqR/d+DAWNi2P0Eh0HQ1VdCfxojl3+D6ouuMJYovsew3y2\n", + "JvlaknclOaHrYrT6VdVTHUaSGpMYDH8KPBg4nWYS8D93W44krS1HdV3AbFV126HHSd4JXDz7OUns\n", + "8kvSElTVokP1ExcMSU6squ/0Ns8Grp3reYP8cZMqyflVdX7XdSyV9XfL+rszzbXD4B+qOw2GJB+i\n", + "WeV5v97dK88DZpKcTjMR+E3g9zssUZLWnE6DoapeMEezE4CS1KFJnHxeC3Z1XcAy7eq6gGXa1XUB\n", + "y7Sr6wKWaVfXBSzDrq4LGIdOVz4vVZKa5jkGSerCoOfOiZt8liStrGTTZtiwbfFnNgwGSVrFmlA4\n", + "cztccOqgN5VwjkGSVrUN25pQGJzBIEmr2vp1w77CoSRJq8LhcfT162DfQdi7o+q6S7quq3v7Dg77\n", + "CoNB0tRrj6MfsuWUZBOGw94dsOWUYYaTvFxV0liM8hN98szL4NJn3HPP5suqLnnmSrzHNGv+7Tdu\n", + "hUvP8nJVSRNh9J/o5xtHP+7Y5R97+oeperVeMhX3SpK0Vsx1ZcwFp8LmrcAKnGDnG0fff2C5R16L\n", + "w1RelSRpDEb7ib43jn5Tu+2cm2HPzuUfe75Q27h1+ceeTPYYJI3B6D7RQzNUkmyi6YEcd2xz3D07\n", + "p2GYahIZDJLGYK4rY1bqE33j0Dj6Sh3vsNGG2iQyGCSN3Gg/0Y/a6ENt0ni5qiQt4vDlntMWam2D\n", + "njsNBklaIwY9d3pVkiSpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1\n", + "GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqSWToMhybuT3Jrk2r62+ya5\n", + "PMmNST6T5IQua5SktabrHsOFwFmz2l4DXF5VDwU+29uWJI1Jp8FQVVcCP5rV/Czgvb3H7wX+1ViL\n", + "kqQ1rusew1weUFW39h7fCjygy2Ikaa05qusCFlJVlaTm2pfk/L7NXVW1ayxFSdKUSDIDzAz9uqo5\n", + "z7tjk+Rk4OKqekRv+wZgpqq+m+RE4Iqq+iezXlNVlbEXK0lTbNBz5yQOJX0SeFHv8YuAT3RYiySt\n", + "OZ32GJJ8CHgKcD+a+YTXA38JfBTYCOwGnldVP571OnsMkjSkQc+dnQ8lLYXBIEnDm+ahJElShwwG\n", + "SVKLwSBJajEYJEktE73ATdJhyabNsGEbrF8H+w7C3h1V113SdV1afQwGaQo0oXDmdrjg1MOtW05J\n", + "NmE4aKU5lCRNhQ3b2qEAzfbGrd3Uo9XMYJCmwvp1c7cfd+x469BaYDBIU2Hfwbnb9x8Ybx1aCwwG\n", + "aSrs3QFbbmq3nXMz7NnZTT1azbwlhjQlmgnojVub4aP9B2DPTieeNQzvlSRJavFeSZKkJTEYJEkt\n", + "BoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqQWg0GS1GIw\n", + "SJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWo7quoD5JNkN/AS4C7ijqs7o\n", + "tiJJWhsmNhiAAmaq6oddFyJJa8mkDyWl6wIkaa2Z5GAo4K+TXJ1ky+ydyTMvSzZt7qAuSVrVJnko\n", + "6YlV9Z0k9wcuT3JDVV15ePelz4AtpySbqLruks6qlKRVZmKDoaq+0/v9vSQfB84A+oLhfOCkU2Hf\n", + "9iQ/rapdHZQpSRMryQwwM/TrqmqxA/98Vf1gaWUtTZJ7AUdW1b4kPwd8BvijqvpMb381I00Az/vb\n", + "qo/OjLM+SZpGSaqqFp27HaTH8Pkk1wAXApfWYkmyMh4AfDwJNDX+2aFQuKf9B8ZQjyStGYP0GI4A\n", + "/hnwEuBxwEeBC6vqxtGXN29NvR7DOTfD57c5xyBJixu0x7BoMMw66NOADwA/B1wDnFtVVy25yiVq\n", + "guGZl8GenYaCJA1mxYIhyf2Afw38LnAr8E7gYuBRwJ9X1cnLrnZIg/5xkqTDVnKO4SqaXsKzq+qW\n", + "vvark/zXpRYoSZpMA80xVNXdY6pnIPYYJGl4g547F135PGmhIEkarUm+JYYkqQOLBkOSJ83R9sTR\n", + "lCNJ6togPYadc7S9Y6ULkSRNhnmvSkpyJvAE4P5J/oDDt8Bej0NQkrRqLXS56jE0IXBk7/chPwF+\n", + "c5RFSZK6M8jlqg+qqm+NqZ6BeLmqJA1vJVc+XzFHc1XV05Za3HIZDJI0vJVc+fzqvsfrgOcAdy61\n", + "MEnSZBvqJno/e1Hypap63AjqGfT97TFI0pBWrMeQ5L59m0cAvwLcexm1SZIm2CBDSV/h8Nel3Qns\n", + "Bl46qoIkSd1a0lBS1xxKkqThreRQ0rHAy4An0fQcrgT+tKoOLrtKSdLEGeRy1YtoFrV9gGb1828D\n", + "x1fVc0df3rw12WOQpCGt5DqG66vqtMXaxslgkKThrdj3MQBf6d036dCBHw98eTnFSZIm1yA9hhuA\n", + "hwJ7aeYYNgLfoLlCqarqkaMuco6a7DFI0pBWcuXzMzh8Z9VDao42SdIqMEiP4f1V9cLF2sbJHoMk\n", + "DW8l5xg2zTrwUcBjl1qYJGmyzRsMSV6bZB/wiCT7Dv0AtwGfHFuFkqSxGmQo6U1V9Zox1TMQh5Ik\n", + "aXgruY7hKRy+V9LPVNXnll7e8hgMkjS8lQyGT3E4GNYBZwBf9ot6JGm6rNjlqlX167MOvAHYvoza\n", + "JEkTbJCrkma7BXjYShciSZoMg9xddWff5hHA6XhLDElatQZZ+fxlDs8x3AV8sKr+5+hKkiR1aZDJ\n", + "52OBU2nC4aZJ+B4GJ58laXjLXvmc5Ogkb6G5ed57gfcBtyR5a5KjV65USdIkWWjy+a3AfYEHV9Vj\n", + "quoxwC8DJwD/aRzFSZLGb96hpCQ3AQ+tqrtntR8JfKOqTh1ZUclZwNuBI4F3VtWbZ+13KEmShrQS\n", + "N9G7e3YoAFTVXcA92ldKL3jeAZwFnAa8IImXx0rSmCwUDF9P8qLZjUleCNwwupI4g2aSe3dV3QF8\n", + "GHj2CN9Pa0SyaXPyzMuS5+1qfm/a3HVN0iRa6HLVlwN/keQlHF638FjgXsDZI6zpJJoJ70NuAX51\n", + "hO+nNaAJgTO3wwV9Q6BbTkk2UXXdJd1VJk2eeYOhqm5J8qvA04CH01yu+umq+uyIa1r4+llpSTZs\n", + "a4cCNNubtwIGg9RnwQVu1cxMf7b3My7fBjb0bW+g6TW0JDm/b3NXVe0abVmabuvXzd1+3LHjrUMa\n", + "nyQzwMywrxtk5fO4XQ08JMnJwD8AzwdeMPtJVXX+WKvSlNs3z8LM/QfGW4c0Pr0PzLsObSc5b5DX\n", + "LeUmeiNVVXcCrwD+Crge+EhVfb3bqjT99u6ALTe12865GfbsnPv50tq16C0xJpHrGLQUzQT0xq3N\n", + "8NH+A7BnpxPPWktW7It6JpHBIEnDW4kFbpKkNWgSJ5870ww1bNjWXMGy7yDs3eFQg6S1xmDocQGU\n", + "JDUcSvqZ+RZAbdzaTT2S1A2D4WdcACVJYDD0cQGUJIHB0McFUJIErmOYdVwXQElavVzgJklqcYGb\n", + "JGlJDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMk\n", + "qcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWo5qusCpEOSTZthwzZYvw72HYS9O6quu6TruqS1\n", + "xmDQRGhC4cztcMGph1u3nJJswnCQxsuhJE2IDdvaoQDN9sat3dQjrV0GgybE+nVztx937HjrkGQw\n", + "aELsOzh3+/4D461DksGgCbF3B2y5qd12zs2wZ2c39UhrV6qq6xqGlqSqKl3XoZXVTEBv3NoMH+0/\n", + "AHt2OvEsrZxBz50TFwxJzgfOAb7Xazq3qi6b9RyDQZKGNOi5cxIvVy3gbVX1tq4LkaS1aFLnGOwN\n", + "SFJHJjUYtib5WpJ3JTmh62IkaS3pZI4hyeXAL86x63XA5zk8v/AG4MSqeums1xfwR31Nu6pq1whK\n", + "laSplWQGmOlrOm8qJ5/7JTkZuLiqHjGr3clnSRrSoOfOiRtKSnJi3+bZwLVd1SJJa9EkXpX05iSn\n", + "01yd9E3g9zuuR5LWlIkeSpqPQ0mSNLypHUqSJHXLYJAktRgMkqQWg0GS1GIwSJJaDAZJUovBIElq\n", + "MRgkSS2TuPJZE6r5hrUN22D9uuY7mvfu8BvWpNXHYNBAmlA4cztccOrh1i2nJJswHKTVxaEkDWjD\n", + "tnYoQLO9cWs39UgaFYNBA1q/bu72444dbx2SRs1g0ID2HZy7ff+B8dYhadQMBg1o7w7YclO77Zyb\n", + "Yc/ObuqRNCredlsDayagN25tho/2H4A9O514lqbHoOdOg0GS1gi/j0GStCQGgySpxWCQJLUYDJKk\n", + "FoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWox\n", + "GCRJLQaDJKnFYJAktXQSDEmem+R/JbkryWNm7Ts3yd8nuSHJP++iPklay7rqMVwLnA18rr8xyWnA\n", + "84HTgLOAP0my6no1SWa6rmE5rL9b1t+daa59GJ2cdKvqhqq6cY5dzwY+VFV3VNVu4CbgjLEWNx4z\n", + "XRewTDNdF7BMM10XsEwzXRewTDNdF7AMM10XMA6T9mn8l4Bb+rZvAU7qqBZJWpOOGtWBk1wO/OIc\n", + "u15bVRcPcahaoZIkSQNIVXfn3SRXAK+qqq/0tl8DUFVv6m1fBpxXVV+Y9TrDQpKWoKqy2HNG1mMY\n", + "Qn+RnwQ+mORtNENIDwG+OPsFg/xhkqSl6epy1bOT7AUeD3w6yaUAVXU98FHgeuBS4GXVZZdGktag\n", + "ToeSJEmTZ9KuShpYkjck+VqSa5J8NsmGrmsaRpK3Jvl672/4iyTHd13TMBZapDipkpzVWzj590n+\n", + "sOt6hpXk3UluTXJt17UMK8mGJFf0/pu5Lsm2rmsaRpJ1Sb7QO99cn+SNXde0FEmOTPLVJAteADS1\n", + "wQC8paoeVVWnA58Azuu6oCF9Bnh4VT0KuBE4t+N6hjXnIsVJleRI4B00CydPA16Q5GHdVjW0C2nq\n", + "n0Z3AP+mqh5OM4T88mn696+qg8BTe+ebRwJPTfKkjstailfSDNUvOFQ0tcFQVfv6No8Dvt9VLUtR\n", + "VZdX1d29zS8AD+yynmEtsEhxUp0B3FRVu6vqDuDDNAsqp0ZVXQn8qOs6lqKqvltV1/Qe7we+TrNu\n", + "aWpU1U97D48BjgR+2GE5Q0vyQGAz8E7aF/3cw9QGA0CSP06yB3gR8Kau61mGlwCXdF3EKncSsLdv\n", + "28WTHUlyMvBomg9EUyPJEUmuAW4FruhdLDNN/gvwauDuxZ440cGQ5PIk187x8y8Bqup1VbUReA/N\n", + "Hz1RFqu/95zXAf+vqj7YYalzGqT+KeJVFhMgyXHAnwOv7PUcpkZV3d0bSnog8E+n6b5JSX4duK2q\n", + "vsoivQWYjHUM86qqXxvwqR9kAj9xL1Z/khfTdO2ePpaChjTEv/80+DbQf4HCBtq3X9GIJTka+Bjw\n", + "gar6RNf1LFVV3Z7k08CvALs6LmdQTwCelWQzsA64d5L3VdXvzvXkie4xLCTJQ/o2nw18tataliLJ\n", + "WTTdumf3Jram2TQsOLwaeEiSk5McQ3MX3092XNOakSTAu4Drq+rtXdczrCT3S3JC7/GxwK8xReec\n", + "qnptVW2oqgcDvwX8zXyhAFMcDMAbe8Ma19Dc8fBVHdczrJ00k+aX9y4f+5OuCxrGfIsUJ1VV3Qm8\n", + "AvgrmqsyPlJVX++2quEk+RBwFfDQJHuT/F7XNQ3hicDv0FzN89XezzRdYXUi8De9880XgIur6rMd\n", + "17QcCw6tusBNktQyzT0GSdIIGAySpBaDQZLUYjBIkloMBklSi8EgSWoxGKRZkqz4rRqSPCjJC1b6\n", + "uNIoGAzSPY1icc+Dgd8ewXGlFWcwSPNIMpNkV5KLel+q9IG+fbuTvDnJ3/W+wOWUXvt7kjyn73mH\n", + "bg//JuDJvRW/r5z1Pmcn+eve4xOTfCPJL4z+L5TmZjBICzud5stNTgN+OckTeu0F/LiqHknzBUBv\n", + "72ufyx8CV1bVo6tqe/+Oqvo48J0krwD+O/D6qrpthf8OaWAGg7SwL1bVP1Rz75hrgJP79n2o9/vD\n", + "wJmLHGexGw1upfkWv4NV9ZGlFCqtFINBWtj/7Xt8F/Pfqv5QT+FOev9fJTmC5tu+BrGhd/wH9O5E\n", + "KnXGYJCW7vl9v6/qPd4NPLb3+FnA0b3H+4D1cx0kyVE0t6T+LeAG4A9GUKs0sIn+oh6pIzXP49nu\n", + "k+RrwEHg0KWoFwB/2bs982XAoUtfvwbc1Wu/cNY8w7nA56rqqiR/B3wpyaeq6hsr8cdIw/K229IS\n", + "JPkm8NiqmqovhJcG4VCStDR+otKqZY9BktRij0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySp5f8D\n", + "oJt+CXUlyycAAAAASUVORK5CYII=\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Plot the input data here\n", + "plt.plot(x,y,'o')\n", + "plt.title('Input data')\n", + "plt.ylabel('Output y')\n", + "plt.xlabel('Input x')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "5) Implement a function which constructs features upto a input polynomial degree $d$!
\n", + "Note: _Constructing higher polynomial features is similar to what you implemented in Exercise 3 (SVM) of the previous exercise sheet._" + ] + }, + { + "cell_type": "code", + "execution_count": 80, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "def construct_poly(x,d):\n", + " ## Implement a method which given an array of size N, \n", + " ## returns an array of dimension (N,d)\n", + " return (np.ones((d,1))*np.asmatrix(x)).T" + ] + }, + { + "cell_type": "code", + "execution_count": 86, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[-3. -3. ]\n", + " [-2.5 -2.5]\n", + " [-1. -1. ]\n", + " [-0.5 -0.5]\n", + " [ 0. 0. ]\n", + " [ 1. 1. ]\n", + " [ 2. 2. ]\n", + " [ 2.5 2.5]\n", + " [ 3. 3. ]\n", + " [ 4. 4. ]]\n", + "[[ 52.75 52.75]\n", + " [ 52.75 52.75]]\n" + ] + }, + { + "data": { + "text/plain": [ + "matrix([[ 1.51220623, 1.51220623]])" + ] + }, + "execution_count": 86, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "#try it out\n", + "X=construct_poly(x,2)\n", + "print(X)\n", + "print(X.T.dot(X))\n", + "wRidge(X,y,1)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "6) Implement the Mean Squared Error Loss (MSE) as a function of the predicted and true values of the target variable!
" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "def MSE(y_predict,y_true):\n", + " ## Implement mean squared error for a given input y and its predictions.\n", + " return ((y_predict-y_true)**2)/y_predict.shape[0]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "7) By comparing the MSE find the degree $d$ for the polynomial that fits the training data best! You might find it useful to use the code below to interactively change the variable $d$, set $\\lambda = 1$ and keep it fixed. Plot the error as a function of different values of $d$!
" + ] + }, + { + "cell_type": "code", + "execution_count": 95, + "metadata": { + "collapsed": false, + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[-3. -2.5 -1. -0.5 0. 1. 2. 2.5 3. 4. ]\n", + "(10, 1) (10,)\n", + "[[ 7.24825764 -79.61751096 -163.32510817 -124.420806 -64.36110538\n", + " 64.00498155 130.78610485 134.65832608 128.64828886 140.65345344]\n", + " [ 2.38125049 -75.48579685 -150.52183853 -115.64776572 -61.80985436\n", + " 53.25835177 113.12139072 116.59247535 111.20503839 121.96654702]\n", + " [ -12.21977095 -63.09065449 -112.11202963 -89.32864491 -54.15610128\n", + " 21.01846246 60.12724836 62.39492317 58.87528698 65.90582776]\n", + " [ -17.0867781 -58.95894038 -99.30875999 -80.55560464 -51.60485026\n", + " 10.27183269 42.46253424 44.32907244 41.4320365 47.21892134]\n", + " [ -21.95378525 -54.82722626 -86.50549036 -71.78256437 -49.05359924\n", + " -0.47479708 24.79782012 26.26322172 23.98878603 28.53201492]\n", + " [ -31.68779954 -46.56379802 -60.89895109 -54.23648382 -43.95109719\n", + " -21.96805662 -10.53160812 -9.86847974 -10.89771491 -8.84179791]\n", + " [ -41.42181383 -38.30036979 -35.29241182 -36.69040328 -38.84859514\n", + " -43.46131617 -45.86103637 -46.00018119 -45.78421585 -46.21561075]\n", + " [ -46.28882098 -34.16865567 -22.48914219 -27.91736301 -36.29734412\n", + " -54.20794594 -63.52575049 -64.06603192 -63.22746633 -64.90251717]\n", + " [ -51.15582813 -30.03694156 -9.68587255 -19.14432273 -33.74609309\n", + " -64.95457571 -81.19046461 -82.13188264 -80.6707168 -83.58942359]\n", + " [ -60.88984242 -21.77351332 15.92066671 -1.59824219 -28.64359104\n", + " -86.44783525 -116.51989285 -118.2635841 -115.55721774 -120.96323643]]\n" + ] + }, + { + "ename": "TypeError", + "evalue": "only length-1 arrays can be converted to Python scalars", + "output_type": "error", + "traceback": [ + "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)", + "\u001b[1;32m\u001b[0m in \u001b[0;36mplot\u001b[1;34m(n)\u001b[0m\n\u001b[0;32m 8\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 9\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"MSE %f\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 11\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n", + "\u001b[1;31mTypeError\u001b[0m: only length-1 arrays can be converted to Python scalars" + ] + }, + { + "data": { + "image/png": [ + "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n", + "AAALEgAACxIB0t1+/AAAGDdJREFUeJzt3Xm43XV97fH3SpgiiMwyFigiJGFIQBDRylHEG7ncUrwO\n", + "0Am1rdw6IZMRuVcClkIgBKoWikJEKYMoKlBAiUgERaYyBUNAVBSUIt4ql0KBENb947uxh5jh7LPP\n", + "73z3sF7Pcx7O3mfv81vy8Cx/57N/v+9XtomIiP4yoXaAiIgYeyn3iIg+lHKPiOhDKfeIiD6Uco+I\n", + "6EMp94iIPtRxuUuaJ+kxSQuHPTdL0iOS7mx9zej0OBERMXJjceb+BWDZ8jYw1/b01tc3x+A4EREx\n", + "Qh2Xu+0bgd8s50fq9HdHRMToNDlz/7CkuyWdJ2m9Bo8TERHLaKrczwa2BaYBjwKnN3SciIhYjtWa\n", + "+KW2f/Xi95LOBa5c9jWSsqhNRMQo2F7l2LuRcpe0me1HWw8PAhYu73UjCditJM2yPat2jtFK/rqS\n", + "v55ezg4jPzHuuNwlXQzsA2wk6WHgeGBI0jTKVTM/BQ7r9DgRETFyHZe77UOW8/S8Tn9vRESMXu5Q\n", + "Hb0FtQN0aEHtAB1aUDtAhxbUDtChBbUDdGBB7QDjQbU265DkXp65R0TUMNLuzJl7REQfSrlHRPSh\n", + "lHtERB9KuUdE9KGUe0REH0q5R0T0oZR7REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oZR7\n", + "REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oY43yI6IiOZJTAT+aqSvT7lHRHQ5iX2BM4Df\n", + "jvQ9KfeIiC4l8WpgDrATcAzwNeCFkbw3M/eIiC4jsb7EXOAm4HvAFJvLbDzS35Fyj4joEhKrSXwQ\n", + "WAysDUy1OdXmmXZ/V8YyERFdQGIGMBf4JbCfzT2d/L6Ue0RERRJTgNOB7YCjgSvbGb+sSMYyEREV\n", + "SGwo8Rngu8C1wE42V4xFsUPKPSJiXEmsIfFRylxdwGSbM2yeG8vjZCwTETEOJAQcQBnB/BjYx2ZR\n", + "U8dLuUdENExiF8qHpZsDh9tc0/QxM5aJiGiIxCYS5wDzgW8Au45HsUPKPSJizEmsKXEMsAh4CtjR\n", + "5rM2S8YrQ8YyERFjpDVXfztwKvBDYG+bB2pkSblHRIwBid0oi3ttABxm8+2aeTKWiYjogMRmEvOA\n", + "q4GLgOm1ix3GoNwlzZP0mKSFw57bQNJ8SQ9IulbSep0eJyKim0hMkvgEsBB4HNjB5hyb5ytHA8bm\n", + "zP0LwIxlnvs4MN/2q4HrWo8jInqehCQOBu4DdgdeazPT5onK0V5Cdud3ukraBrjS9s6tx4uBfWw/\n", + "JmlTYIHtHZd5j22r44NHRIwTiT0pc/VJwBE23x3/DCPrzqZm7q+0/Vjr+8eAVzZ0nIiIxklsKXEB\n", + "5Vr1c4E9ahR7Oxq/Wsa2JS33zwNJs4Y9XGB7QdN5IiJGSmJtyg5IHwbOpszVnxzfDBoChtp+X4Nj\n", + "mSHb/yZpM+D6jGUioldITAD+FDgZ+D4w0+ZndVMVtccyVwCHtr4/lPKnTERE15PYG7gZ+AhwsM3B\n", + "3VLs7ej4zF3SxcA+wEaU+fongcuBS4E/AB4C3mX7t8u8L2fuEdE1JLYGZgOvB44FLrJHthn1eBpp\n", + "d47JWGY0Uu4R0Q0kXk65XPt/AZ8G5tg8VTfVitUey0REdDWJCRLvpWyasRVlxcYTurnY25G1ZSJi\n", + "4EjsQ7le/RngIJtbK0cacyn3iBgYEttRVmzcHZgJXDpWe5Z2m4xlIqLvSbxC4lTgFuAOyr6lX+7X\n", + "YoeUe0T0MYmJEocB9wMbAjvbnGTzn5WjNS5jmYjoSxJvoexb+htgf5s7KkcaVyn3iOgrEq8G5gBT\n", + "KUsHfL2fxy8rkrFMRPQFifUlzgBuAm4Epth8bRCLHVLuEdHjJFaX+BDlevVJlFI/zebZytGqylgm\n", + "InqWxAzKXP2XwFtsFq7iLQMj5R4RPUdiCnA6sB1wFPAvgzp+WZGMZSKiZ0hsJPFZ4LvAt4CdbK5M\n", + "sf++lHtEdD2JNSSOoOxbamBHmzNtnqscrWtlLBMRXUtCwAGUEcyDwD42i+qm6g0p94joShK7UD4s\n", + "3Rw43OaaypF6SsYyEdFVJDaROAeYD3ydshRvir1NKfeI6AoSa0p8DFgEPEWZq/+jzZLK0XpSxjIR\n", + "UVVrrv52ylK89wJ72zxQN1XvS7lHRDUSu1E2zVgfeL/NdZUj9Y2MZSJi3ElsJjEPuAq4EJieYh9b\n", + "KfeIGDcSkySOo4xfHqfM1T9ns7RytL6TsUxENK41V383cApwO7CHzU/qpupvKfeIaJTEnpS5+iTg\n", + "L21uqBxpIGQsExGNkNhS4gLKteqfB16TYh8/KfeIGFMSa0vMAu4GfgbsYHO+zQt1kw2WjGUiYkxI\n", + "TAD+DPh7yk5I021+XjfV4Eq5R0THJPYGzqSs2Phum5sqRxp4KfeIGDWJrYHZwOuBjwMXZ/zSHTJz\n", + "j4i2Sbxc4iTgXylrwexgc2GKvXuk3CNixCQmSrwPuB/YkrJi44k2T1eOFsvIWCYiRkRiH8r16k8D\n", + "B9rcVjlSrETKPSJWSmI7yoqNuwEzga9kz9Lul7FMRCyXxCskTgVuoSwZMNnm0hR7b0i5R8RLSKwm\n", + "cRhlrr4BsJPNyTbPVI4WbchYJiJ+R2I/yr6lvwbeZnNn5UgxSo2Wu6SHgP8HLAWW2N6zyeNFxOhI\n", + "7ADMASYDRwOXZ/zS25o+czcwZPvfGz5ORIyCxPrAJ4E/p9yM9A6bZ+umirEwHjN3jcMxIqINEqtL\n", + "fJgyV18LmGozJ8XeP8bjzP3bkpYC59j+fMPHi4hVkHgbZa7+MLCvzcLKkaIBTZf7620/KmljYL6k\n", + "xbZvbPiYEbEcElOB04FtgSOBqzNX71+NlrvtR1v/fFzS14E9KUuBAiBp1rCXL7C9oMk8EYNIYiPg\n", + "BOCdwEnAWTZL6qaKkZI0BAy1/T67mf/jlvQyYKLtJyWtDVwLnGD72tbPbTvz+IiGSKwBfAg4FrgY\n", + "OMHm/9ZNFZ0aaXc2eeb+SuDrkl48zoUvFntENKe1GfUfUy5tfAB4o819dVPFeGvszH2VB86Ze8SY\n", + "k9iV8mHppsCRNt+qHCnG2Ei7M8sPRPQBiVdKfI4y/vwqZSneFPsAS7lH9DCJtSRmAj8EnqRsmnG2\n", + "zfOVo0VlWVsmoge15ur/k7IU7z3A62x+VDdVdJOUe0SPkdidsmnGK4C/tvlO5UjRhTKWiegREptL\n", + "fAH4F+BLwG4p9liRlHtEl5OYJPG/gYXAY5S5+rk2SytHiy6WsUxEl2rN1Q8GTgFuBfaw+UndVNEr\n", + "Uu4RXUhiL8pcfQ3gL2xuqBwpekzGMhFdRGIriQuBy4BzKGfrKfZoW8o9ogtIrCNxInAX8BPKXP18\n", + "mxcqR4selbFMREUSE4C/oKzWeAMw3ebndVNFP0i5R1Qi8QbKXH0p8E6bH1SOFH0k5R4xziS2pexX\n", + "uhet5Xgzfomxlpl7xDiRWFfiZOB24F5gR5sLU+zRhJR7RMMkJkr8NbAY2BzYxeZEm6crR4s+lrFM\n", + "RIMk3kSZq/8HcKDNbZUjxYBIuUc0QOJVwGnANGAm8JVsRh3jKWOZiDEksZ7EHOBm4BZgss2lKfYY\n", + "byn3iDEgsZrE31Lm6usBO9mcYvNM5WgxoDKWieiQxFsp+5Y+DsywuatypIiUe8RoSewIzAF2BI4G\n", + "Ls/4JbpFxjIRbZLYQOIfgO8B1wNTbb6RYo9uknKPGCGJ1SU+Qpmrr075sPR0m2crR4v4PRnLRKxC\n", + "a9OM/YHTgZ8Db7a5t26qiJVLuUeshMRUyoelWwNHAVdn/BK9IGOZiOWQ2FjiLMpM/SpgZ5urUuzR\n", + "K1LuEcNIrCFxFLAIWEJZ3OvTNksqR4toS8YyEfxurn4gZcmA+4E/sllcN1XE6KXcY+BJ7EpZ3GsT\n", + "4IM211aOFNGxjGViYElsKvF54FrgK8C0FHv0i5R7DByJtSQ+Ttkw4wnKZtRn2zxfOVrEmMlYJgZG\n", + "a67+DuBU4C5gL5sH66aKaEbKPQaCxO7AmcDLgffZXF85UkSjMpaJviaxucT5wJXAF4HdU+wxCFLu\n", + "0ZckXibxf4B7gEcp16ufa7O0crSIcZGxTPSV1lz9EOAUym5Ie9j8tG6qiPHX2Jm7pBmSFkv6kaSZ\n", + "TR0n4kUSewE3AUcCf2bzrhR7DCrZY79UhqSJlLv83gL8ArgNOMT2fcNeY9sa84PHwJHYinKmvg9w\n", + "HHCBzQt1U0U0Y6Td2dSZ+57Ag7Yfsr0EuIRya3fEmJFYR+JEymWND1KuV/9iij2iuXLfAnh42ONH\n", + "Ws9FdExigsShlE0ztgOm2xxv81TlaBFdo6kPVLMsajRC4g2U69WfB95hc3PlSBFdqaly/wWw1bDH\n", + "W1HO3l9C0qxhDxfYXtBQnuhxEtsCs4G9gJnAJVlbPQaBpCFgqO33NfSB6mqUD1T3BX4J3Eo+UI1R\n", + "kFgXOBZ4P2Xlxrk2T9dNFVHPSLuzkTN3289L+hDwLWAicN7wYo9YFYmJwHuBEymrNu5s88u6qSJ6\n", + "RyNn7iM6cM7cYwUk3kQ5S38SOMLm9sqRIrpG1TP3iNGQeBVlJ6RpwDHAZZmrR4xO1paJ6iTWk5hD\n", + "WS7gZmCyzVdT7BGjl3KPaiRWk/hbyvXq6wJTbWbbPFM5WkTPy1gmqpB4KzAX+BUww+auypEi+krK\n", + "PcaVxI7AHGAH4GjgioxfIsZexjIxLiQ2kPgH4EbgO5QRzOUp9ohmpNyjURKrS3yEMldfDZhiM9fm\n", + "ucrRIvpaxjLRiNamGfsDpwM/A95sc2/dVBGDI+UeY05iKuXD0q0pG2dck/FLxPjKWCbGjMTGEmcB\n", + "1wNXUZYMuDrFHjH+Uu7RMYk1JY4GFgHPUTaj/rTNksrRIgZWxjIxaq25+oGUSxvvA95gc3/dVBEB\n", + "KfcYJYlplMW9NgY+YHNt5UgRMUzGMtEWiU0lzgW+CXwZmJZij+g+KfcYEYm1JI4F7gV+Q5mr/5PN\n", + "85WjRcRyZCwTK9Waq78DOBW4E9jL5sG6qSJiVVLusUISr6HM1V8OvM/m+sqRImKEMpaJ3yOxhcQX\n", + "gSuA84HdU+wRvSXlHr8j8TKJTwL3AL8AdrA5z2Zp5WgR0aaMZQKJCcAhwCnATZQz9YeqhoqIjqTc\n", + "B5zE6yhz9YnAITbfqxwpIsZAxjIDSmJriYuBrwBnAa9NsUf0j5T7gJFYR+JTwB3AA5S5+pdsXqgc\n", + "LSLGUMp9QEhMkHgPcD+wLeXO0uNtnqqbLCKakJn7AJB4I2Wu/hzwdptbKkeKiIal3PuYxB9S7izd\n", + "E5gJXJK11SMGQ8YyfUhiXYnZwG3AXZR1YC5OsUcMjpR7H5GYKPE3lLn6JpSdkP7O5unK0SJinGUs\n", + "0yck3kyZqz8BHGDzr5UjRURFKfceJ7E9cBqwC/Ax4LKMXyIiY5keJbG+xFzgB62vKTZfTbFHBKTc\n", + "e47EahIfABYD6wBTbWbbPFM5WkR0kYxleojEfwPmAo8Bb7W5u3KkiOhSKfceIDEZOB3YHjgauCLj\n", + "l4hYmYxlupjEhhKfAW4A5lNGMJen2CNiVVLuXUhiDYmPUubqE4DJNmfYPFc5WkT0iEbKXdIsSY9I\n", + "urP1NaOJ4/QbCUkcACwE3gYM2XzQ5teVo0VEj2lq5m5gru25Df3+viOxM+XD0i2BI4BrMn6JiNFq\n", + "ciyjBn9335DYWOJs4DrgcmAXm6tT7BHRiSbL/cOS7pZ0nqT1GjxOT5JYU+IY4D7gWcriXp+1WVI5\n", + "WkT0AdmjO0GUNB/YdDk/Og64GXi89fhTwGa2/2qZ9xs4YdhTC2wvGFWYHiIh4E+AOcAi4Gib++um\n", + "iohuJWkIGBr21PG2VzkZGXW5j5SkbYArbe+8zPMeScB+IjGdsrjXhsCRNvMrR4qIHjPS7mzqapnN\n", + "hj08iHL1x8CS2FTiPOAa4GJgeoo9IprU1NUysyVNo1w181PgsIaO09UkJlGufDkSmEfZjPqJuqki\n", + "YhA0PpZZ4YH7eCzTmqu/C5gN3AF8zObBuqkioh+MtDuztswYk9iDMldfG3iPzYK6iSJiEGX5gTEi\n", + "sYXElyjXqs8DXpNij4haUu4dkniZxCeBe4CHKXP1eTZLK0eLiAGWscwoSUwA/hT4e+AmYHebh6qG\n", + "iohoSbmPgsTrgDMpSywcYvP9ypEiIl4i5d4Gia2BU4A/Ao4FLrR5oW6qiIjfl5n7CEisI/F3lMsa\n", + "76fM1S9IsUdEt0q5r4TEBIn3Ugp9a2CazSybpypHi4hYqYxlVkDijZTr1Z8FDrK5tXKkiIgRS7kv\n", + "Q+IPgVOBPYCZwJeztnpE9JqMZVok1pWYDdwG3ElZX/2SFHtE9KKBL3eJiRJ/Q5mrbwzsbHOSzX9W\n", + "jhYRMWoDPZaR2Jeyb+lvgf9uc0flSBERY2Igy11ie8pOSDsDxwBfy/glIvrJQI1lJNaXmAv8APg+\n", + "MMXmshR7RPSbgSh3idUkPgAspizFO9XmVJtnKkeLiGhE349lJGYApwOPAvvZ3FM5UkRE4/q23CUm\n", + "U0r9VcDRwJUZv0TEoOi7sYzEhhKfAW4A5gM72VyRYo+IQdI35S6xhsRHgfsoS/FOtjnD5rnK0SIi\n", + "xl3Pj2Vam1EfQBnB/BgYsllUN1VERF09Xe4SO1NuQtoCONzmmsqRIiK6Qk+OZSQ2kfgn4NvAN4Bd\n", + "U+wREf+lp8pdYk2JY4BFwNOUxb3+0WZJ5WgREV2lJ8Yyrbn6QcBpwA+BvW0eqJsqIqJ7dX25S0yn\n", + "bJqxAfB+m+sqR4qI6HpdO5aR2ExiHnANcBEwPcUeETEyXVfuEpMkPgEsBB6nbEb9OZullaNFRPSM\n", + "rhnLtObq7wJmA7cDr7X5cd1UERG9qSvKXWJPylx9EnCozXcrR4qI6GlVxzISW0pcQLlW/VxgjxR7\n", + "RETnap+53w2cTZmrP1k5S0RE35BdZ7FESQZvY/OzKgEiInqQJNvWKl9Xs9xHEjAiIv7LSLuz6y6F\n", + "jIiIzo263CW9U9IPJS2VtNsyPztW0o8kLZb01s5jRkREOzo5c19IWe/lhuFPSpoCvBuYAswAzpLU\n", + "d38hSBqqnaETyV9X8tfTy9nbMerStb3Y9vIW7zoQuNj2EtsPAQ8Ce472OF1sqHaADg3VDtChodoB\n", + "OjRUO0CHhmoH6MBQ7QDjoYkz6s2BR4Y9foSymUZERIyTlV7nLmk+sOlyfvQJ21e2cZxsTh0RMY46\n", + "vhRS0vXAUbbvaD3+OIDtU1qPvwkcb/uWZd6Xwo+IGIWRXAo5VneoDj/QFcBFkl7c23R74NbRhIuI\n", + "iNHp5FLIgyQ9DOwFXCXpGgDbi4BLKVvhXQN8wLXulIqIGFDV7lCNiIjmVF4VUp+SdLekuyRdJ2mr\n", + "mnnaJek0Sfe1/jd8TdIramdqx8puROtWkma0bo77kaSZtfO0S9I8SY9JWlg7S7skbSXp+tZ/M/dK\n", + "+kjtTO2QtJakW1p9s0jSybUzjYakiZLulLTSi1pq31x0qu1dbU+jLPt7fOU87boWmGp7V+AB4NjK\n", + "edq13BvRupWkicBnKTfHTQEOkTS5bqq2fYGSvxctAY6wPZUyjv1gL/37t/0M8KZW3+wCvEnSGyrH\n", + "Go3DKWPvlY5dqpa77eHL/K4D/LpWltGwPd/2C62HtwBb1szTrpXciNat9gQetP2Q7SXAJZSb5nqG\n", + "7RuB39TOMRq2/832Xa3v/wO4j3JfS8+w/XTr2zWAicC/V4zTNklbAvtT9r9Y6UUptc/ckXSSpJ8D\n", + "hwKn1M7TgfcBV9cO0ee2AB4e9jg3yFUiaRtgOuWkpmdImiDpLuAx4PrWBSC95AzgGOCFVb2w8XKX\n", + "NF/SwuV8/Q8A28fZ/gPgfErwrrKq/K3XHAc8Z/uiilGXayT5e0g+/e8CktYBvgoc3jqD7xm2X2iN\n", + "ZbYE3thL68xIOgD4le07WcVZO4zDTky29xvhSy+iC898V5Vf0nsofybtOy6B2tTGv/9e8Atg+Ifu\n", + "W/HSpS6iYZJWBy4D/tn2N2rnGS3bT0i6CngNsKBynJHaG/hjSfsDawHrSvqS7b9c3otrXy2z/bCH\n", + "BwJ31soyGpJmUP5EOrD1YU0v64Wbym4Htpe0jaQ1KKuPXlE508CQJOA8YJHtM2vnaZekjSSt1/p+\n", + "ErAfPdQ5tj9heyvb2wIHA99ZUbFD/Zn7ya0RwV2UldqOqpynXZ+hfBA8v3Vp0lm1A7VjRTeidSvb\n", + "zwMfAr5FuVrgy7bvq5uqPZIuBm4CXi3pYUnvrZ2pDa8H/pxylcmdra9euvJnM+A7rb65BbjS9nWV\n", + "M3VipWPK3MQUEdGHap+5R0REA1LuERF9KOUeEdGHUu4REX0o5R4R0YdS7hERfSjlHhHRh1LuERF9\n", + "6P8Do+dj0Sq+0gEAAAAASUVORK5CYII=\n" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "##This function provides an interactive mode to change polynomial degree. \n", + "@interact(n=[1,16])\n", + "def plot(n):\n", + " X = construct_poly(x,n)\n", + " w = wRidge(X,y,1.0)\n", + " plt.plot(x,X.dot(w.T))\n", + " print(x)\n", + " print(X.dot(w.T).shape, y.shape)\n", + " print(MSE(X.dot(w.T),y))\n", + " plt.title(\"MSE %f\" % MSE(X.dot(w.T),y))\n", + " plt.plot(x,y)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "8) Apply models with different values of $d$ after being trained on the training dataset, to the test data available [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_test.csv). Compare the errors on the test data to the ones from the training by plotting the error curves as functions of the polynomial degree in a single plot! What do you conclude?
" + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "text/plain": [ + "array([[ 1.00000000e+02, 1.00000000e+01, 1.00000000e+00],\n", + " [ 1.00000000e+01, 1.00000000e+00, 1.00000000e-01],\n", + " [ 1.00000000e+00, 1.00000000e-01, 1.00000000e-02]])" + ] + }, + "execution_count": 82, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "A.T.dot(A)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "collapsed": false + }, + "outputs": [], + "source": [ + "import scipy\n", + "from scipy import linalg\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1\n", + "[[ 9.80101960e-03 9.80101960e-04 9.80101960e-05]\n", + " [ 9.80101960e-04 9.80101960e-05 9.80101960e-06]\n", + " [ 9.80101960e-05 9.80101960e-06 9.80101960e-07]]\n", + "True\n" + ] + } + ], + "source": [ + "A = np.array([[10,1,0.1],[10,1,0.2],[10,1.1,0.7]])\n", + "a = np.array([[10,1,0.1]])\n", + "A = a.T.dot(a)\n", + "\n", + "B,rank = linalg.pinv(A,return_rank=True)\n", + "print rank\n", + "print B\n", + "print np.allclose(A,A.dot(B.dot(A)))" + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "## Read test data here" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "9) With a fixed optimal $d$, change the value of $\\lambda$ to one of the following values $[0.1, 1.0, 10.0]$ and find the minimum MSE!
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Hand in printed copy of completed notebook." + ] + } + ], + "metadata": { + "annotations": { + "author": "", + "categories": [ + "intelligent-systems-1-2015" + ], + "date": "2015-04-30", + "location": "Beginning of next lecture", + "parent": "IS_SS2015", + "submission_date": "2015-05-07", + "subtitle": "Exercise Sheet 3, Linear Regression", + "tags": [ + "IntelligenSystems", + "Course" + ], + "title": "Intelligent Systems 1 - Summer Semester 2015" + }, + "celltoolbar": "Edit Metadata", + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.4.3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}