{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"preamble": true
},
"source": [
"(Defining latex commands: not to be shown...)\n",
"$$\n",
"\\newcommand{\\norm}[1]{\\left \\| #1 \\right \\|}\n",
"\\DeclareMathOperator{\\minimize}{minimize}\n",
"\\newcommand{\\real}{\\mathbb{R}}\n",
"\\newcommand{\\normal}{\\mathcal{N}}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#Intelligent Systems Sheet 3\n",
"Maximus Mutschler & Jan-Peter Hohloch"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Gaussian Algebra (25 Points)\n",
"\n",
"Prove that the product of two univariate (scalar) Gaussian distributions is a Gaussian again, i.e. show, by explicitly performing the required arithmetic transformations, that\n",
"\n",
"\\begin{equation}\n",
" \\normal(x;\\mu,\\sigma^2)\\normal(x;m,s^2) = \\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right].\n",
"\\end{equation}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Proof\n",
"\\begin{align*}\n",
" f(x) &= \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\cdot e^{\\frac{-(x-\\mu)^2}{2\\sigma^2}}\\cdot \\frac{1}{\\sqrt{2\\pi s^2}}\\cdot e^{\\frac{-(x-m)^2}{2s^2}}\\\\\n",
" &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\alpha}\\\\\n",
" \\text{where }\\alpha &= \\frac{s^2(x-\\mu)^2+\\sigma^2(x-m)^2}{2\\sigma^2 s^2}\\\\\n",
" &=\\frac{(s^2+\\sigma^2)x^2-2(s^2\\mu+\\sigma^2 m)x+s^2\\mu^2+\\sigma^2m^2}{2\\sigma^2s^2}\\\\\n",
" &=\\frac{x^2-2\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}x+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
" &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
" &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}+\\underbrace{\\frac{\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}_\\beta\\\\\n",
" \\beta&= \\frac{\\frac{s^4\\mu^2+s^2\\sigma^2\\mu^2+s^2\\sigma^2m^2+\\sigma^4m^2-\\left(s^4\\mu^2+2s^2\\sigma^2\\mu m+\\sigma^4m^2\\right)}{\\left(s^2+\\sigma^2\\right)^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
" &=\\frac{s^2\\sigma^2\\left(\\mu^2-2\\mu m+m^2\\right)}{2\\sigma^2s^2\\left(s^2+\\sigma^2\\right)}\\\\\n",
" &=\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}\\\\\n",
" \\Rightarrow f(x)&=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-(\\alpha-\\beta)-\\beta}\\\\\n",
" &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}\\cdot e^{-\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}}\\\\\n",
" &=\\frac{\\sqrt{2\\pi(s^2+\\sigma^2)}\\sqrt{2\\pi\\frac{\\sigma^2s^2}{s^2+\\sigma^2}}}{2\\pi s\\sigma}\\normal\\left[x;\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2},\\frac{\\sigma^2s^2}{s^2+\\sigma^2}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n",
" &=\\frac{\\sqrt{2\\pi\\cdot 2\\pi\\sigma^2s^2}}{2\\pi s\\sigma}\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n",
" &=\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\hspace{2cm} q.e.d.\n",
"\\end{align*}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Maximum Likelihood Estimator of Simple Linear Regression (25 Points)\n",
"\n",
"Derive the formula $\\mathbf{w}_{MLE} = (X^TX)^{-1}X^T\\mathbf{y}$ from the lecture, by calculating the derivative of $p(\\mathbf{y}\\,|X,\\mathbf{w}) = \\normal(\\mathbf{y}\\,|X\\mathbf{w}, \\sigma^2I)$ with respect to $\\mathbf{w}$, setting it to zero and solving it for $\\mathbf{w}$.\n",
"\n",
"\n",
"Note: _To refresh your linear algebra you might find it useful to have a look in [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/pdfs/Murray_cribsheet.pdf)._"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##wtf?\n",
"\n",
"$\\vec{w}=X^{-1}\\vec{y}$, zumindest wenn ich das ausrechne..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Linear regression (50 Points)\n",
"\n",
"In this exercise you will perform a regression analysis on a toy dataset. You will implement ridge regression and learn how to find a good model through a comparative performance analysis.\n",
"\n",
"1) Download the [training set](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_train.csv)! <br>\n",
"2) Implement $\\mathbf{w}_{RIDGE}$ as a function of a given $X, \\mathbf{y}$ array and a regularization parameter $\\lambda$!"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
":0: FutureWarning: IPython widgets are experimental and may change in the future.\n"
]
}
],
"source": [
"# Loading the required packages\n",
"%matplotlib inline\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"from IPython.html.widgets import interact"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [],
"source": [
"def wRidge(X,y,lamb):\n",
" # Change the following line and implement the ridge regression estimator wRidge\n",
" a = (lamb*np.eye(X.shape[0])+X.dot(X.T))\n",
" b=np.linalg.inv(a)\n",
" c=b.T.dot(X)\n",
" return c.T.dot(y) # TODO pbyl wrong like this"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3) Load \"ex1_train.csv\" into a numpy array! The first column in the csv file is $X$ and the second column is $\\mathbf{y}$, assign them to each variable!"
]
},
{
"cell_type": "code",
"execution_count": 73,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Read ex1_train.csv and assign the first column and \n",
"# second column to variables x and y respectively.\n",
"from numpy import genfromtxt\n",
"a=np.genfromtxt('ex1_train.csv',delimiter=' ')\n",
"x=a[:,0].T\n",
"y=a[:,1].T"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"4) Plot the training data with appropriate labels on each axes!"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {
"collapsed": false,
"scrolled": false
},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.text.Text at 0x7f25abb529e8>"
]
},
"execution_count": 74,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
"AAALEgAACxIB0t1+/AAAFmxJREFUeJzt3XuQZGd93vHvo1tWWIsEBmNZ7CIsQYJYQICREZcwQBzE\n",
"xoHIGDCOMRi05Spgl4oJZQQJkkM53BLC7lJ2EgHiZm4yBiOQZGSsNUpUXAQISxFClsKyKwwSV7Eb\n",
"dhNdfvnj9LJ9RnPpnpnu0z3z/VRNTZ/3dJ/+zZZ0nn7f97ynU1VIknTIEV0XIEmaLAaDJKnFYJAk\n",
"tRgMkqQWg0GS1GIwSJJaDAapA0nek+QNXdchzcVg0KqVZHeSp4/hfc5P8v4hX1a9n0GOvyvJS4ev\n",
"TFoag0Gr2cAn345kwOdN8t+gVchg0JqQ5MVJ/keStyb5YZL/neSsvv27krwxyReS3J7kE0nu09s3\n",
"k2TvrOPtTvL03jHOBZ6fZF+Sr87z/o9O8pUkP0nyYWBd3777JPlUktt6tV2c5KTevj8Gngy8o3f8\n",
"Hb327Un29Gq9OsmTVvifTGuYwaC15AzgBuDngbcA75q1/4XA7wEnAncCOxY4VgFVVZcB/xH4cFWt\n",
"r6pHz35ikmOATwDvBe4DXAQ8h8M9gfRq2dj7OQC8g+YNXgdcCby8d/xtvdd8EXhU73gfBC7qvY+0\n",
"bAaD1pJvVdW7qrlB2PuAE5P8Qm9fAe+rquur6qfAvweel2SQ4Z6w8LDQ44Gjqmp7Vd1VVR8DvnRo\n",
"Z1X9sKo+XlUHq2o/TdA8ZY73oO81f1ZVP6qqu6vqbcA/Av7xALVKizIYtJZ899CD3skf4Li+/f3D\n",
"RXuAo4H7rcD7/hLw7Vlt36J3sk9yryT/rTc8dTvwt8Dxs0KpNc+Q5N8muT7Jj5P8CDh+hWqVDAap\n",
"z8ZZj+8Avg/8H+Beh3YkORK4f99zF5sc/g5w0qy2B/W97lXAQ4Ezqup4mt5Cfy9kdig8GXg18Nyq\n",
"OqGq7gPczuCT2dKCDAapEeB3kjwsyb2A/wBc1Bt2uhFYl2RzkqOBf0czdHPId4GTFxh2ugq4M8m2\n",
"JEcn+Q3gcX37j6OZV7g9yX2B82a9/lbglL7t9TRzIN9PckyS1wP3XsofLc3FYNBaMdelqzXr8fuB\n",
"99B8wj8G2AZQVbcDLwPeCdwC7Kc97HRR7/cPklx9jzeuugP4DeDFwA+A5wEf63vK24FjaXonVwGX\n",
"zqptO/CbvSuW3g5c1vu5EdhNEyp7FvzrpSGkyy/qSfJu4F8At1XVI3pt5wPnAN/rPe3c3pUf0sgk\n",
"uQJ4f1W9u+tapK513WO4EDhrVlsBb6uqR/d+DAWNi2P0Eh0HQ1VdCfxojl3+D6ouuMJYovsew3y2\n",
"JvlaknclOaHrYrT6VdVTHUaSGpMYDH8KPBg4nWYS8D93W44krS1HdV3AbFV126HHSd4JXDz7OUns\n",
"8kvSElTVokP1ExcMSU6squ/0Ns8Grp3reYP8cZMqyflVdX7XdSyV9XfL+rszzbXD4B+qOw2GJB+i\n",
"WeV5v97dK88DZpKcTjMR+E3g9zssUZLWnE6DoapeMEezE4CS1KFJnHxeC3Z1XcAy7eq6gGXa1XUB\n",
"y7Sr6wKWaVfXBSzDrq4LGIdOVz4vVZKa5jkGSerCoOfOiZt8liStrGTTZtiwbfFnNgwGSVrFmlA4\n",
"cztccOqgN5VwjkGSVrUN25pQGJzBIEmr2vp1w77CoSRJq8LhcfT162DfQdi7o+q6S7quq3v7Dg77\n",
"CoNB0tRrj6MfsuWUZBOGw94dsOWUYYaTvFxV0liM8hN98szL4NJn3HPP5suqLnnmSrzHNGv+7Tdu\n",
"hUvP8nJVSRNh9J/o5xtHP+7Y5R97+oeperVeMhX3SpK0Vsx1ZcwFp8LmrcAKnGDnG0fff2C5R16L\n",
"w1RelSRpDEb7ib43jn5Tu+2cm2HPzuUfe75Q27h1+ceeTPYYJI3B6D7RQzNUkmyi6YEcd2xz3D07\n",
"p2GYahIZDJLGYK4rY1bqE33j0Dj6Sh3vsNGG2iQyGCSN3Gg/0Y/a6ENt0ni5qiQt4vDlntMWam2D\n",
"njsNBklaIwY9d3pVkiSpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1\n",
"GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqSWToMhybuT3Jrk2r62+ya5\n",
"PMmNST6T5IQua5SktabrHsOFwFmz2l4DXF5VDwU+29uWJI1Jp8FQVVcCP5rV/Czgvb3H7wX+1ViL\n",
"kqQ1rusew1weUFW39h7fCjygy2Ikaa05qusCFlJVlaTm2pfk/L7NXVW1ayxFSdKUSDIDzAz9uqo5\n",
"z7tjk+Rk4OKqekRv+wZgpqq+m+RE4Iqq+iezXlNVlbEXK0lTbNBz5yQOJX0SeFHv8YuAT3RYiySt\n",
"OZ32GJJ8CHgKcD+a+YTXA38JfBTYCOwGnldVP571OnsMkjSkQc+dnQ8lLYXBIEnDm+ahJElShwwG\n",
"SVKLwSBJajEYJEktE73ATdJhyabNsGEbrF8H+w7C3h1V113SdV1afQwGaQo0oXDmdrjg1MOtW05J\n",
"NmE4aKU5lCRNhQ3b2qEAzfbGrd3Uo9XMYJCmwvp1c7cfd+x469BaYDBIU2Hfwbnb9x8Ybx1aCwwG\n",
"aSrs3QFbbmq3nXMz7NnZTT1azbwlhjQlmgnojVub4aP9B2DPTieeNQzvlSRJavFeSZKkJTEYJEkt\n",
"BoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqQWg0GS1GIw\n",
"SJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWo7quoD5JNkN/AS4C7ijqs7o\n",
"tiJJWhsmNhiAAmaq6oddFyJJa8mkDyWl6wIkaa2Z5GAo4K+TXJ1ky+ydyTMvSzZt7qAuSVrVJnko\n",
"6YlV9Z0k9wcuT3JDVV15ePelz4AtpySbqLruks6qlKRVZmKDoaq+0/v9vSQfB84A+oLhfOCkU2Hf\n",
"9iQ/rapdHZQpSRMryQwwM/TrqmqxA/98Vf1gaWUtTZJ7AUdW1b4kPwd8BvijqvpMb381I00Az/vb\n",
"qo/OjLM+SZpGSaqqFp27HaTH8Pkk1wAXApfWYkmyMh4AfDwJNDX+2aFQuKf9B8ZQjyStGYP0GI4A\n",
"/hnwEuBxwEeBC6vqxtGXN29NvR7DOTfD57c5xyBJixu0x7BoMMw66NOADwA/B1wDnFtVVy25yiVq\n",
"guGZl8GenYaCJA1mxYIhyf2Afw38LnAr8E7gYuBRwJ9X1cnLrnZIg/5xkqTDVnKO4SqaXsKzq+qW\n",
"vvark/zXpRYoSZpMA80xVNXdY6pnIPYYJGl4g547F135PGmhIEkarUm+JYYkqQOLBkOSJ83R9sTR\n",
"lCNJ6togPYadc7S9Y6ULkSRNhnmvSkpyJvAE4P5J/oDDt8Bej0NQkrRqLXS56jE0IXBk7/chPwF+\n",
"c5RFSZK6M8jlqg+qqm+NqZ6BeLmqJA1vJVc+XzFHc1XV05Za3HIZDJI0vJVc+fzqvsfrgOcAdy61\n",
"MEnSZBvqJno/e1Hypap63AjqGfT97TFI0pBWrMeQ5L59m0cAvwLcexm1SZIm2CBDSV/h8Nel3Qns\n",
"Bl46qoIkSd1a0lBS1xxKkqThreRQ0rHAy4An0fQcrgT+tKoOLrtKSdLEGeRy1YtoFrV9gGb1828D\n",
"x1fVc0df3rw12WOQpCGt5DqG66vqtMXaxslgkKThrdj3MQBf6d036dCBHw98eTnFSZIm1yA9hhuA\n",
"hwJ7aeYYNgLfoLlCqarqkaMuco6a7DFI0pBWcuXzMzh8Z9VDao42SdIqMEiP4f1V9cLF2sbJHoMk\n",
"DW8l5xg2zTrwUcBjl1qYJGmyzRsMSV6bZB/wiCT7Dv0AtwGfHFuFkqSxGmQo6U1V9Zox1TMQh5Ik\n",
"aXgruY7hKRy+V9LPVNXnll7e8hgMkjS8lQyGT3E4GNYBZwBf9ot6JGm6rNjlqlX167MOvAHYvoza\n",
"JEkTbJCrkma7BXjYShciSZoMg9xddWff5hHA6XhLDElatQZZ+fxlDs8x3AV8sKr+5+hKkiR1aZDJ\n",
"52OBU2nC4aZJ+B4GJ58laXjLXvmc5Ogkb6G5ed57gfcBtyR5a5KjV65USdIkWWjy+a3AfYEHV9Vj\n",
"quoxwC8DJwD/aRzFSZLGb96hpCQ3AQ+tqrtntR8JfKOqTh1ZUclZwNuBI4F3VtWbZ+13KEmShrQS\n",
"N9G7e3YoAFTVXcA92ldKL3jeAZwFnAa8IImXx0rSmCwUDF9P8qLZjUleCNwwupI4g2aSe3dV3QF8\n",
"GHj2CN9Pa0SyaXPyzMuS5+1qfm/a3HVN0iRa6HLVlwN/keQlHF638FjgXsDZI6zpJJoJ70NuAX51\n",
"hO+nNaAJgTO3wwV9Q6BbTkk2UXXdJd1VJk2eeYOhqm5J8qvA04CH01yu+umq+uyIa1r4+llpSTZs\n",
"a4cCNNubtwIGg9RnwQVu1cxMf7b3My7fBjb0bW+g6TW0JDm/b3NXVe0abVmabuvXzd1+3LHjrUMa\n",
"nyQzwMywrxtk5fO4XQ08JMnJwD8AzwdeMPtJVXX+WKvSlNs3z8LM/QfGW4c0Pr0PzLsObSc5b5DX\n",
"LeUmeiNVVXcCrwD+Crge+EhVfb3bqjT99u6ALTe12865GfbsnPv50tq16C0xJpHrGLQUzQT0xq3N\n",
"8NH+A7BnpxPPWktW7It6JpHBIEnDW4kFbpKkNWgSJ5870ww1bNjWXMGy7yDs3eFQg6S1xmDocQGU\n",
"JDUcSvqZ+RZAbdzaTT2S1A2D4WdcACVJYDD0cQGUJIHB0McFUJIErmOYdVwXQElavVzgJklqcYGb\n",
"JGlJDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMk\n",
"qcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWo5qusCpEOSTZthwzZYvw72HYS9O6quu6TruqS1\n",
"xmDQRGhC4cztcMGph1u3nJJswnCQxsuhJE2IDdvaoQDN9sat3dQjrV0GgybE+nVztx937HjrkGQw\n",
"aELsOzh3+/4D461DksGgCbF3B2y5qd12zs2wZ2c39UhrV6qq6xqGlqSqKl3XoZXVTEBv3NoMH+0/\n",
"AHt2OvEsrZxBz50TFwxJzgfOAb7Xazq3qi6b9RyDQZKGNOi5cxIvVy3gbVX1tq4LkaS1aFLnGOwN\n",
"SFJHJjUYtib5WpJ3JTmh62IkaS3pZI4hyeXAL86x63XA5zk8v/AG4MSqeums1xfwR31Nu6pq1whK\n",
"laSplWQGmOlrOm8qJ5/7JTkZuLiqHjGr3clnSRrSoOfOiRtKSnJi3+bZwLVd1SJJa9EkXpX05iSn\n",
"01yd9E3g9zuuR5LWlIkeSpqPQ0mSNLypHUqSJHXLYJAktRgMkqQWg0GS1GIwSJJaDAZJUovBIElq\n",
"MRgkSS2TuPJZE6r5hrUN22D9uuY7mvfu8BvWpNXHYNBAmlA4cztccOrh1i2nJJswHKTVxaEkDWjD\n",
"tnYoQLO9cWs39UgaFYNBA1q/bu72444dbx2SRs1g0ID2HZy7ff+B8dYhadQMBg1o7w7YclO77Zyb\n",
"Yc/ObuqRNCredlsDayagN25tho/2H4A9O514lqbHoOdOg0GS1gi/j0GStCQGgySpxWCQJLUYDJKk\n",
"FoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWox\n",
"GCRJLQaDJKnFYJAktXQSDEmem+R/JbkryWNm7Ts3yd8nuSHJP++iPklay7rqMVwLnA18rr8xyWnA\n",
"84HTgLOAP0my6no1SWa6rmE5rL9b1t+daa59GJ2cdKvqhqq6cY5dzwY+VFV3VNVu4CbgjLEWNx4z\n",
"XRewTDNdF7BMM10XsEwzXRewTDNdF7AMM10XMA6T9mn8l4Bb+rZvAU7qqBZJWpOOGtWBk1wO/OIc\n",
"u15bVRcPcahaoZIkSQNIVXfn3SRXAK+qqq/0tl8DUFVv6m1fBpxXVV+Y9TrDQpKWoKqy2HNG1mMY\n",
"Qn+RnwQ+mORtNENIDwG+OPsFg/xhkqSl6epy1bOT7AUeD3w6yaUAVXU98FHgeuBS4GXVZZdGktag\n",
"ToeSJEmTZ9KuShpYkjck+VqSa5J8NsmGrmsaRpK3Jvl672/4iyTHd13TMBZapDipkpzVWzj590n+\n",
"sOt6hpXk3UluTXJt17UMK8mGJFf0/pu5Lsm2rmsaRpJ1Sb7QO99cn+SNXde0FEmOTPLVJAteADS1\n",
"wQC8paoeVVWnA58Azuu6oCF9Bnh4VT0KuBE4t+N6hjXnIsVJleRI4B00CydPA16Q5GHdVjW0C2nq\n",
"n0Z3AP+mqh5OM4T88mn696+qg8BTe+ebRwJPTfKkjstailfSDNUvOFQ0tcFQVfv6No8Dvt9VLUtR\n",
"VZdX1d29zS8AD+yynmEtsEhxUp0B3FRVu6vqDuDDNAsqp0ZVXQn8qOs6lqKqvltV1/Qe7we+TrNu\n",
"aWpU1U97D48BjgR+2GE5Q0vyQGAz8E7aF/3cw9QGA0CSP06yB3gR8Kau61mGlwCXdF3EKncSsLdv\n",
"28WTHUlyMvBomg9EUyPJEUmuAW4FruhdLDNN/gvwauDuxZ440cGQ5PIk187x8y8Bqup1VbUReA/N\n",
"Hz1RFqu/95zXAf+vqj7YYalzGqT+KeJVFhMgyXHAnwOv7PUcpkZV3d0bSnog8E+n6b5JSX4duK2q\n",
"vsoivQWYjHUM86qqXxvwqR9kAj9xL1Z/khfTdO2ePpaChjTEv/80+DbQf4HCBtq3X9GIJTka+Bjw\n",
"gar6RNf1LFVV3Z7k08CvALs6LmdQTwCelWQzsA64d5L3VdXvzvXkie4xLCTJQ/o2nw18tataliLJ\n",
"WTTdumf3Jram2TQsOLwaeEiSk5McQ3MX3092XNOakSTAu4Drq+rtXdczrCT3S3JC7/GxwK8xReec\n",
"qnptVW2oqgcDvwX8zXyhAFMcDMAbe8Ma19Dc8fBVHdczrJ00k+aX9y4f+5OuCxrGfIsUJ1VV3Qm8\n",
"AvgrmqsyPlJVX++2quEk+RBwFfDQJHuT/F7XNQ3hicDv0FzN89XezzRdYXUi8De9880XgIur6rMd\n",
"17QcCw6tusBNktQyzT0GSdIIGAySpBaDQZLUYjBIkloMBklSi8EgSWoxGKRZkqz4rRqSPCjJC1b6\n",
"uNIoGAzSPY1icc+Dgd8ewXGlFWcwSPNIMpNkV5KLel+q9IG+fbuTvDnJ3/W+wOWUXvt7kjyn73mH\n",
"bg//JuDJvRW/r5z1Pmcn+eve4xOTfCPJL4z+L5TmZjBICzud5stNTgN+OckTeu0F/LiqHknzBUBv\n",
"72ufyx8CV1bVo6tqe/+Oqvo48J0krwD+O/D6qrpthf8OaWAGg7SwL1bVP1Rz75hrgJP79n2o9/vD\n",
"wJmLHGexGw1upfkWv4NV9ZGlFCqtFINBWtj/7Xt8F/Pfqv5QT+FOev9fJTmC5tu+BrGhd/wH9O5E\n",
"KnXGYJCW7vl9v6/qPd4NPLb3+FnA0b3H+4D1cx0kyVE0t6T+LeAG4A9GUKs0sIn+oh6pIzXP49nu\n",
"k+RrwEHg0KWoFwB/2bs982XAoUtfvwbc1Wu/cNY8w7nA56rqqiR/B3wpyaeq6hsr8cdIw/K229IS\n",
"JPkm8NiqmqovhJcG4VCStDR+otKqZY9BktRij0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySp5f8D\n",
"oJt+CXUlyycAAAAASUVORK5CYII=\n"
],
"text/plain": [
"<matplotlib.figure.Figure at 0x7f25abb6fb00>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Plot the input data here\n",
"plt.plot(x,y,'o')\n",
"plt.title('Input data')\n",
"plt.ylabel('Output y')\n",
"plt.xlabel('Input x')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"5) Implement a function which constructs features upto a input polynomial degree $d$!<br>\n",
"Note: _Constructing higher polynomial features is similar to what you implemented in Exercise 3 (SVM) of the previous exercise sheet._"
]
},
{
"cell_type": "code",
"execution_count": 80,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def construct_poly(x,d):\n",
" ## Implement a method which given an array of size N, \n",
" ## returns an array of dimension (N,d)\n",
" return (np.ones((d,1))*np.asmatrix(x)).T"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[-3. -3. ]\n",
" [-2.5 -2.5]\n",
" [-1. -1. ]\n",
" [-0.5 -0.5]\n",
" [ 0. 0. ]\n",
" [ 1. 1. ]\n",
" [ 2. 2. ]\n",
" [ 2.5 2.5]\n",
" [ 3. 3. ]\n",
" [ 4. 4. ]]\n",
"[[ 52.75 52.75]\n",
" [ 52.75 52.75]]\n"
]
},
{
"data": {
"text/plain": [
"matrix([[ 1.51220623, 1.51220623]])"
]
},
"execution_count": 86,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#try it out\n",
"X=construct_poly(x,2)\n",
"print(X)\n",
"print(X.T.dot(X))\n",
"wRidge(X,y,1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"6) Implement the Mean Squared Error Loss (MSE) as a function of the predicted and true values of the target variable! <br>"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def MSE(y_predict,y_true):\n",
" ## Implement mean squared error for a given input y and its predictions.\n",
" return ((y_predict-y_true)**2)/y_predict.shape[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"7) By comparing the MSE find the degree $d$ for the polynomial that fits the training data best! You might find it useful to use the code below to interactively change the variable $d$, set $\\lambda = 1$ and keep it fixed. Plot the error as a function of different values of $d$!<br>"
]
},
{
"cell_type": "code",
"execution_count": 95,
"metadata": {
"collapsed": false,
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[-3. -2.5 -1. -0.5 0. 1. 2. 2.5 3. 4. ]\n",
"(10, 1) (10,)\n",
"[[ 7.24825764 -79.61751096 -163.32510817 -124.420806 -64.36110538\n",
" 64.00498155 130.78610485 134.65832608 128.64828886 140.65345344]\n",
" [ 2.38125049 -75.48579685 -150.52183853 -115.64776572 -61.80985436\n",
" 53.25835177 113.12139072 116.59247535 111.20503839 121.96654702]\n",
" [ -12.21977095 -63.09065449 -112.11202963 -89.32864491 -54.15610128\n",
" 21.01846246 60.12724836 62.39492317 58.87528698 65.90582776]\n",
" [ -17.0867781 -58.95894038 -99.30875999 -80.55560464 -51.60485026\n",
" 10.27183269 42.46253424 44.32907244 41.4320365 47.21892134]\n",
" [ -21.95378525 -54.82722626 -86.50549036 -71.78256437 -49.05359924\n",
" -0.47479708 24.79782012 26.26322172 23.98878603 28.53201492]\n",
" [ -31.68779954 -46.56379802 -60.89895109 -54.23648382 -43.95109719\n",
" -21.96805662 -10.53160812 -9.86847974 -10.89771491 -8.84179791]\n",
" [ -41.42181383 -38.30036979 -35.29241182 -36.69040328 -38.84859514\n",
" -43.46131617 -45.86103637 -46.00018119 -45.78421585 -46.21561075]\n",
" [ -46.28882098 -34.16865567 -22.48914219 -27.91736301 -36.29734412\n",
" -54.20794594 -63.52575049 -64.06603192 -63.22746633 -64.90251717]\n",
" [ -51.15582813 -30.03694156 -9.68587255 -19.14432273 -33.74609309\n",
" -64.95457571 -81.19046461 -82.13188264 -80.6707168 -83.58942359]\n",
" [ -60.88984242 -21.77351332 15.92066671 -1.59824219 -28.64359104\n",
" -86.44783525 -116.51989285 -118.2635841 -115.55721774 -120.96323643]]\n"
]
},
{
"ename": "TypeError",
"evalue": "only length-1 arrays can be converted to Python scalars",
"output_type": "error",
"traceback": [
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[1;31mTypeError\u001b[0m Traceback (most recent call last)",
"\u001b[1;32m<ipython-input-95-b005297af287>\u001b[0m in \u001b[0;36mplot\u001b[1;34m(n)\u001b[0m\n\u001b[0;32m 8\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 9\u001b[0m \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"MSE %f\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m 11\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
"\u001b[1;31mTypeError\u001b[0m: only length-1 arrays can be converted to Python scalars"
]
},
{
"data": {
"image/png": [
"iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
"AAALEgAACxIB0t1+/AAAGDdJREFUeJzt3Xm43XV97fH3SpgiiMwyFigiJGFIQBDRylHEG7ncUrwO\n",
"0Am1rdw6IZMRuVcClkIgBKoWikJEKYMoKlBAiUgERaYyBUNAVBSUIt4ql0KBENb947uxh5jh7LPP\n",
"73z3sF7Pcx7O3mfv81vy8Cx/57N/v+9XtomIiP4yoXaAiIgYeyn3iIg+lHKPiOhDKfeIiD6Uco+I\n",
"6EMp94iIPtRxuUuaJ+kxSQuHPTdL0iOS7mx9zej0OBERMXJjceb+BWDZ8jYw1/b01tc3x+A4EREx\n",
"Qh2Xu+0bgd8s50fq9HdHRMToNDlz/7CkuyWdJ2m9Bo8TERHLaKrczwa2BaYBjwKnN3SciIhYjtWa\n",
"+KW2f/Xi95LOBa5c9jWSsqhNRMQo2F7l2LuRcpe0me1HWw8PAhYu73UjCditJM2yPat2jtFK/rqS\n",
"v55ezg4jPzHuuNwlXQzsA2wk6WHgeGBI0jTKVTM/BQ7r9DgRETFyHZe77UOW8/S8Tn9vRESMXu5Q\n",
"Hb0FtQN0aEHtAB1aUDtAhxbUDtChBbUDdGBB7QDjQbU265DkXp65R0TUMNLuzJl7REQfSrlHRPSh\n",
"lHtERB9KuUdE9KGUe0REH0q5R0T0oZR7REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oZR7\n",
"REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oY43yI6IiOZJTAT+aqSvT7lHRHQ5iX2BM4Df\n",
"jvQ9KfeIiC4l8WpgDrATcAzwNeCFkbw3M/eIiC4jsb7EXOAm4HvAFJvLbDzS35Fyj4joEhKrSXwQ\n",
"WAysDUy1OdXmmXZ/V8YyERFdQGIGMBf4JbCfzT2d/L6Ue0RERRJTgNOB7YCjgSvbGb+sSMYyEREV\n",
"SGwo8Rngu8C1wE42V4xFsUPKPSJiXEmsIfFRylxdwGSbM2yeG8vjZCwTETEOJAQcQBnB/BjYx2ZR\n",
"U8dLuUdENExiF8qHpZsDh9tc0/QxM5aJiGiIxCYS5wDzgW8Au45HsUPKPSJizEmsKXEMsAh4CtjR\n",
"5rM2S8YrQ8YyERFjpDVXfztwKvBDYG+bB2pkSblHRIwBid0oi3ttABxm8+2aeTKWiYjogMRmEvOA\n",
"q4GLgOm1ix3GoNwlzZP0mKSFw57bQNJ8SQ9IulbSep0eJyKim0hMkvgEsBB4HNjB5hyb5ytHA8bm\n",
"zP0LwIxlnvs4MN/2q4HrWo8jInqehCQOBu4DdgdeazPT5onK0V5Cdud3ukraBrjS9s6tx4uBfWw/\n",
"JmlTYIHtHZd5j22r44NHRIwTiT0pc/VJwBE23x3/DCPrzqZm7q+0/Vjr+8eAVzZ0nIiIxklsKXEB\n",
"5Vr1c4E9ahR7Oxq/Wsa2JS33zwNJs4Y9XGB7QdN5IiJGSmJtyg5IHwbOpszVnxzfDBoChtp+X4Nj\n",
"mSHb/yZpM+D6jGUioldITAD+FDgZ+D4w0+ZndVMVtccyVwCHtr4/lPKnTERE15PYG7gZ+AhwsM3B\n",
"3VLs7ej4zF3SxcA+wEaU+fongcuBS4E/AB4C3mX7t8u8L2fuEdE1JLYGZgOvB44FLrJHthn1eBpp\n",
"d47JWGY0Uu4R0Q0kXk65XPt/AZ8G5tg8VTfVitUey0REdDWJCRLvpWyasRVlxcYTurnY25G1ZSJi\n",
"4EjsQ7le/RngIJtbK0cacyn3iBgYEttRVmzcHZgJXDpWe5Z2m4xlIqLvSbxC4lTgFuAOyr6lX+7X\n",
"YoeUe0T0MYmJEocB9wMbAjvbnGTzn5WjNS5jmYjoSxJvoexb+htgf5s7KkcaVyn3iOgrEq8G5gBT\n",
"KUsHfL2fxy8rkrFMRPQFifUlzgBuAm4Epth8bRCLHVLuEdHjJFaX+BDlevVJlFI/zebZytGqylgm\n",
"InqWxAzKXP2XwFtsFq7iLQMj5R4RPUdiCnA6sB1wFPAvgzp+WZGMZSKiZ0hsJPFZ4LvAt4CdbK5M\n",
"sf++lHtEdD2JNSSOoOxbamBHmzNtnqscrWtlLBMRXUtCwAGUEcyDwD42i+qm6g0p94joShK7UD4s\n",
"3Rw43OaaypF6SsYyEdFVJDaROAeYD3ydshRvir1NKfeI6AoSa0p8DFgEPEWZq/+jzZLK0XpSxjIR\n",
"UVVrrv52ylK89wJ72zxQN1XvS7lHRDUSu1E2zVgfeL/NdZUj9Y2MZSJi3ElsJjEPuAq4EJieYh9b\n",
"KfeIGDcSkySOo4xfHqfM1T9ns7RytL6TsUxENK41V383cApwO7CHzU/qpupvKfeIaJTEnpS5+iTg\n",
"L21uqBxpIGQsExGNkNhS4gLKteqfB16TYh8/KfeIGFMSa0vMAu4GfgbsYHO+zQt1kw2WjGUiYkxI\n",
"TAD+DPh7yk5I021+XjfV4Eq5R0THJPYGzqSs2Phum5sqRxp4KfeIGDWJrYHZwOuBjwMXZ/zSHTJz\n",
"j4i2Sbxc4iTgXylrwexgc2GKvXuk3CNixCQmSrwPuB/YkrJi44k2T1eOFsvIWCYiRkRiH8r16k8D\n",
"B9rcVjlSrETKPSJWSmI7yoqNuwEzga9kz9Lul7FMRCyXxCskTgVuoSwZMNnm0hR7b0i5R8RLSKwm\n",
"cRhlrr4BsJPNyTbPVI4WbchYJiJ+R2I/yr6lvwbeZnNn5UgxSo2Wu6SHgP8HLAWW2N6zyeNFxOhI\n",
"7ADMASYDRwOXZ/zS25o+czcwZPvfGz5ORIyCxPrAJ4E/p9yM9A6bZ+umirEwHjN3jcMxIqINEqtL\n",
"fJgyV18LmGozJ8XeP8bjzP3bkpYC59j+fMPHi4hVkHgbZa7+MLCvzcLKkaIBTZf7620/KmljYL6k\n",
"xbZvbPiYEbEcElOB04FtgSOBqzNX71+NlrvtR1v/fFzS14E9KUuBAiBp1rCXL7C9oMk8EYNIYiPg\n",
"BOCdwEnAWTZL6qaKkZI0BAy1/T67mf/jlvQyYKLtJyWtDVwLnGD72tbPbTvz+IiGSKwBfAg4FrgY\n",
"OMHm/9ZNFZ0aaXc2eeb+SuDrkl48zoUvFntENKe1GfUfUy5tfAB4o819dVPFeGvszH2VB86Ze8SY\n",
"k9iV8mHppsCRNt+qHCnG2Ei7M8sPRPQBiVdKfI4y/vwqZSneFPsAS7lH9DCJtSRmAj8EnqRsmnG2\n",
"zfOVo0VlWVsmoge15ur/k7IU7z3A62x+VDdVdJOUe0SPkdidsmnGK4C/tvlO5UjRhTKWiegREptL\n",
"fAH4F+BLwG4p9liRlHtEl5OYJPG/gYXAY5S5+rk2SytHiy6WsUxEl2rN1Q8GTgFuBfaw+UndVNEr\n",
"Uu4RXUhiL8pcfQ3gL2xuqBwpekzGMhFdRGIriQuBy4BzKGfrKfZoW8o9ogtIrCNxInAX8BPKXP18\n",
"mxcqR4selbFMREUSE4C/oKzWeAMw3ebndVNFP0i5R1Qi8QbKXH0p8E6bH1SOFH0k5R4xziS2pexX\n",
"uhet5Xgzfomxlpl7xDiRWFfiZOB24F5gR5sLU+zRhJR7RMMkJkr8NbAY2BzYxeZEm6crR4s+lrFM\n",
"RIMk3kSZq/8HcKDNbZUjxYBIuUc0QOJVwGnANGAm8JVsRh3jKWOZiDEksZ7EHOBm4BZgss2lKfYY\n",
"byn3iDEgsZrE31Lm6usBO9mcYvNM5WgxoDKWieiQxFsp+5Y+DsywuatypIiUe8RoSewIzAF2BI4G\n",
"Ls/4JbpFxjIRbZLYQOIfgO8B1wNTbb6RYo9uknKPGCGJ1SU+Qpmrr075sPR0m2crR4v4PRnLRKxC\n",
"a9OM/YHTgZ8Db7a5t26qiJVLuUeshMRUyoelWwNHAVdn/BK9IGOZiOWQ2FjiLMpM/SpgZ5urUuzR\n",
"K1LuEcNIrCFxFLAIWEJZ3OvTNksqR4toS8YyEfxurn4gZcmA+4E/sllcN1XE6KXcY+BJ7EpZ3GsT\n",
"4IM211aOFNGxjGViYElsKvF54FrgK8C0FHv0i5R7DByJtSQ+Ttkw4wnKZtRn2zxfOVrEmMlYJgZG\n",
"a67+DuBU4C5gL5sH66aKaEbKPQaCxO7AmcDLgffZXF85UkSjMpaJviaxucT5wJXAF4HdU+wxCFLu\n",
"0ZckXibxf4B7gEcp16ufa7O0crSIcZGxTPSV1lz9EOAUym5Ie9j8tG6qiPHX2Jm7pBmSFkv6kaSZ\n",
"TR0n4kUSewE3AUcCf2bzrhR7DCrZY79UhqSJlLv83gL8ArgNOMT2fcNeY9sa84PHwJHYinKmvg9w\n",
"HHCBzQt1U0U0Y6Td2dSZ+57Ag7Yfsr0EuIRya3fEmJFYR+JEymWND1KuV/9iij2iuXLfAnh42ONH\n",
"Ws9FdExigsShlE0ztgOm2xxv81TlaBFdo6kPVLMsajRC4g2U69WfB95hc3PlSBFdqaly/wWw1bDH\n",
"W1HO3l9C0qxhDxfYXtBQnuhxEtsCs4G9gJnAJVlbPQaBpCFgqO33NfSB6mqUD1T3BX4J3Eo+UI1R\n",
"kFgXOBZ4P2Xlxrk2T9dNFVHPSLuzkTN3289L+hDwLWAicN7wYo9YFYmJwHuBEymrNu5s88u6qSJ6\n",
"RyNn7iM6cM7cYwUk3kQ5S38SOMLm9sqRIrpG1TP3iNGQeBVlJ6RpwDHAZZmrR4xO1paJ6iTWk5hD\n",
"WS7gZmCyzVdT7BGjl3KPaiRWk/hbyvXq6wJTbWbbPFM5WkTPy1gmqpB4KzAX+BUww+auypEi+krK\n",
"PcaVxI7AHGAH4GjgioxfIsZexjIxLiQ2kPgH4EbgO5QRzOUp9ohmpNyjURKrS3yEMldfDZhiM9fm\n",
"ucrRIvpaxjLRiNamGfsDpwM/A95sc2/dVBGDI+UeY05iKuXD0q0pG2dck/FLxPjKWCbGjMTGEmcB\n",
"1wNXUZYMuDrFHjH+Uu7RMYk1JY4GFgHPUTaj/rTNksrRIgZWxjIxaq25+oGUSxvvA95gc3/dVBEB\n",
"KfcYJYlplMW9NgY+YHNt5UgRMUzGMtEWiU0lzgW+CXwZmJZij+g+KfcYEYm1JI4F7gV+Q5mr/5PN\n",
"85WjRcRyZCwTK9Waq78DOBW4E9jL5sG6qSJiVVLusUISr6HM1V8OvM/m+sqRImKEMpaJ3yOxhcQX\n",
"gSuA84HdU+wRvSXlHr8j8TKJTwL3AL8AdrA5z2Zp5WgR0aaMZQKJCcAhwCnATZQz9YeqhoqIjqTc\n",
"B5zE6yhz9YnAITbfqxwpIsZAxjIDSmJriYuBrwBnAa9NsUf0j5T7gJFYR+JTwB3AA5S5+pdsXqgc\n",
"LSLGUMp9QEhMkHgPcD+wLeXO0uNtnqqbLCKakJn7AJB4I2Wu/hzwdptbKkeKiIal3PuYxB9S7izd\n",
"E5gJXJK11SMGQ8YyfUhiXYnZwG3AXZR1YC5OsUcMjpR7H5GYKPE3lLn6JpSdkP7O5unK0SJinGUs\n",
"0yck3kyZqz8BHGDzr5UjRURFKfceJ7E9cBqwC/Ax4LKMXyIiY5keJbG+xFzgB62vKTZfTbFHBKTc\n",
"e47EahIfABYD6wBTbWbbPFM5WkR0kYxleojEfwPmAo8Bb7W5u3KkiOhSKfceIDEZOB3YHjgauCLj\n",
"l4hYmYxlupjEhhKfAW4A5lNGMJen2CNiVVLuXUhiDYmPUubqE4DJNmfYPFc5WkT0iEbKXdIsSY9I\n",
"urP1NaOJ4/QbCUkcACwE3gYM2XzQ5teVo0VEj2lq5m5gru25Df3+viOxM+XD0i2BI4BrMn6JiNFq\n",
"ciyjBn9335DYWOJs4DrgcmAXm6tT7BHRiSbL/cOS7pZ0nqT1GjxOT5JYU+IY4D7gWcriXp+1WVI5\n",
"WkT0AdmjO0GUNB/YdDk/Og64GXi89fhTwGa2/2qZ9xs4YdhTC2wvGFWYHiIh4E+AOcAi4Gib++um\n",
"iohuJWkIGBr21PG2VzkZGXW5j5SkbYArbe+8zPMeScB+IjGdsrjXhsCRNvMrR4qIHjPS7mzqapnN\n",
"hj08iHL1x8CS2FTiPOAa4GJgeoo9IprU1NUysyVNo1w181PgsIaO09UkJlGufDkSmEfZjPqJuqki\n",
"YhA0PpZZ4YH7eCzTmqu/C5gN3AF8zObBuqkioh+MtDuztswYk9iDMldfG3iPzYK6iSJiEGX5gTEi\n",
"sYXElyjXqs8DXpNij4haUu4dkniZxCeBe4CHKXP1eTZLK0eLiAGWscwoSUwA/hT4e+AmYHebh6qG\n",
"iohoSbmPgsTrgDMpSywcYvP9ypEiIl4i5d4Gia2BU4A/Ao4FLrR5oW6qiIjfl5n7CEisI/F3lMsa\n",
"76fM1S9IsUdEt0q5r4TEBIn3Ugp9a2CazSybpypHi4hYqYxlVkDijZTr1Z8FDrK5tXKkiIgRS7kv\n",
"Q+IPgVOBPYCZwJeztnpE9JqMZVok1pWYDdwG3ElZX/2SFHtE9KKBL3eJiRJ/Q5mrbwzsbHOSzX9W\n",
"jhYRMWoDPZaR2Jeyb+lvgf9uc0flSBERY2Igy11ie8pOSDsDxwBfy/glIvrJQI1lJNaXmAv8APg+\n",
"MMXmshR7RPSbgSh3idUkPgAspizFO9XmVJtnKkeLiGhE349lJGYApwOPAvvZ3FM5UkRE4/q23CUm\n",
"U0r9VcDRwJUZv0TEoOi7sYzEhhKfAW4A5gM72VyRYo+IQdI35S6xhsRHgfsoS/FOtjnD5rnK0SIi\n",
"xl3Pj2Vam1EfQBnB/BgYsllUN1VERF09Xe4SO1NuQtoCONzmmsqRIiK6Qk+OZSQ2kfgn4NvAN4Bd\n",
"U+wREf+lp8pdYk2JY4BFwNOUxb3+0WZJ5WgREV2lJ8Yyrbn6QcBpwA+BvW0eqJsqIqJ7dX25S0yn\n",
"bJqxAfB+m+sqR4qI6HpdO5aR2ExiHnANcBEwPcUeETEyXVfuEpMkPgEsBB6nbEb9OZullaNFRPSM\n",
"rhnLtObq7wJmA7cDr7X5cd1UERG9qSvKXWJPylx9EnCozXcrR4qI6GlVxzISW0pcQLlW/VxgjxR7\n",
"RETnap+53w2cTZmrP1k5S0RE35BdZ7FESQZvY/OzKgEiInqQJNvWKl9Xs9xHEjAiIv7LSLuz6y6F\n",
"jIiIzo263CW9U9IPJS2VtNsyPztW0o8kLZb01s5jRkREOzo5c19IWe/lhuFPSpoCvBuYAswAzpLU\n",
"d38hSBqqnaETyV9X8tfTy9nbMerStb3Y9vIW7zoQuNj2EtsPAQ8Ce472OF1sqHaADg3VDtChodoB\n",
"OjRUO0CHhmoH6MBQ7QDjoYkz6s2BR4Y9foSymUZERIyTlV7nLmk+sOlyfvQJ21e2cZxsTh0RMY46\n",
"vhRS0vXAUbbvaD3+OIDtU1qPvwkcb/uWZd6Xwo+IGIWRXAo5VneoDj/QFcBFkl7c23R74NbRhIuI\n",
"iNHp5FLIgyQ9DOwFXCXpGgDbi4BLKVvhXQN8wLXulIqIGFDV7lCNiIjmVF4VUp+SdLekuyRdJ2mr\n",
"mnnaJek0Sfe1/jd8TdIramdqx8puROtWkma0bo77kaSZtfO0S9I8SY9JWlg7S7skbSXp+tZ/M/dK\n",
"+kjtTO2QtJakW1p9s0jSybUzjYakiZLulLTSi1pq31x0qu1dbU+jLPt7fOU87boWmGp7V+AB4NjK\n",
"edq13BvRupWkicBnKTfHTQEOkTS5bqq2fYGSvxctAY6wPZUyjv1gL/37t/0M8KZW3+wCvEnSGyrH\n",
"Go3DKWPvlY5dqpa77eHL/K4D/LpWltGwPd/2C62HtwBb1szTrpXciNat9gQetP2Q7SXAJZSb5nqG\n",
"7RuB39TOMRq2/832Xa3v/wO4j3JfS8+w/XTr2zWAicC/V4zTNklbAvtT9r9Y6UUptc/ckXSSpJ8D\n",
"hwKn1M7TgfcBV9cO0ee2AB4e9jg3yFUiaRtgOuWkpmdImiDpLuAx4PrWBSC95AzgGOCFVb2w8XKX\n",
"NF/SwuV8/Q8A28fZ/gPgfErwrrKq/K3XHAc8Z/uiilGXayT5e0g+/e8CktYBvgoc3jqD7xm2X2iN\n",
"ZbYE3thL68xIOgD4le07WcVZO4zDTky29xvhSy+iC898V5Vf0nsofybtOy6B2tTGv/9e8Atg+Ifu\n",
"W/HSpS6iYZJWBy4D/tn2N2rnGS3bT0i6CngNsKBynJHaG/hjSfsDawHrSvqS7b9c3otrXy2z/bCH\n",
"BwJ31soyGpJmUP5EOrD1YU0v64Wbym4Htpe0jaQ1KKuPXlE508CQJOA8YJHtM2vnaZekjSSt1/p+\n",
"ErAfPdQ5tj9heyvb2wIHA99ZUbFD/Zn7ya0RwV2UldqOqpynXZ+hfBA8v3Vp0lm1A7VjRTeidSvb\n",
"zwMfAr5FuVrgy7bvq5uqPZIuBm4CXi3pYUnvrZ2pDa8H/pxylcmdra9euvJnM+A7rb65BbjS9nWV\n",
"M3VipWPK3MQUEdGHap+5R0REA1LuERF9KOUeEdGHUu4REX0o5R4R0YdS7hERfSjlHhHRh1LuERF9\n",
"6P8Do+dj0Sq+0gEAAAAASUVORK5CYII=\n"
],
"text/plain": [
"<matplotlib.figure.Figure at 0x7f25ab83a5c0>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"##This function provides an interactive mode to change polynomial degree. \n",
"@interact(n=[1,16])\n",
"def plot(n):\n",
" X = construct_poly(x,n)\n",
" w = wRidge(X,y,1.0)\n",
" plt.plot(x,X.dot(w.T))\n",
" print(x)\n",
" print(X.dot(w.T).shape, y.shape)\n",
" print(MSE(X.dot(w.T),y))\n",
" plt.title(\"MSE %f\" % MSE(X.dot(w.T),y))\n",
" plt.plot(x,y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"8) Apply models with different values of $d$ after being trained on the training dataset, to the test data available [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_test.csv). Compare the errors on the test data to the ones from the training by plotting the error curves as functions of the polynomial degree in a single plot! What do you conclude? <br>"
]
},
{
"cell_type": "code",
"execution_count": 82,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1.00000000e+02, 1.00000000e+01, 1.00000000e+00],\n",
" [ 1.00000000e+01, 1.00000000e+00, 1.00000000e-01],\n",
" [ 1.00000000e+00, 1.00000000e-01, 1.00000000e-02]])"
]
},
"execution_count": 82,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A.T.dot(A)"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import scipy\n",
"from scipy import linalg\n",
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1\n",
"[[ 9.80101960e-03 9.80101960e-04 9.80101960e-05]\n",
" [ 9.80101960e-04 9.80101960e-05 9.80101960e-06]\n",
" [ 9.80101960e-05 9.80101960e-06 9.80101960e-07]]\n",
"True\n"
]
}
],
"source": [
"A = np.array([[10,1,0.1],[10,1,0.2],[10,1.1,0.7]])\n",
"a = np.array([[10,1,0.1]])\n",
"A = a.T.dot(a)\n",
"\n",
"B,rank = linalg.pinv(A,return_rank=True)\n",
"print rank\n",
"print B\n",
"print np.allclose(A,A.dot(B.dot(A)))"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"## Read test data here"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"9) With a fixed optimal $d$, change the value of $\\lambda$ to one of the following values $[0.1, 1.0, 10.0]$ and find the minimum MSE!<br>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Hand in printed copy of completed notebook."
]
}
],
"metadata": {
"annotations": {
"author": "",
"categories": [
"intelligent-systems-1-2015"
],
"date": "2015-04-30",
"location": "Beginning of next lecture",
"parent": "IS_SS2015",
"submission_date": "2015-05-07",
"subtitle": "Exercise Sheet 3, Linear Regression",
"tags": [
"IntelligenSystems",
"Course"
],
"title": "Intelligent Systems 1 - Summer Semester 2015"
},
"celltoolbar": "Edit Metadata",
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Jan-Peter Hohloch