{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "preamble": true
   },
   "source": [
    "(Defining latex commands: not to be shown...)\n",
    "$$\n",
    "\\newcommand{\\norm}[1]{\\left \\| #1 \\right \\|}\n",
    "\\DeclareMathOperator{\\minimize}{minimize}\n",
    "\\newcommand{\\real}{\\mathbb{R}}\n",
    "\\newcommand{\\normal}{\\mathcal{N}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#Intelligent Systems Sheet 3\n",
    "Maximus Mutschler & Jan-Peter Hohloch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Gaussian Algebra (25 Points)\n",
    "\n",
    "Prove that the product of two univariate (scalar) Gaussian distributions is a Gaussian again, i.e. show, by explicitly performing the required arithmetic transformations, that\n",
    "\n",
    "\\begin{equation}\n",
    "   \\normal(x;\\mu,\\sigma^2)\\normal(x;m,s^2) = \\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right].\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Proof\n",
    "\\begin{align*}\n",
    "   f(x) &= \\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\cdot e^{\\frac{-(x-\\mu)^2}{2\\sigma^2}}\\cdot \\frac{1}{\\sqrt{2\\pi s^2}}\\cdot e^{\\frac{-(x-m)^2}{2s^2}}\\\\\n",
    "   &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\alpha}\\\\\n",
    "   \\text{where }\\alpha &= \\frac{s^2(x-\\mu)^2+\\sigma^2(x-m)^2}{2\\sigma^2 s^2}\\\\\n",
    "   &=\\frac{(s^2+\\sigma^2)x^2-2(s^2\\mu+\\sigma^2 m)x+s^2\\mu^2+\\sigma^2m^2}{2\\sigma^2s^2}\\\\\n",
    "   &=\\frac{x^2-2\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}x+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
    "   &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2+\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
    "   &=\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}+\\underbrace{\\frac{\\frac{s^2\\mu^2+\\sigma^2m^2}{s^2+\\sigma^2}-\\left(\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}_\\beta\\\\\n",
    "   \\beta&= \\frac{\\frac{s^4\\mu^2+s^2\\sigma^2\\mu^2+s^2\\sigma^2m^2+\\sigma^4m^2-\\left(s^4\\mu^2+2s^2\\sigma^2\\mu m+\\sigma^4m^2\\right)}{\\left(s^2+\\sigma^2\\right)^2}}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}\\\\\n",
    "   &=\\frac{s^2\\sigma^2\\left(\\mu^2-2\\mu m+m^2\\right)}{2\\sigma^2s^2\\left(s^2+\\sigma^2\\right)}\\\\\n",
    "   &=\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}\\\\\n",
    "   \\Rightarrow f(x)&=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-(\\alpha-\\beta)-\\beta}\\\\\n",
    "   &=\\frac{1}{2\\pi s\\sigma}\\cdot e^{-\\frac{\\left(x-\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2}\\right)^2}{\\frac{2\\sigma^2s^2}{s^2+\\sigma^2}}}\\cdot e^{-\\frac{(\\mu-m)^2}{2(s^2+\\sigma^2)}}\\\\\n",
    "   &=\\frac{\\sqrt{2\\pi(s^2+\\sigma^2)}\\sqrt{2\\pi\\frac{\\sigma^2s^2}{s^2+\\sigma^2}}}{2\\pi s\\sigma}\\normal\\left[x;\\frac{s^2\\mu+\\sigma^2 m}{s^2+\\sigma^2},\\frac{\\sigma^2s^2}{s^2+\\sigma^2}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n",
    "   &=\\frac{\\sqrt{2\\pi\\cdot 2\\pi\\sigma^2s^2}}{2\\pi s\\sigma}\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\\\\n",
    "   &=\\normal\\left[x; \\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\left(\\frac \\mu{\\sigma^2}+\\frac m{s^2}\\right),\\left(\\frac 1{\\sigma^2}+\\frac 1{s^2}\\right)^{-1}\\right]\\normal\\left[\\mu;m,\\sigma^2+s^2\\right]\\hspace{2cm} q.e.d.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Maximum Likelihood Estimator of Simple Linear Regression (25 Points)\n",
    "\n",
    "Derive the formula $\\mathbf{w}_{MLE} = (X^TX)^{-1}X^T\\mathbf{y}$ from the lecture, by calculating the derivative of $p(\\mathbf{y}\\,|X,\\mathbf{w}) = \\normal(\\mathbf{y}\\,|X\\mathbf{w}, \\sigma^2I)$ with respect to $\\mathbf{w}$, setting it to zero and solving it for $\\mathbf{w}$.\n",
    "\n",
    "\n",
    "Note: _To refresh your linear algebra you might find it useful to have a look in [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/pdfs/Murray_cribsheet.pdf)._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##wtf?\n",
    "\n",
    "$\\vec{w}=X^{-1}\\vec{y}$, zumindest wenn ich das ausrechne..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Linear regression (50 Points)\n",
    "\n",
    "In this exercise you will perform a regression analysis on a toy dataset. You will implement ridge regression and learn how to find a good model through a comparative performance analysis.\n",
    "\n",
    "1) Download the [training set](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_train.csv)! <br>\n",
    "2) Implement $\\mathbf{w}_{RIDGE}$ as a function of a given $X, \\mathbf{y}$ array and a regularization parameter $\\lambda$!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ":0: FutureWarning: IPython widgets are experimental and may change in the future.\n"
     ]
    }
   ],
   "source": [
    "# Loading the required packages\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.html.widgets import interact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def wRidge(X,y,lamb):\n",
    "    # Change the following line and implement the ridge regression estimator wRidge\n",
    "    a = (lamb*np.eye(X.shape[0])+X.dot(X.T))\n",
    "    b=np.linalg.inv(a)\n",
    "    c=b.T.dot(X)\n",
    "    return c.T.dot(y) # TODO pbyl wrong like this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3) Load \"ex1_train.csv\" into a numpy array! The first column in the csv file is $X$ and the second column is $\\mathbf{y}$, assign them to each variable!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Read ex1_train.csv and assign the first column and \n",
    "# second column to variables x and y respectively.\n",
    "from numpy import genfromtxt\n",
    "a=np.genfromtxt('ex1_train.csv',delimiter=' ')\n",
    "x=a[:,0].T\n",
    "y=a[:,1].T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4) Plot the training data with appropriate labels on each axes!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7f25abb529e8>"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEZCAYAAACTsIJzAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAFmxJREFUeJzt3XuQZGd93vHvo1tWWIsEBmNZ7CIsQYJYQICREZcwQBzE\n",
       "xoHIGDCOMRi05Spgl4oJZQQJkkM53BLC7lJ2EgHiZm4yBiOQZGSsNUpUXAQISxFClsKyKwwSV7Eb\n",
       "dhNdfvnj9LJ9RnPpnpnu0z3z/VRNTZ/3dJ/+zZZ0nn7f97ynU1VIknTIEV0XIEmaLAaDJKnFYJAk\n",
       "tRgMkqQWg0GS1GIwSJJaDAapA0nek+QNXdchzcVg0KqVZHeSp4/hfc5P8v4hX1a9n0GOvyvJS4ev\n",
       "TFoag0Gr2cAn345kwOdN8t+gVchg0JqQ5MVJ/keStyb5YZL/neSsvv27krwxyReS3J7kE0nu09s3\n",
       "k2TvrOPtTvL03jHOBZ6fZF+Sr87z/o9O8pUkP0nyYWBd3777JPlUktt6tV2c5KTevj8Gngy8o3f8\n",
       "Hb327Un29Gq9OsmTVvifTGuYwaC15AzgBuDngbcA75q1/4XA7wEnAncCOxY4VgFVVZcB/xH4cFWt\n",
       "r6pHz35ikmOATwDvBe4DXAQ8h8M9gfRq2dj7OQC8g+YNXgdcCby8d/xtvdd8EXhU73gfBC7qvY+0\n",
       "bAaD1pJvVdW7qrlB2PuAE5P8Qm9fAe+rquur6qfAvweel2SQ4Z6w8LDQ44Gjqmp7Vd1VVR8DvnRo\n",
       "Z1X9sKo+XlUHq2o/TdA8ZY73oO81f1ZVP6qqu6vqbcA/Av7xALVKizIYtJZ899CD3skf4Li+/f3D\n",
       "RXuAo4H7rcD7/hLw7Vlt36J3sk9yryT/rTc8dTvwt8Dxs0KpNc+Q5N8muT7Jj5P8CDh+hWqVDAap\n",
       "z8ZZj+8Avg/8H+Beh3YkORK4f99zF5sc/g5w0qy2B/W97lXAQ4Ezqup4mt5Cfy9kdig8GXg18Nyq\n",
       "OqGq7gPczuCT2dKCDAapEeB3kjwsyb2A/wBc1Bt2uhFYl2RzkqOBf0czdHPId4GTFxh2ugq4M8m2\n",
       "JEcn+Q3gcX37j6OZV7g9yX2B82a9/lbglL7t9TRzIN9PckyS1wP3XsofLc3FYNBaMdelqzXr8fuB\n",
       "99B8wj8G2AZQVbcDLwPeCdwC7Kc97HRR7/cPklx9jzeuugP4DeDFwA+A5wEf63vK24FjaXonVwGX\n",
       "zqptO/CbvSuW3g5c1vu5EdhNEyp7FvzrpSGkyy/qSfJu4F8At1XVI3pt5wPnAN/rPe3c3pUf0sgk\n",
       "uQJ4f1W9u+tapK513WO4EDhrVlsBb6uqR/d+DAWNi2P0Eh0HQ1VdCfxojl3+D6ouuMJYovsew3y2\n",
       "JvlaknclOaHrYrT6VdVTHUaSGpMYDH8KPBg4nWYS8D93W44krS1HdV3AbFV126HHSd4JXDz7OUns\n",
       "8kvSElTVokP1ExcMSU6squ/0Ns8Grp3reYP8cZMqyflVdX7XdSyV9XfL+rszzbXD4B+qOw2GJB+i\n",
       "WeV5v97dK88DZpKcTjMR+E3g9zssUZLWnE6DoapeMEezE4CS1KFJnHxeC3Z1XcAy7eq6gGXa1XUB\n",
       "y7Sr6wKWaVfXBSzDrq4LGIdOVz4vVZKa5jkGSerCoOfOiZt8liStrGTTZtiwbfFnNgwGSVrFmlA4\n",
       "cztccOqgN5VwjkGSVrUN25pQGJzBIEmr2vp1w77CoSRJq8LhcfT162DfQdi7o+q6S7quq3v7Dg77\n",
       "CoNB0tRrj6MfsuWUZBOGw94dsOWUYYaTvFxV0liM8hN98szL4NJn3HPP5suqLnnmSrzHNGv+7Tdu\n",
       "hUvP8nJVSRNh9J/o5xtHP+7Y5R97+oeperVeMhX3SpK0Vsx1ZcwFp8LmrcAKnGDnG0fff2C5R16L\n",
       "w1RelSRpDEb7ib43jn5Tu+2cm2HPzuUfe75Q27h1+ceeTPYYJI3B6D7RQzNUkmyi6YEcd2xz3D07\n",
       "p2GYahIZDJLGYK4rY1bqE33j0Dj6Sh3vsNGG2iQyGCSN3Gg/0Y/a6ENt0ni5qiQt4vDlntMWam2D\n",
       "njsNBklaIwY9d3pVkiSpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1\n",
       "GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqSWToMhybuT3Jrk2r62+ya5\n",
       "PMmNST6T5IQua5SktabrHsOFwFmz2l4DXF5VDwU+29uWJI1Jp8FQVVcCP5rV/Czgvb3H7wX+1ViL\n",
       "kqQ1rusew1weUFW39h7fCjygy2Ikaa05qusCFlJVlaTm2pfk/L7NXVW1ayxFSdKUSDIDzAz9uqo5\n",
       "z7tjk+Rk4OKqekRv+wZgpqq+m+RE4Iqq+iezXlNVlbEXK0lTbNBz5yQOJX0SeFHv8YuAT3RYiySt\n",
       "OZ32GJJ8CHgKcD+a+YTXA38JfBTYCOwGnldVP571OnsMkjSkQc+dnQ8lLYXBIEnDm+ahJElShwwG\n",
       "SVKLwSBJajEYJEktE73ATdJhyabNsGEbrF8H+w7C3h1V113SdV1afQwGaQo0oXDmdrjg1MOtW05J\n",
       "NmE4aKU5lCRNhQ3b2qEAzfbGrd3Uo9XMYJCmwvp1c7cfd+x469BaYDBIU2Hfwbnb9x8Ybx1aCwwG\n",
       "aSrs3QFbbmq3nXMz7NnZTT1azbwlhjQlmgnojVub4aP9B2DPTieeNQzvlSRJavFeSZKkJTEYJEkt\n",
       "BoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWoxGCRJLQaDJKnFYJAktRgMkqQWg0GS1GIw\n",
       "SJJaDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWo7quoD5JNkN/AS4C7ijqs7o\n",
       "tiJJWhsmNhiAAmaq6oddFyJJa8mkDyWl6wIkaa2Z5GAo4K+TXJ1ky+ydyTMvSzZt7qAuSVrVJnko\n",
       "6YlV9Z0k9wcuT3JDVV15ePelz4AtpySbqLruks6qlKRVZmKDoaq+0/v9vSQfB84A+oLhfOCkU2Hf\n",
       "9iQ/rapdHZQpSRMryQwwM/TrqmqxA/98Vf1gaWUtTZJ7AUdW1b4kPwd8BvijqvpMb381I00Az/vb\n",
       "qo/OjLM+SZpGSaqqFp27HaTH8Pkk1wAXApfWYkmyMh4AfDwJNDX+2aFQuKf9B8ZQjyStGYP0GI4A\n",
       "/hnwEuBxwEeBC6vqxtGXN29NvR7DOTfD57c5xyBJixu0x7BoMMw66NOADwA/B1wDnFtVVy25yiVq\n",
       "guGZl8GenYaCJA1mxYIhyf2Afw38LnAr8E7gYuBRwJ9X1cnLrnZIg/5xkqTDVnKO4SqaXsKzq+qW\n",
       "vvark/zXpRYoSZpMA80xVNXdY6pnIPYYJGl4g547F135PGmhIEkarUm+JYYkqQOLBkOSJ83R9sTR\n",
       "lCNJ6togPYadc7S9Y6ULkSRNhnmvSkpyJvAE4P5J/oDDt8Bej0NQkrRqLXS56jE0IXBk7/chPwF+\n",
       "c5RFSZK6M8jlqg+qqm+NqZ6BeLmqJA1vJVc+XzFHc1XV05Za3HIZDJI0vJVc+fzqvsfrgOcAdy61\n",
       "MEnSZBvqJno/e1Hypap63AjqGfT97TFI0pBWrMeQ5L59m0cAvwLcexm1SZIm2CBDSV/h8Nel3Qns\n",
       "Bl46qoIkSd1a0lBS1xxKkqThreRQ0rHAy4An0fQcrgT+tKoOLrtKSdLEGeRy1YtoFrV9gGb1828D\n",
       "x1fVc0df3rw12WOQpCGt5DqG66vqtMXaxslgkKThrdj3MQBf6d036dCBHw98eTnFSZIm1yA9hhuA\n",
       "hwJ7aeYYNgLfoLlCqarqkaMuco6a7DFI0pBWcuXzMzh8Z9VDao42SdIqMEiP4f1V9cLF2sbJHoMk\n",
       "DW8l5xg2zTrwUcBjl1qYJGmyzRsMSV6bZB/wiCT7Dv0AtwGfHFuFkqSxGmQo6U1V9Zox1TMQh5Ik\n",
       "aXgruY7hKRy+V9LPVNXnll7e8hgMkjS8lQyGT3E4GNYBZwBf9ot6JGm6rNjlqlX167MOvAHYvoza\n",
       "JEkTbJCrkma7BXjYShciSZoMg9xddWff5hHA6XhLDElatQZZ+fxlDs8x3AV8sKr+5+hKkiR1aZDJ\n",
       "52OBU2nC4aZJ+B4GJ58laXjLXvmc5Ogkb6G5ed57gfcBtyR5a5KjV65USdIkWWjy+a3AfYEHV9Vj\n",
       "quoxwC8DJwD/aRzFSZLGb96hpCQ3AQ+tqrtntR8JfKOqTh1ZUclZwNuBI4F3VtWbZ+13KEmShrQS\n",
       "N9G7e3YoAFTVXcA92ldKL3jeAZwFnAa8IImXx0rSmCwUDF9P8qLZjUleCNwwupI4g2aSe3dV3QF8\n",
       "GHj2CN9Pa0SyaXPyzMuS5+1qfm/a3HVN0iRa6HLVlwN/keQlHF638FjgXsDZI6zpJJoJ70NuAX51\n",
       "hO+nNaAJgTO3wwV9Q6BbTkk2UXXdJd1VJk2eeYOhqm5J8qvA04CH01yu+umq+uyIa1r4+llpSTZs\n",
       "a4cCNNubtwIGg9RnwQVu1cxMf7b3My7fBjb0bW+g6TW0JDm/b3NXVe0abVmabuvXzd1+3LHjrUMa\n",
       "nyQzwMywrxtk5fO4XQ08JMnJwD8AzwdeMPtJVXX+WKvSlNs3z8LM/QfGW4c0Pr0PzLsObSc5b5DX\n",
       "LeUmeiNVVXcCrwD+Crge+EhVfb3bqjT99u6ALTe12865GfbsnPv50tq16C0xJpHrGLQUzQT0xq3N\n",
       "8NH+A7BnpxPPWktW7It6JpHBIEnDW4kFbpKkNWgSJ5870ww1bNjWXMGy7yDs3eFQg6S1xmDocQGU\n",
       "JDUcSvqZ+RZAbdzaTT2S1A2D4WdcACVJYDD0cQGUJIHB0McFUJIErmOYdVwXQElavVzgJklqcYGb\n",
       "JGlJDAZJUovBIElqMRgkSS0GgySpxWCQJLUYDJKkFoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMk\n",
       "qcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWo5qusCpEOSTZthwzZYvw72HYS9O6quu6TruqS1\n",
       "xmDQRGhC4cztcMGph1u3nJJswnCQxsuhJE2IDdvaoQDN9sat3dQjrV0GgybE+nVztx937HjrkGQw\n",
       "aELsOzh3+/4D461DksGgCbF3B2y5qd12zs2wZ2c39UhrV6qq6xqGlqSqKl3XoZXVTEBv3NoMH+0/\n",
       "AHt2OvEsrZxBz50TFwxJzgfOAb7Xazq3qi6b9RyDQZKGNOi5cxIvVy3gbVX1tq4LkaS1aFLnGOwN\n",
       "SFJHJjUYtib5WpJ3JTmh62IkaS3pZI4hyeXAL86x63XA5zk8v/AG4MSqeums1xfwR31Nu6pq1whK\n",
       "laSplWQGmOlrOm8qJ5/7JTkZuLiqHjGr3clnSRrSoOfOiRtKSnJi3+bZwLVd1SJJa9EkXpX05iSn\n",
       "01yd9E3g9zuuR5LWlIkeSpqPQ0mSNLypHUqSJHXLYJAktRgMkqQWg0GS1GIwSJJaDAZJUovBIElq\n",
       "MRgkSS2TuPJZE6r5hrUN22D9uuY7mvfu8BvWpNXHYNBAmlA4cztccOrh1i2nJJswHKTVxaEkDWjD\n",
       "tnYoQLO9cWs39UgaFYNBA1q/bu72444dbx2SRs1g0ID2HZy7ff+B8dYhadQMBg1o7w7YclO77Zyb\n",
       "Yc/ObuqRNCredlsDayagN25tho/2H4A9O514lqbHoOdOg0GS1gi/j0GStCQGgySpxWCQJLUYDJKk\n",
       "FoNBktRiMEiSWgwGSVKLwSBJajEYJEktBoMkqcVgkCS1GAySpBaDQZLUYjBIkloMBklSi8EgSWox\n",
       "GCRJLQaDJKnFYJAktXQSDEmem+R/JbkryWNm7Ts3yd8nuSHJP++iPklay7rqMVwLnA18rr8xyWnA\n",
       "84HTgLOAP0my6no1SWa6rmE5rL9b1t+daa59GJ2cdKvqhqq6cY5dzwY+VFV3VNVu4CbgjLEWNx4z\n",
       "XRewTDNdF7BMM10XsEwzXRewTDNdF7AMM10XMA6T9mn8l4Bb+rZvAU7qqBZJWpOOGtWBk1wO/OIc\n",
       "u15bVRcPcahaoZIkSQNIVXfn3SRXAK+qqq/0tl8DUFVv6m1fBpxXVV+Y9TrDQpKWoKqy2HNG1mMY\n",
       "Qn+RnwQ+mORtNENIDwG+OPsFg/xhkqSl6epy1bOT7AUeD3w6yaUAVXU98FHgeuBS4GXVZZdGktag\n",
       "ToeSJEmTZ9KuShpYkjck+VqSa5J8NsmGrmsaRpK3Jvl672/4iyTHd13TMBZapDipkpzVWzj590n+\n",
       "sOt6hpXk3UluTXJt17UMK8mGJFf0/pu5Lsm2rmsaRpJ1Sb7QO99cn+SNXde0FEmOTPLVJAteADS1\n",
       "wQC8paoeVVWnA58Azuu6oCF9Bnh4VT0KuBE4t+N6hjXnIsVJleRI4B00CydPA16Q5GHdVjW0C2nq\n",
       "n0Z3AP+mqh5OM4T88mn696+qg8BTe+ebRwJPTfKkjstailfSDNUvOFQ0tcFQVfv6No8Dvt9VLUtR\n",
       "VZdX1d29zS8AD+yynmEtsEhxUp0B3FRVu6vqDuDDNAsqp0ZVXQn8qOs6lqKqvltV1/Qe7we+TrNu\n",
       "aWpU1U97D48BjgR+2GE5Q0vyQGAz8E7aF/3cw9QGA0CSP06yB3gR8Kau61mGlwCXdF3EKncSsLdv\n",
       "28WTHUlyMvBomg9EUyPJEUmuAW4FruhdLDNN/gvwauDuxZ440cGQ5PIk187x8y8Bqup1VbUReA/N\n",
       "Hz1RFqu/95zXAf+vqj7YYalzGqT+KeJVFhMgyXHAnwOv7PUcpkZV3d0bSnog8E+n6b5JSX4duK2q\n",
       "vsoivQWYjHUM86qqXxvwqR9kAj9xL1Z/khfTdO2ePpaChjTEv/80+DbQf4HCBtq3X9GIJTka+Bjw\n",
       "gar6RNf1LFVV3Z7k08CvALs6LmdQTwCelWQzsA64d5L3VdXvzvXkie4xLCTJQ/o2nw18tataliLJ\n",
       "WTTdumf3Jram2TQsOLwaeEiSk5McQ3MX3092XNOakSTAu4Drq+rtXdczrCT3S3JC7/GxwK8xReec\n",
       "qnptVW2oqgcDvwX8zXyhAFMcDMAbe8Ma19Dc8fBVHdczrJ00k+aX9y4f+5OuCxrGfIsUJ1VV3Qm8\n",
       "AvgrmqsyPlJVX++2quEk+RBwFfDQJHuT/F7XNQ3hicDv0FzN89XezzRdYXUi8De9880XgIur6rMd\n",
       "17QcCw6tusBNktQyzT0GSdIIGAySpBaDQZLUYjBIkloMBklSi8EgSWoxGKRZkqz4rRqSPCjJC1b6\n",
       "uNIoGAzSPY1icc+Dgd8ewXGlFWcwSPNIMpNkV5KLel+q9IG+fbuTvDnJ3/W+wOWUXvt7kjyn73mH\n",
       "bg//JuDJvRW/r5z1Pmcn+eve4xOTfCPJL4z+L5TmZjBICzud5stNTgN+OckTeu0F/LiqHknzBUBv\n",
       "72ufyx8CV1bVo6tqe/+Oqvo48J0krwD+O/D6qrpthf8OaWAGg7SwL1bVP1Rz75hrgJP79n2o9/vD\n",
       "wJmLHGexGw1upfkWv4NV9ZGlFCqtFINBWtj/7Xt8F/Pfqv5QT+FOev9fJTmC5tu+BrGhd/wH9O5E\n",
       "KnXGYJCW7vl9v6/qPd4NPLb3+FnA0b3H+4D1cx0kyVE0t6T+LeAG4A9GUKs0sIn+oh6pIzXP49nu\n",
       "k+RrwEHg0KWoFwB/2bs982XAoUtfvwbc1Wu/cNY8w7nA56rqqiR/B3wpyaeq6hsr8cdIw/K229IS\n",
       "JPkm8NiqmqovhJcG4VCStDR+otKqZY9BktRij0GS1GIwSJJaDAZJUovBIElqMRgkSS0GgySp5f8D\n",
       "oJt+CXUlyycAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f25abb6fb00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the input data here\n",
    "plt.plot(x,y,'o')\n",
    "plt.title('Input data')\n",
    "plt.ylabel('Output y')\n",
    "plt.xlabel('Input x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5) Implement a function which constructs features upto a input polynomial degree $d$!<br>\n",
    "Note: _Constructing higher polynomial features is similar to what you implemented in Exercise 3 (SVM) of the previous exercise sheet._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def construct_poly(x,d):\n",
    "    ## Implement a method which given an array of size N, \n",
    "    ## returns an array of dimension (N,d)\n",
    "    return (np.ones((d,1))*np.asmatrix(x)).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-3.  -3. ]\n",
      " [-2.5 -2.5]\n",
      " [-1.  -1. ]\n",
      " [-0.5 -0.5]\n",
      " [ 0.   0. ]\n",
      " [ 1.   1. ]\n",
      " [ 2.   2. ]\n",
      " [ 2.5  2.5]\n",
      " [ 3.   3. ]\n",
      " [ 4.   4. ]]\n",
      "[[ 52.75  52.75]\n",
      " [ 52.75  52.75]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "matrix([[ 1.51220623,  1.51220623]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#try it out\n",
    "X=construct_poly(x,2)\n",
    "print(X)\n",
    "print(X.T.dot(X))\n",
    "wRidge(X,y,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6) Implement the Mean Squared Error Loss (MSE) as a function of the predicted and true values of the target variable! <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def MSE(y_predict,y_true):\n",
    "    ## Implement mean squared error for a given input y and its predictions.\n",
    "    return ((y_predict-y_true)**2)/y_predict.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7) By comparing the MSE find the degree $d$ for the polynomial that fits the training data best! You might find it useful to use the code below to interactively change the variable $d$, set $\\lambda = 1$ and keep it fixed. Plot the error as a function of different values of $d$!<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-3.  -2.5 -1.  -0.5  0.   1.   2.   2.5  3.   4. ]\n",
      "(10, 1) (10,)\n",
      "[[   7.24825764  -79.61751096 -163.32510817 -124.420806    -64.36110538\n",
      "    64.00498155  130.78610485  134.65832608  128.64828886  140.65345344]\n",
      " [   2.38125049  -75.48579685 -150.52183853 -115.64776572  -61.80985436\n",
      "    53.25835177  113.12139072  116.59247535  111.20503839  121.96654702]\n",
      " [ -12.21977095  -63.09065449 -112.11202963  -89.32864491  -54.15610128\n",
      "    21.01846246   60.12724836   62.39492317   58.87528698   65.90582776]\n",
      " [ -17.0867781   -58.95894038  -99.30875999  -80.55560464  -51.60485026\n",
      "    10.27183269   42.46253424   44.32907244   41.4320365    47.21892134]\n",
      " [ -21.95378525  -54.82722626  -86.50549036  -71.78256437  -49.05359924\n",
      "    -0.47479708   24.79782012   26.26322172   23.98878603   28.53201492]\n",
      " [ -31.68779954  -46.56379802  -60.89895109  -54.23648382  -43.95109719\n",
      "   -21.96805662  -10.53160812   -9.86847974  -10.89771491   -8.84179791]\n",
      " [ -41.42181383  -38.30036979  -35.29241182  -36.69040328  -38.84859514\n",
      "   -43.46131617  -45.86103637  -46.00018119  -45.78421585  -46.21561075]\n",
      " [ -46.28882098  -34.16865567  -22.48914219  -27.91736301  -36.29734412\n",
      "   -54.20794594  -63.52575049  -64.06603192  -63.22746633  -64.90251717]\n",
      " [ -51.15582813  -30.03694156   -9.68587255  -19.14432273  -33.74609309\n",
      "   -64.95457571  -81.19046461  -82.13188264  -80.6707168   -83.58942359]\n",
      " [ -60.88984242  -21.77351332   15.92066671   -1.59824219  -28.64359104\n",
      "   -86.44783525 -116.51989285 -118.2635841  -115.55721774 -120.96323643]]\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "only length-1 arrays can be converted to Python scalars",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-95-b005297af287>\u001b[0m in \u001b[0;36mplot\u001b[1;34m(n)\u001b[0m\n\u001b[0;32m      8\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m     \u001b[0mprint\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m     \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtitle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"MSE %f\"\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mMSE\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mw\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mT\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     11\u001b[0m     \u001b[0mplt\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mplot\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mx\u001b[0m\u001b[1;33m,\u001b[0m\u001b[0my\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: only length-1 arrays can be converted to Python scalars"
     ]
    },
    {
     "data": {
      "image/png": [
       "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\n",
       "AAALEgAACxIB0t1+/AAAGDdJREFUeJzt3Xm43XV97fH3SpgiiMwyFigiJGFIQBDRylHEG7ncUrwO\n",
       "0Am1rdw6IZMRuVcClkIgBKoWikJEKYMoKlBAiUgERaYyBUNAVBSUIt4ql0KBENb947uxh5jh7LPP\n",
       "73z3sF7Pcx7O3mfv81vy8Cx/57N/v+9XtomIiP4yoXaAiIgYeyn3iIg+lHKPiOhDKfeIiD6Uco+I\n",
       "6EMp94iIPtRxuUuaJ+kxSQuHPTdL0iOS7mx9zej0OBERMXJjceb+BWDZ8jYw1/b01tc3x+A4EREx\n",
       "Qh2Xu+0bgd8s50fq9HdHRMToNDlz/7CkuyWdJ2m9Bo8TERHLaKrczwa2BaYBjwKnN3SciIhYjtWa\n",
       "+KW2f/Xi95LOBa5c9jWSsqhNRMQo2F7l2LuRcpe0me1HWw8PAhYu73UjCditJM2yPat2jtFK/rqS\n",
       "v55ezg4jPzHuuNwlXQzsA2wk6WHgeGBI0jTKVTM/BQ7r9DgRETFyHZe77UOW8/S8Tn9vRESMXu5Q\n",
       "Hb0FtQN0aEHtAB1aUDtAhxbUDtChBbUDdGBB7QDjQbU265DkXp65R0TUMNLuzJl7REQfSrlHRPSh\n",
       "lHtERB9KuUdE9KGUe0REH0q5R0T0oZR7REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oZR7\n",
       "REQfSrlHRPShlHtERB9KuUdE9KGUe0REH0q5R0T0oY43yI6IiOZJTAT+aqSvT7lHRHQ5iX2BM4Df\n",
       "jvQ9KfeIiC4l8WpgDrATcAzwNeCFkbw3M/eIiC4jsb7EXOAm4HvAFJvLbDzS35Fyj4joEhKrSXwQ\n",
       "WAysDUy1OdXmmXZ/V8YyERFdQGIGMBf4JbCfzT2d/L6Ue0RERRJTgNOB7YCjgSvbGb+sSMYyEREV\n",
       "SGwo8Rngu8C1wE42V4xFsUPKPSJiXEmsIfFRylxdwGSbM2yeG8vjZCwTETEOJAQcQBnB/BjYx2ZR\n",
       "U8dLuUdENExiF8qHpZsDh9tc0/QxM5aJiGiIxCYS5wDzgW8Au45HsUPKPSJizEmsKXEMsAh4CtjR\n",
       "5rM2S8YrQ8YyERFjpDVXfztwKvBDYG+bB2pkSblHRIwBid0oi3ttABxm8+2aeTKWiYjogMRmEvOA\n",
       "q4GLgOm1ix3GoNwlzZP0mKSFw57bQNJ8SQ9IulbSep0eJyKim0hMkvgEsBB4HNjB5hyb5ytHA8bm\n",
       "zP0LwIxlnvs4MN/2q4HrWo8jInqehCQOBu4DdgdeazPT5onK0V5Cdud3ukraBrjS9s6tx4uBfWw/\n",
       "JmlTYIHtHZd5j22r44NHRIwTiT0pc/VJwBE23x3/DCPrzqZm7q+0/Vjr+8eAVzZ0nIiIxklsKXEB\n",
       "5Vr1c4E9ahR7Oxq/Wsa2JS33zwNJs4Y9XGB7QdN5IiJGSmJtyg5IHwbOpszVnxzfDBoChtp+X4Nj\n",
       "mSHb/yZpM+D6jGUioldITAD+FDgZ+D4w0+ZndVMVtccyVwCHtr4/lPKnTERE15PYG7gZ+AhwsM3B\n",
       "3VLs7ej4zF3SxcA+wEaU+fongcuBS4E/AB4C3mX7t8u8L2fuEdE1JLYGZgOvB44FLrJHthn1eBpp\n",
       "d47JWGY0Uu4R0Q0kXk65XPt/AZ8G5tg8VTfVitUey0REdDWJCRLvpWyasRVlxcYTurnY25G1ZSJi\n",
       "4EjsQ7le/RngIJtbK0cacyn3iBgYEttRVmzcHZgJXDpWe5Z2m4xlIqLvSbxC4lTgFuAOyr6lX+7X\n",
       "YoeUe0T0MYmJEocB9wMbAjvbnGTzn5WjNS5jmYjoSxJvoexb+htgf5s7KkcaVyn3iOgrEq8G5gBT\n",
       "KUsHfL2fxy8rkrFMRPQFifUlzgBuAm4Epth8bRCLHVLuEdHjJFaX+BDlevVJlFI/zebZytGqylgm\n",
       "InqWxAzKXP2XwFtsFq7iLQMj5R4RPUdiCnA6sB1wFPAvgzp+WZGMZSKiZ0hsJPFZ4LvAt4CdbK5M\n",
       "sf++lHtEdD2JNSSOoOxbamBHmzNtnqscrWtlLBMRXUtCwAGUEcyDwD42i+qm6g0p94joShK7UD4s\n",
       "3Rw43OaaypF6SsYyEdFVJDaROAeYD3ydshRvir1NKfeI6AoSa0p8DFgEPEWZq/+jzZLK0XpSxjIR\n",
       "UVVrrv52ylK89wJ72zxQN1XvS7lHRDUSu1E2zVgfeL/NdZUj9Y2MZSJi3ElsJjEPuAq4EJieYh9b\n",
       "KfeIGDcSkySOo4xfHqfM1T9ns7RytL6TsUxENK41V383cApwO7CHzU/qpupvKfeIaJTEnpS5+iTg\n",
       "L21uqBxpIGQsExGNkNhS4gLKteqfB16TYh8/KfeIGFMSa0vMAu4GfgbsYHO+zQt1kw2WjGUiYkxI\n",
       "TAD+DPh7yk5I021+XjfV4Eq5R0THJPYGzqSs2Phum5sqRxp4KfeIGDWJrYHZwOuBjwMXZ/zSHTJz\n",
       "j4i2Sbxc4iTgXylrwexgc2GKvXuk3CNixCQmSrwPuB/YkrJi44k2T1eOFsvIWCYiRkRiH8r16k8D\n",
       "B9rcVjlSrETKPSJWSmI7yoqNuwEzga9kz9Lul7FMRCyXxCskTgVuoSwZMNnm0hR7b0i5R8RLSKwm\n",
       "cRhlrr4BsJPNyTbPVI4WbchYJiJ+R2I/yr6lvwbeZnNn5UgxSo2Wu6SHgP8HLAWW2N6zyeNFxOhI\n",
       "7ADMASYDRwOXZ/zS25o+czcwZPvfGz5ORIyCxPrAJ4E/p9yM9A6bZ+umirEwHjN3jcMxIqINEqtL\n",
       "fJgyV18LmGozJ8XeP8bjzP3bkpYC59j+fMPHi4hVkHgbZa7+MLCvzcLKkaIBTZf7620/KmljYL6k\n",
       "xbZvbPiYEbEcElOB04FtgSOBqzNX71+NlrvtR1v/fFzS14E9KUuBAiBp1rCXL7C9oMk8EYNIYiPg\n",
       "BOCdwEnAWTZL6qaKkZI0BAy1/T67mf/jlvQyYKLtJyWtDVwLnGD72tbPbTvz+IiGSKwBfAg4FrgY\n",
       "OMHm/9ZNFZ0aaXc2eeb+SuDrkl48zoUvFntENKe1GfUfUy5tfAB4o819dVPFeGvszH2VB86Ze8SY\n",
       "k9iV8mHppsCRNt+qHCnG2Ei7M8sPRPQBiVdKfI4y/vwqZSneFPsAS7lH9DCJtSRmAj8EnqRsmnG2\n",
       "zfOVo0VlWVsmoge15ur/k7IU7z3A62x+VDdVdJOUe0SPkdidsmnGK4C/tvlO5UjRhTKWiegREptL\n",
       "fAH4F+BLwG4p9liRlHtEl5OYJPG/gYXAY5S5+rk2SytHiy6WsUxEl2rN1Q8GTgFuBfaw+UndVNEr\n",
       "Uu4RXUhiL8pcfQ3gL2xuqBwpekzGMhFdRGIriQuBy4BzKGfrKfZoW8o9ogtIrCNxInAX8BPKXP18\n",
       "mxcqR4selbFMREUSE4C/oKzWeAMw3ebndVNFP0i5R1Qi8QbKXH0p8E6bH1SOFH0k5R4xziS2pexX\n",
       "uhet5Xgzfomxlpl7xDiRWFfiZOB24F5gR5sLU+zRhJR7RMMkJkr8NbAY2BzYxeZEm6crR4s+lrFM\n",
       "RIMk3kSZq/8HcKDNbZUjxYBIuUc0QOJVwGnANGAm8JVsRh3jKWOZiDEksZ7EHOBm4BZgss2lKfYY\n",
       "byn3iDEgsZrE31Lm6usBO9mcYvNM5WgxoDKWieiQxFsp+5Y+DsywuatypIiUe8RoSewIzAF2BI4G\n",
       "Ls/4JbpFxjIRbZLYQOIfgO8B1wNTbb6RYo9uknKPGCGJ1SU+Qpmrr075sPR0m2crR4v4PRnLRKxC\n",
       "a9OM/YHTgZ8Db7a5t26qiJVLuUeshMRUyoelWwNHAVdn/BK9IGOZiOWQ2FjiLMpM/SpgZ5urUuzR\n",
       "K1LuEcNIrCFxFLAIWEJZ3OvTNksqR4toS8YyEfxurn4gZcmA+4E/sllcN1XE6KXcY+BJ7EpZ3GsT\n",
       "4IM211aOFNGxjGViYElsKvF54FrgK8C0FHv0i5R7DByJtSQ+Ttkw4wnKZtRn2zxfOVrEmMlYJgZG\n",
       "a67+DuBU4C5gL5sH66aKaEbKPQaCxO7AmcDLgffZXF85UkSjMpaJviaxucT5wJXAF4HdU+wxCFLu\n",
       "0ZckXibxf4B7gEcp16ufa7O0crSIcZGxTPSV1lz9EOAUym5Ie9j8tG6qiPHX2Jm7pBmSFkv6kaSZ\n",
       "TR0n4kUSewE3AUcCf2bzrhR7DCrZY79UhqSJlLv83gL8ArgNOMT2fcNeY9sa84PHwJHYinKmvg9w\n",
       "HHCBzQt1U0U0Y6Td2dSZ+57Ag7Yfsr0EuIRya3fEmJFYR+JEymWND1KuV/9iij2iuXLfAnh42ONH\n",
       "Ws9FdExigsShlE0ztgOm2xxv81TlaBFdo6kPVLMsajRC4g2U69WfB95hc3PlSBFdqaly/wWw1bDH\n",
       "W1HO3l9C0qxhDxfYXtBQnuhxEtsCs4G9gJnAJVlbPQaBpCFgqO33NfSB6mqUD1T3BX4J3Eo+UI1R\n",
       "kFgXOBZ4P2Xlxrk2T9dNFVHPSLuzkTN3289L+hDwLWAicN7wYo9YFYmJwHuBEymrNu5s88u6qSJ6\n",
       "RyNn7iM6cM7cYwUk3kQ5S38SOMLm9sqRIrpG1TP3iNGQeBVlJ6RpwDHAZZmrR4xO1paJ6iTWk5hD\n",
       "WS7gZmCyzVdT7BGjl3KPaiRWk/hbyvXq6wJTbWbbPFM5WkTPy1gmqpB4KzAX+BUww+auypEi+krK\n",
       "PcaVxI7AHGAH4GjgioxfIsZexjIxLiQ2kPgH4EbgO5QRzOUp9ohmpNyjURKrS3yEMldfDZhiM9fm\n",
       "ucrRIvpaxjLRiNamGfsDpwM/A95sc2/dVBGDI+UeY05iKuXD0q0pG2dck/FLxPjKWCbGjMTGEmcB\n",
       "1wNXUZYMuDrFHjH+Uu7RMYk1JY4GFgHPUTaj/rTNksrRIgZWxjIxaq25+oGUSxvvA95gc3/dVBEB\n",
       "KfcYJYlplMW9NgY+YHNt5UgRMUzGMtEWiU0lzgW+CXwZmJZij+g+KfcYEYm1JI4F7gV+Q5mr/5PN\n",
       "85WjRcRyZCwTK9Waq78DOBW4E9jL5sG6qSJiVVLusUISr6HM1V8OvM/m+sqRImKEMpaJ3yOxhcQX\n",
       "gSuA84HdU+wRvSXlHr8j8TKJTwL3AL8AdrA5z2Zp5WgR0aaMZQKJCcAhwCnATZQz9YeqhoqIjqTc\n",
       "B5zE6yhz9YnAITbfqxwpIsZAxjIDSmJriYuBrwBnAa9NsUf0j5T7gJFYR+JTwB3AA5S5+pdsXqgc\n",
       "LSLGUMp9QEhMkHgPcD+wLeXO0uNtnqqbLCKakJn7AJB4I2Wu/hzwdptbKkeKiIal3PuYxB9S7izd\n",
       "E5gJXJK11SMGQ8YyfUhiXYnZwG3AXZR1YC5OsUcMjpR7H5GYKPE3lLn6JpSdkP7O5unK0SJinGUs\n",
       "0yck3kyZqz8BHGDzr5UjRURFKfceJ7E9cBqwC/Ax4LKMXyIiY5keJbG+xFzgB62vKTZfTbFHBKTc\n",
       "e47EahIfABYD6wBTbWbbPFM5WkR0kYxleojEfwPmAo8Bb7W5u3KkiOhSKfceIDEZOB3YHjgauCLj\n",
       "l4hYmYxlupjEhhKfAW4A5lNGMJen2CNiVVLuXUhiDYmPUubqE4DJNmfYPFc5WkT0iEbKXdIsSY9I\n",
       "urP1NaOJ4/QbCUkcACwE3gYM2XzQ5teVo0VEj2lq5m5gru25Df3+viOxM+XD0i2BI4BrMn6JiNFq\n",
       "ciyjBn9335DYWOJs4DrgcmAXm6tT7BHRiSbL/cOS7pZ0nqT1GjxOT5JYU+IY4D7gWcriXp+1WVI5\n",
       "WkT0AdmjO0GUNB/YdDk/Og64GXi89fhTwGa2/2qZ9xs4YdhTC2wvGFWYHiIh4E+AOcAi4Gib++um\n",
       "iohuJWkIGBr21PG2VzkZGXW5j5SkbYArbe+8zPMeScB+IjGdsrjXhsCRNvMrR4qIHjPS7mzqapnN\n",
       "hj08iHL1x8CS2FTiPOAa4GJgeoo9IprU1NUysyVNo1w181PgsIaO09UkJlGufDkSmEfZjPqJuqki\n",
       "YhA0PpZZ4YH7eCzTmqu/C5gN3AF8zObBuqkioh+MtDuztswYk9iDMldfG3iPzYK6iSJiEGX5gTEi\n",
       "sYXElyjXqs8DXpNij4haUu4dkniZxCeBe4CHKXP1eTZLK0eLiAGWscwoSUwA/hT4e+AmYHebh6qG\n",
       "iohoSbmPgsTrgDMpSywcYvP9ypEiIl4i5d4Gia2BU4A/Ao4FLrR5oW6qiIjfl5n7CEisI/F3lMsa\n",
       "76fM1S9IsUdEt0q5r4TEBIn3Ugp9a2CazSybpypHi4hYqYxlVkDijZTr1Z8FDrK5tXKkiIgRS7kv\n",
       "Q+IPgVOBPYCZwJeztnpE9JqMZVok1pWYDdwG3ElZX/2SFHtE9KKBL3eJiRJ/Q5mrbwzsbHOSzX9W\n",
       "jhYRMWoDPZaR2Jeyb+lvgf9uc0flSBERY2Igy11ie8pOSDsDxwBfy/glIvrJQI1lJNaXmAv8APg+\n",
       "MMXmshR7RPSbgSh3idUkPgAspizFO9XmVJtnKkeLiGhE349lJGYApwOPAvvZ3FM5UkRE4/q23CUm\n",
       "U0r9VcDRwJUZv0TEoOi7sYzEhhKfAW4A5gM72VyRYo+IQdI35S6xhsRHgfsoS/FOtjnD5rnK0SIi\n",
       "xl3Pj2Vam1EfQBnB/BgYsllUN1VERF09Xe4SO1NuQtoCONzmmsqRIiK6Qk+OZSQ2kfgn4NvAN4Bd\n",
       "U+wREf+lp8pdYk2JY4BFwNOUxb3+0WZJ5WgREV2lJ8Yyrbn6QcBpwA+BvW0eqJsqIqJ7dX25S0yn\n",
       "bJqxAfB+m+sqR4qI6HpdO5aR2ExiHnANcBEwPcUeETEyXVfuEpMkPgEsBB6nbEb9OZullaNFRPSM\n",
       "rhnLtObq7wJmA7cDr7X5cd1UERG9qSvKXWJPylx9EnCozXcrR4qI6GlVxzISW0pcQLlW/VxgjxR7\n",
       "RETnap+53w2cTZmrP1k5S0RE35BdZ7FESQZvY/OzKgEiInqQJNvWKl9Xs9xHEjAiIv7LSLuz6y6F\n",
       "jIiIzo263CW9U9IPJS2VtNsyPztW0o8kLZb01s5jRkREOzo5c19IWe/lhuFPSpoCvBuYAswAzpLU\n",
       "d38hSBqqnaETyV9X8tfTy9nbMerStb3Y9vIW7zoQuNj2EtsPAQ8Ce472OF1sqHaADg3VDtChodoB\n",
       "OjRUO0CHhmoH6MBQ7QDjoYkz6s2BR4Y9foSymUZERIyTlV7nLmk+sOlyfvQJ21e2cZxsTh0RMY46\n",
       "vhRS0vXAUbbvaD3+OIDtU1qPvwkcb/uWZd6Xwo+IGIWRXAo5VneoDj/QFcBFkl7c23R74NbRhIuI\n",
       "iNHp5FLIgyQ9DOwFXCXpGgDbi4BLKVvhXQN8wLXulIqIGFDV7lCNiIjmVF4VUp+SdLekuyRdJ2mr\n",
       "mnnaJek0Sfe1/jd8TdIramdqx8puROtWkma0bo77kaSZtfO0S9I8SY9JWlg7S7skbSXp+tZ/M/dK\n",
       "+kjtTO2QtJakW1p9s0jSybUzjYakiZLulLTSi1pq31x0qu1dbU+jLPt7fOU87boWmGp7V+AB4NjK\n",
       "edq13BvRupWkicBnKTfHTQEOkTS5bqq2fYGSvxctAY6wPZUyjv1gL/37t/0M8KZW3+wCvEnSGyrH\n",
       "Go3DKWPvlY5dqpa77eHL/K4D/LpWltGwPd/2C62HtwBb1szTrpXciNat9gQetP2Q7SXAJZSb5nqG\n",
       "7RuB39TOMRq2/832Xa3v/wO4j3JfS8+w/XTr2zWAicC/V4zTNklbAvtT9r9Y6UUptc/ckXSSpJ8D\n",
       "hwKn1M7TgfcBV9cO0ee2AB4e9jg3yFUiaRtgOuWkpmdImiDpLuAx4PrWBSC95AzgGOCFVb2w8XKX\n",
       "NF/SwuV8/Q8A28fZ/gPgfErwrrKq/K3XHAc8Z/uiilGXayT5e0g+/e8CktYBvgoc3jqD7xm2X2iN\n",
       "ZbYE3thL68xIOgD4le07WcVZO4zDTky29xvhSy+iC898V5Vf0nsofybtOy6B2tTGv/9e8Atg+Ifu\n",
       "W/HSpS6iYZJWBy4D/tn2N2rnGS3bT0i6CngNsKBynJHaG/hjSfsDawHrSvqS7b9c3otrXy2z/bCH\n",
       "BwJ31soyGpJmUP5EOrD1YU0v64Wbym4Htpe0jaQ1KKuPXlE508CQJOA8YJHtM2vnaZekjSSt1/p+\n",
       "ErAfPdQ5tj9heyvb2wIHA99ZUbFD/Zn7ya0RwV2UldqOqpynXZ+hfBA8v3Vp0lm1A7VjRTeidSvb\n",
       "zwMfAr5FuVrgy7bvq5uqPZIuBm4CXi3pYUnvrZ2pDa8H/pxylcmdra9euvJnM+A7rb65BbjS9nWV\n",
       "M3VipWPK3MQUEdGHap+5R0REA1LuERF9KOUeEdGHUu4REX0o5R4R0YdS7hERfSjlHhHRh1LuERF9\n",
       "6P8Do+dj0Sq+0gEAAAAASUVORK5CYII=\n"
      ],
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f25ab83a5c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "##This function provides an interactive mode to change polynomial degree. \n",
    "@interact(n=[1,16])\n",
    "def plot(n):\n",
    "    X = construct_poly(x,n)\n",
    "    w = wRidge(X,y,1.0)\n",
    "    plt.plot(x,X.dot(w.T))\n",
    "    print(x)\n",
    "    print(X.dot(w.T).shape, y.shape)\n",
    "    print(MSE(X.dot(w.T),y))\n",
    "    plt.title(\"MSE %f\" % MSE(X.dot(w.T),y))\n",
    "    plt.plot(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8) Apply models with different values of $d$ after being trained on the training dataset, to the test data available [here](http://webdav.tuebingen.mpg.de/lectures/ei-SS2015/data/ex1_test.csv). Compare the errors on the test data to the ones from the training by plotting the error curves as functions of the polynomial degree in a single plot! What do you conclude? <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1.00000000e+02,   1.00000000e+01,   1.00000000e+00],\n",
       "       [  1.00000000e+01,   1.00000000e+00,   1.00000000e-01],\n",
       "       [  1.00000000e+00,   1.00000000e-01,   1.00000000e-02]])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A.T.dot(A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import scipy\n",
    "from scipy import linalg\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "[[  9.80101960e-03   9.80101960e-04   9.80101960e-05]\n",
      " [  9.80101960e-04   9.80101960e-05   9.80101960e-06]\n",
      " [  9.80101960e-05   9.80101960e-06   9.80101960e-07]]\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "A = np.array([[10,1,0.1],[10,1,0.2],[10,1.1,0.7]])\n",
    "a = np.array([[10,1,0.1]])\n",
    "A = a.T.dot(a)\n",
    "\n",
    "B,rank = linalg.pinv(A,return_rank=True)\n",
    "print rank\n",
    "print B\n",
    "print np.allclose(A,A.dot(B.dot(A)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Read test data here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "9) With a fixed optimal $d$, change the value of $\\lambda$ to one of the following values $[0.1, 1.0, 10.0]$ and find the minimum MSE!<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hand in printed copy of completed notebook."
   ]
  }
 ],
 "metadata": {
  "annotations": {
   "author": "",
   "categories": [
    "intelligent-systems-1-2015"
   ],
   "date": "2015-04-30",
   "location": "Beginning of next lecture",
   "parent": "IS_SS2015",
   "submission_date": "2015-05-07",
   "subtitle": "Exercise Sheet 3, Linear Regression",
   "tags": [
    "IntelligenSystems",
    "Course"
   ],
   "title": "Intelligent Systems 1 - Summer Semester 2015"
  },
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
