Skip to main content

Where do variance and bias come from ?

What are variance and bias ?

Variance and bias are assertions to evaluate our model based on different datasets. Both of them happen together and we have to find the methodology to trade-off them. A good model is a small variance and bias model. 

 In wikipedia, we have definitions of variance and bias, formally:
The bias is an error from erroneous assumptions in the learning algorithm.
The variance is an error from sensitivity to small fluctuations in the training set. 
(source: https://en.wikipedia.org/wiki/Bias–variance_tradeoff)

However, I want to make them more normal and practical, so we could understand them as bellowing definitions:
Variance asserts fluctuation of precision of model against training data, test data with real data(new data). One model has a hight precision in training and test phase but has low precision with real data, It means this model has hight variance. Hight variance also is called as overfitting: Too good for fitting model but bad for fitting real data.  


Bias asserts fluctuation of precision of model against training data and evaluation data. One model has high precision in training data but has low precision in evaluation data, it means it has hight bias. 
Hight bias also is called under-fitting: Too bad for fitting model in training phase. 




Where do they come from ?
Now we will explore where variance and bias come from.  For notational convenience, abbreviate h = h(x). First, note that for any random variable X, we have:
                                                         $Var(X) = E[(X - \mu)^2]  = E[X^2] - E[X]^2$
Rearrange, we get: $E[X^2] = E[(X-\mu)^2] + E[X]^2$
Since the h function is deterministic: E[h] = h
Assume, we have true function estimation that give us:
$y = h(x) + \epsilon$ with $epsilon \sim (0, \sigma^2)$
Implies: $E[y] = E[h + \epsilon] = E[h] = h$
Estimate for y: $ \hat{y}$

We use Expectation of mean squared error to calculate the error of estimation:
$E[(y - \hat{y})^2] = E[y^2] - 2E[y*\hat{y}] + E[\hat{y}^2]$
$=Var(y) + E[y]^2 - 2E[y * \hat{y}] + Var(\hat{y}) + E[\hat{y}]^2 $
$= Var(y) + Var(\hat{y}) + h^2 - 2hE[\hat{y}] + E[\hat{y}]^2$
$ = Var(y) + Var(\hat{y}) + E[(h-\hat{y})^2]$
$ = \sigma^2 + Var(\hat{y}) + bias^2$

As you can see,  $ Var(\hat{y})$ and $bias^2$ related to $ \hat{y} $, the estimation of $y$.
$ Var(\hat{y})$ is variance which we want to find out.
$bias^2$  is square of bias which we want to find out.

How to trade off 
As we know, variance and bias occur together. Additionally, they occur in opposite way. It means, when variance hight, bias is low and otherwise. We have to balance both of them.  The common way is finding the intersection point of variance and bias based on complexity of model. 
Another way is using regularizations. The common regularized techniques are Lasso and Ridge that we can explore in a new topic.

we also can use some tips to trade-off variance and bias:
-       Collect more training data      (fix variance)
-       Add more features                  (fix variance)
-       Remove redundant features    (fix bias)
-       Add polynomial features        (fix bias)
-       Increase lambda                      (fix variance)
-       Decrease lambda                     (fix bias)


Labels:

Comments

Post a Comment

Popular posts from this blog

NLG pipeline

NLG ¶ NLG stands for Natural Languate Generation. NLG is one field of AI aims to generate the understandable and appropriate texts from raw data. We should differientiate the concepts of NLG with NLP and NLU. NLP is natural languate processing. This is a field in AI working on text generally. NLP contains Speed recornigion, Speed synthesis, NLG and NLU. NLU and NLG is subsets of NLP. While NLG generate the text, NLU uses text as input and generate some pattern such as Sentiment Analysis, Summary. The pipeline of NLG NLG can be divided into 3 phases: Document planning, Microplanning and Realisation. The purpose of Document planning is to chose what to say and the purpos of Microplanning and Realisation is to find how to say. There are some components in each phase. In traditional NLG system, we have 5 components: Content Determination, Text Structure, Aggregation, Lexicalisation, Reffering expression, Realisation. Content Determination Content Determination is sets of enti...
Post a Comment
Read more

Generative Adversarial Networks

Generative Adversarial Networks (GAN) is one a Neural Network architecture which simulates zero-sum game. There are 2 parts of this Neural Network. The first is called Generator and other is called Discriminator. Generator tries to mimic data and make the fake data likes the real data in distribution. Meanwhile, the Discriminator tries to maximize the difference between real data and fake data. It is reason we call zero-sum game. Two parts are coaction with each other. This structure makes the GAN to be a interesting Neural Network architecture and it has many application in both academic and industry. In modeling, the GAN is an approach of equilibrium Networks such as Boltzmann Machine did. It is an optimization problem with objectives of: minimize Generator and maximum Discriminator simultaneously.  $max_{D}min_{G} V(D,G)$  $max_{D}min_{G} V(D,G) = E_{x\sim p_{data}(x)}[log(D(x))] + E_{z\sim p_{z}(z)}[log(1 - D(G(z)))]$ $V(D,G)$ is optimization problem subject to G ...
Post a Comment
Read more

The basic concepts of deep neural networks

We will explore 3 basic concepts of deep neural networks. The first part will explain how back-propagation algorithm works. The second talks about softmax function. And the last one is cost function. Inside Backpropagation algorithm The main idea of Backpropagation algorithm is aims to minimize the overall neural network error. To do that, we find out the partial derivative of error and weight vector. Take a look in a simple Neural Network: From Sigmoid function to Softmax function and Cross-entropy cost function In the classification problem with multinomial distribution, how do we know which label is choosen? Softmax function is used to do that. In practice, some people might also use one2All mechanism to detect label by traning many logistic regression models. However, it is not effective solution. Softmax function helps us to determine the label also we can calculate the cost from this easily by using Cross-entropy cost function. Sigmoid function Sigmoid function co...
Post a Comment
Read more