In machine learning, a model is a mathematical function that maps input variables to an output variable. The process of learning a model from data involves selecting the right kind of model, which can be parametric or non-parametric. The main difference between parametric and non-parametric models is the assumptions they make about the underlying distribution of the data.
**Parametric models:**
Parametric models assume that the underlying data follows a specific distribution function. The model has a fixed number of parameters that are estimated based on the data to fit this distribution. Once the parameters are estimated, the model can be used to make predictions on new data.
One of the key advantages of parametric models is their simplicity and computational efficiency. They are often easier to train and require less data than non-parametric models. Examples of parametric models include linear regression, logistic regression, and Gaussian Naive Bayes.
For example, in linear regression, the model assumes that the relationship between the input variables X and the output variable Y is linear, and the parameters of the model are the slope and intercept of the line. The model is trained by minimizing the sum of squared errors between the predicted values and the actual values of the output variable.
The linear regression model can be represented as:
y = β0 + β1x1 + β2x2 + ... + βnxn + ϵ
where y is the predicted output variable, x1, x2, ..., xn are the input variables, β0, β1, β2, ..., βn are the parameters of the model, and ϵ is the error term.
**Non-parametric models:**
Non-parametric models do not make assumptions about the underlying distribution of the data. Instead, they estimate the distribution from the data directly. Non-parametric models can be more flexible and powerful than parametric models, but they require more data and are often computationally expensive. Examples of non-parametric models include k-nearest neighbors, decision trees, and random forests.
For example, in k-nearest neighbors, the model predicts the output variable by finding the k training examples that are closest to the new input and using their output values to estimate the output variable. The value of k is chosen based on a validation set or through cross-validation.
The k-nearest neighbor model can be represented as:
$y = \frac{1}{k}\sum_{i=1}^{k} y_i$
where yi is the output value of the i-th training example that is closest to the new input, and k is the number of neighbors to consider.
In summary, parametric models assume a specific distribution function and have a fixed number of parameters that are estimated from the data, while non-parametric models do not assume a specific distribution and estimate the distribution from the data directly. The choice between these two types of models depends on the problem at hand, the amount of data available, and the desired level of complexity in the model.