In the context of Support Vector Machines (SVM), a kernel function is a function that maps the original input space into a higher-dimensional space, where it becomes easier to separate data into different classes. The whole point of using kernel methods is to deal with data that is not linearly separable: by transforming data into a higher dimensional space, it is possible to find a separating hyperplane that is linear in this new space.
To put it simply, a kernel function allows the SVM algorithm to operate in a higher-dimensional feature space without explicitly computing the coordinates of the data in that space, which would be computationally expensive or even impossible.
Formally, the SVM algorithm seeks to maximize the margin, i.e., the distance between the decision boundary and the closest points that belong to each class. A kernel function allows the SVM to define a decision boundary in the higher-dimensional space while still being defined in the original space.
The choice of kernel function is very important in SVM, as it will affect the shape of the decision boundary and thus the accuracy of classification. Here are some examples of kernel functions that are commonly used:
- Linear kernel: the simplest kernel, it simply computes the dot product between two vectors in the original feature space. It is useful when data is linearly separable.
- Polynomial kernel: this kernel function computes the dot product of vectors in a higher dimensional space, where each feature corresponds to a monomial term of a certain degree. For example, a 2nd-degree polynomial kernel in two dimensions would map the input (x, y) into a feature space containing (1, x, y, x^2, xy, y^2). The degree of the kernel determines how complex the decision boundary can be.
- Radial Basis Function (RBF) kernel: this kernel function computes the dot product between two Gaussian functions centered on each input vector. Its parameter is the bandwidth or Gaussian width, which governs how far the influence of a single training example reaches. An RBF kernel can handle any shape of decision boundaries, but it is susceptible to overfitting if the bandwidth is set too high.
- Sigmoid kernel: this kernel function computes the hyperbolic tangent of the dot product between two vectors, multiplied by a constant and plus an offset. It is similar to a neural network activation function and can model complex non-linear boundaries, but it is less commonly used than the other kernels.In summary, the kernel function in SVM allows for a flexible decision boundary that can separate data points that are not linearly separable. Different kernels can be used to control the complexity and shape of the decision boundary.