A new alternative to the classic Multi-Layer Perceptron has emerged. Will it replace the traditional approach?

## Introduction

A recent __publication__ has introduced a groundbreaking approach in the deep learning field: Kolmogorov-Arnold Networks (KANs). This new machine learning algorithm promises to be a formidable alternative to the well-established Multi-Layer Perceptrons (MLPs). MLPs, or fully-connected feedforward neural networks, are foundational in modern deep learning due to their ability to approximate complex relationships between input and output data.

Kolmogorov-Arnold Networks (KANs) aim to address some of the limitations of MLPs. While MLPs are powerful, they come with significant drawbacks. They can be parameter-intensive, especially in models like transformers where they use up many parameters not directly involved in embedding data. Additionally, MLPs often function as black boxes, making their predictions harder to interpret compared to other model components, such as attention layers.

The key question remains: * Are KANs the ultimate solution for building models that can understand and predict complex patterns?* This article will delve into the fundamental concepts and ideas behind KANs, exploring their potential as a superior alternative to MLPs.

## Limitations of MLPs

Multi-Layer Perceptrons (MLPs) are fundamental to modern neural networks. These networks are composed of layers of interconnected nodes, or “neurons,” that approximate complex, non-linear functions by learning from data. Each neuron applies a fixed activation function to the weighted sum of its inputs, transforming input data into the desired output through multiple layers of abstraction. MLPs have been instrumental in achieving breakthroughs across various domains, including computer vision and speech recognition.

*MLP with input layer with 3 nodes, 2 hidden layers with 10 nodes each, and an output layer with 1 node. *__Credits__

Despite their successes, MLPs face several significant limitations:

1. **Fixed Activation Functions**: Each node in an MLP uses a predetermined activation function, such as ReLU or Sigmoid. While effective in many scenarios, these fixed functions can limit the network’s flexibility and adaptability, making it difficult to optimize certain types of functions or adapt to specific data characteristics.

2. **Interpretability Issues**: MLPs are often described as “black boxes.” As they grow in complexity, it becomes increasingly difficult to understand their decision-making processes. The fixed activation functions and complex weight matrices obscure the network’s inner workings, making it challenging to interpret and trust the model’s predictions without extensive analysis.

These limitations underscore the need for alternatives that provide greater flexibility and interpretability, leading to innovations like Kolmogorov-Arnold Networks (KANs).

## What Are Kolmogorov–Arnold Networks (KAN)?

Kolmogorov–Arnold Networks (KAN) are a type of neural network inspired by a mathematical theorem known as the **Kolmogorov–Arnold representation theorem**. This theorem provides a theoretical foundation for representing continuous multivariate functions as a superposition of continuous univariate functions and an additional function.

The theorem states that:

If f is a multivariate continuous function on a bounded domain, then f can be written as a finite composition of continuous functions of a single variable and the binary operation of addition. More specifically, for a smooth f : [0, 1]n → R,

To understand this, let us take an example of a multivariate equation like this,

This is a multivariate function because *y* depends on *x1, x2, ...xn*. . According to the theorem, we can express this as a combination of single-variable functions. This allows us to decompose the multivariable equation into several individual equations, each involving one variable and another function of it. By summing the outputs of these individual functions and then applying another univariate function, we can represent the original multivariate function as shown here.

Univariate Function

Passing the summed output to another function (Single composition)

Further, instead of making one composition, we do multiple compositions, such as *m* different compositions and sum them up.

Also, We can rewrite the above equation to,

## KAN Architecture: Unraveling the Design

The Kolmogorov-Arnold Network (KAN) features a unique architectural design that distinguishes it from traditional neural networks. In this section, we will explore the structure of KANs, detailing the intricacies of their design and explaining their operation.

Unlike conventional neural networks, which rely on fixed activation functions at individual nodes, KANs introduce a paradigm shift by incorporating learnable activation functions along the edges of the network graph. This fundamental departure from traditional architectures allows KANs to dynamically adapt their activation functions based on the input data, significantly enhancing their flexibility and expressive power.

The architecture of a KAN consists of interconnected layers, each comprising nodes and edges that transmit and transform information. At the input layer, raw data is fed into the network, where it undergoes a series of transformations as it propagates through successive layers. The activation functions embedded within the edges play a pivotal role in shaping these transformations, enabling the network to learn complex mappings between input and output data.

A key innovation of KANs is their use of B-splines as the basis for their learnable activation functions. B-splines are mathematical functions that provide a flexible and adaptive framework for modeling complex data patterns. By parameterizing these splines, KANs can capture intricate relationships within the data, allowing them to generalize more effectively to unseen examples.

Moreover, the architecture of KANs is highly scalable and extensible. While the basic framework of a KAN consists of a two-layer structure, it can be easily extended to accommodate deeper and more complex architectures. This scalability enables KANs to tackle a wide range of tasks, from simple regression problems to complex pattern recognition tasks, with ease and efficiency.

## B-splines

B-splines, or Basis splines, are a type of mathematical function used to create smooth and flexible curves. Imagine you have a bunch of points that you want to connect with a smooth line. Instead of drawing a straight line between each point, which would look jagged, you use B-splines to create a smooth, flowing curve that passes near these points.

* A cubic B-spline curve. *__Credits__

**Think of it like this:**

1. **Flexible Rubber Band**: Picture a flexible rubber band that you stretch and shape to fit smoothly around a series of pins stuck in a board. The rubber band represents the B-spline, and the pins represent the data points you want the curve to pass near.

2. **Segments**: A B-spline curve is made up of several smaller curve segments. These segments are joined together smoothly, so there are no sharp corners or abrupt changes in direction. This smoothness is a key characteristic of B-splines.

3. **Control Points**: The shape of the B-spline curve is controlled by a set of points called control points. The curve doesn’t necessarily pass through these control points; instead, they act like magnets, pulling the curve in certain directions to create the desired shape.

4. **Adjustable Smoothness**: You can adjust how smooth or flexible the curve is by changing the number of control points and how they influence the curve. More control points give you more flexibility to shape the curve, while fewer points make the curve simpler and smoother.

**Why Use B-Splines?**

• **Smoothness**: They create smooth curves that are useful in graphics, animation, and design.

• **Flexibility**: They can model complex shapes and patterns more easily than straight lines or simple curves.

• **Adaptability**: In data modeling, B-splines can adapt to fit complex data patterns, making them useful for tasks like curve fitting and function approximation.

B-splines are like flexible rubber bands used to draw smooth curves through or near a series of points. They provide a way to create smooth, adaptable curves that are useful in many fields, from computer graphics to data modeling. By adjusting the control points, you can shape the curve to fit your needs precisely.

## Backpropagation in KAN

At the core of training Kolmogorov-Arnold Networks (KANs) is backpropagation, a key technique in machine learning that helps neural networks adjust their parameters based on errors observed during training. In KANs, backpropagation is essential for fine-tuning both the weights of the network and the coefficients of the learnable activation functions.

**How Backpropagation Works in KAN:**

1. **Initialization**:

• Training starts by randomly setting the initial values of the network’s weights and the coefficients for the activation functions.

2. **Forward Pass**:

• Input data is fed through the network layer by layer, producing predictions at the output.

3. **Loss Calculation**:

• The predictions are compared to the actual data (ground truth) to calculate the loss, which measures how far off the predictions are from the real values.

4. **Backward Pass (Backpropagation)**:

• Using the chain rule from calculus, the gradients (slopes) of the loss with respect to each parameter (weights and coefficients) are calculated.

• These gradients indicate how to change each parameter to reduce the loss.

5. **Parameter Update**:

• The network’s parameters are adjusted using these gradients. This is typically done with optimization techniques like gradient descent, stochastic gradient descent, or Adam optimizer.

**Challenges and Solutions:**

**Challenges**:

• **Stability and Convergence**: Ensuring the training process is stable and the network learns effectively can be tough because of the complex interactions between weights and activation functions, leading to non-linear and non-convex optimization landscapes.

**Solutions**:

• **Regularization**: Techniques like dropout (randomly turning off some neurons during training) and weight decay (penalizing large weights) help prevent overfitting and improve generalization.

• **Normalization**: Batch normalization and layer normalization help stabilize and speed up training by normalizing the inputs of each layer.

• **Optimization**: Carefully choosing the right optimization algorithms and learning rates can significantly impact the efficiency and success of training.

## Difference between MLPs and KANs

Below are few differences between MLPs and KAN:

**Activation Functions**:

• **MLP**: Uses fixed activation functions like ReLU or Sigmoid for each node. These functions do not change during training.

• **KAN**: Uses learnable activation functions. These functions can adapt and change during training, allowing for more dynamic and tailored transformations.

**Weights**:

• **MLP**: Employs linear weights. The output of each node is a weighted sum of the inputs, followed by the activation function.

• **KAN**: Uses parameterized splines instead of linear weights. Splines are flexible curves that can be adjusted to fit complex patterns in the data.

**Interpretability**:

• **MLP**: Often seen as a “black box” due to its complexity, making it hard to understand how it makes decisions.

• **KAN**: More interpretable because the learnable activation functions and splines provide a clearer view of how the network processes data.

**Flexibility and Adaptability**:

• **MLP**: Less flexible and adaptable compared to KANs, as the fixed activation functions limit its ability to adjust to different data patterns.

• **KAN**: Highly flexible and adaptable, capable of better fitting complex and varying data through its learnable activation functions and parameterized splines.

**Training Time**:

• **MLP**: Generally has faster training times due to simpler, fixed activation functions and linear weights.

• **KAN**: Tends to have slower training times because of the complexity involved in learning the activation functions and adjusting the splines.

**Theoretical Basis**:

• **MLP**: Based on the Universal Approximation Theorem, which states that neural networks can approximate any continuous function with enough nodes and layers.

• **KAN**: Based on the Kolmogorov-Arnold Representation Theorem, which provides a mathematical framework for representing multivariate functions as sums of univariate functions, leading to the design of KANs.

MLPs are simpler and faster to train but less flexible and harder to interpret. KANs offer greater flexibility and interpretability at the cost of increased training complexity and time.

## Protein Sequence Classification using KAN

Here we trains a Kolmogorov-Arnold Network (KAN) on a synthetic protein sequence classification task. It involves one-hot encoding protein sequences, generating a dataset, defining and training the KAN model, and evaluating its performance. The KAN model uses symbolic regression to learn interpretable models of the input data.

### 1.Install and import libraries, define the protein window size.

### 2. One-Hot Encoding Function

• **Amino Acid List**: Defines the list of possible amino acids.

• **one_hot_encode Function**: Converts a protein sequence into a one-hot encoded numpy array. Each amino acid in the sequence is represented as a vector with one position set to 1 and all others set to 0.

### 3. Dataset Generation Function

• **generate_sample_protein_dataset Function**: Generates a synthetic dataset of protein sequences.

• **Labels**: Half the sequences have label 1 with ‘K’ (Lysine) in the center, and the other half have label 0 with ‘S’ (Serine) in the center.

• **One-Hot Encoding**: Each sequence is one-hot encoded and then flattened.

• **Splitting the Data**: The data is split into training and testing sets (50% each).

• **Conversion to Tensors**: Converts the lists to PyTorch tensors.

### 4. Generate Dataset

### 5. Define and Train the KAN Model

• **Model Definition**: Initializes a KAN model with an input size of 105 (21 amino acids * 5 positions), 3 hidden neurons, and 2 output neurons (assuming a binary classification task).

• **Accuracy Functions**: Defines functions to calculate training and testing accuracy.

• **Training the Model**: Trains the model using the LBFGS optimizer for 5 steps, evaluating the training and testing accuracy.

### 6. Auto Symbolic

• **Auto Symbolic**: Automatically generates symbolic representations of the learned functions using a library of functions (e.g., ‘x’, ‘x^2’).

• **Extract Formulas**: Extracts the symbolic formulas learned by the model.

### 7. Plotting

Plot the structure of the trained KAN model.

You can find the entire code __here__.

## Challenges and Limitations of KANs

While Kolmogorov-Arnold Networks (KANs) provide several benefits over traditional Multi-Layer Perceptrons (MLPs), they also encounter various challenges and limitations.

1. **Complexity of Learning**: Training KANs can be difficult, especially with large datasets or complex optimization landscapes. Learning adaptive activation functions and optimizing spline parameters require significant computational resources and may need specialized training techniques.

2. **Interpretability Trade-offs**: Although KANs are more interpretable than MLPs, they still present trade-offs in terms of model complexity and clarity. Learnable activation functions along edges can obscure some interpretability, particularly in deeper, multi-layer architectures.

3. **Generalization to High-Dimensional Data**: KANs perform well on many tasks but may struggle with high-dimensional data that has intricate relationships between variables. The use of univariate functions to represent multivariate functions can limit the model’s ability to capture complex feature interactions.

4. **Sensitivity to Hyperparameters**: Like other neural network architectures, KANs are sensitive to hyperparameters such as learning rate, regularization strength, and network structure. Proper selection and tuning of these hyperparameters are crucial for the performance and convergence of KANs, requiring careful experimentation.

5. **Computational Overhead**: KANs can have significant computational overhead, particularly during training and inference. The adaptive activation functions and spline parameters may require more computational resources than traditional MLPs, leading to longer training times and higher computational costs.

6. **Model Complexity and Scalability**: Although KANs offer architectural flexibility, deeper architectures with multiple layers and complex activation functions can increase model complexity and computational overhead. Scaling KANs to handle large datasets and complex tasks efficiently while maintaining interpretability is a significant challenge.

## Conclusion

Kolmogorov-Arnold Networks (KANs) present a groundbreaking alternative to traditional Multi-Layer Perceptrons (MLPs), offering several key innovations that address the limitations of their predecessors. By leveraging learnable activation functions on the edges rather than fixed functions at the nodes, KANs introduce a new level of flexibility and adaptability. However, significant concerns need to be addressed before KANs can potentially replace MLPs in machine learning.

The primary issue is that KANs can’t utilize GPU parallel processing, preventing them from taking advantage of the fast-batched matrix multiplications that GPUs offer. This limitation results in very slow training times since KANs’ unique activation functions can’t efficiently leverage batch computation or process multiple data points in parallel. Hence, if speed is crucial, MLPs are a better choice.

However, if you prioritize interpretability and accuracy and can tolerate slower training, KANs are a promising option. They provide enhanced flexibility and adaptability, potentially leading to better performance on complex tasks where these attributes are critical.

## Opmerkingen