AI Term of the Moment
AI worm
Redirected from: neural network model
Definition: neural network
The architecture of today's small and large language models, which provides the source of learned knowledge that chatbots such as ChatGPT, Gemini and Claude use to generate answers.
Servers and GPUs
Data centers can have thousands of servers. The more servers, the larger the neural network, the larger the language model and the more comprehensive the training. Desktop and mobile devices run the chatbots that communicate with the models in the data center, and desktop and laptop computers may be also used to train small models; for example, see DGX Spark.
Layers and Nodes
A Single Neuron/Node
Tracing a Neuron
The neural network is a pattern detection system. The text in sentences (word, part of a word, etc.) is turned into a token that is converted into an "embedding," which is a one-dimensional array of numbers known as a "vector." Each layer transforms the embeddings mathematically by looking at the tokens in the sentence and making the embeddings more meaningful. The last layer predicts the next token.
A Tiny Network
A Larger Network
There Are Many Network Designs
The following diagrams from the Asimov Institute in the Netherlands reveal the variety of neural network architectures. Only a few layers are shown in each example; however, there can be hundreds or thousands of layers depending on the network type. For the type of neural network that is easier to grasp, see convolutional neural network.
Neural Network Architectures
Servers and GPUs
Data centers can have thousands of servers. The more servers, the larger the neural network, the larger the language model and the more comprehensive the training. Desktop and mobile devices run the chatbots that communicate with the models in the data center, and desktop and laptop computers may be also used to train small models; for example, see DGX Spark.
A neural network is a "math machine" that learns from examples. It comprises multiple layers of computational units called "nodes" or "neurons" that are mathematically connected to each other (see diagrams below). There is an input layer, an output layer and any number of "hidden" layers in between.
A neuron, or "node," is a mathematical processing unit within the neural network, and a single GPU processes many neurons simultaneously. The activation function determines if the value leaving the neuron should be altered. The weights and bias values are modified in the training passes when predictions are compared against the correct answers. If this all sounds very strange... it is! See AI weights and biases.
Tracing a Neuron
The neural network is a pattern detection system. The text in sentences (word, part of a word, etc.) is turned into a token that is converted into an "embedding," which is a one-dimensional array of numbers known as a "vector." Each layer transforms the embeddings mathematically by looking at the tokens in the sentence and making the embeddings more meaningful. The last layer predicts the next token.
This neural network example has only 17 neurons/nodes in three hidden layers and might be used for limited pattern recognition in a small sensor.
This neural network example has 8,192 neurons/nodes in eight hidden layers for a total of 65 thousand nodes and 4.3 billion connections that are adjusted in the training stages. Now imagine a neural network with 500 billion nodes and you can begin to visualize the complexity of large AI models.
There Are Many Network Designs
The following diagrams from the Asimov Institute in the Netherlands reveal the variety of neural network architectures. Only a few layers are shown in each example; however, there can be hundreds or thousands of layers depending on the network type. For the type of neural network that is easier to grasp, see convolutional neural network.
Network designs are one of the most researched areas of AI in the 21st century. The examples above are only a small sample. (Images courtesy of Fjodor van Veen and Stefan Leijnen (2019). The Neural Network Zoo.)