What a Transformer in AI is?
Transformers in AI are a type of artificial neural network architecture that is used to solve the problem of transformation of input sequences into output sequences in deep learning applications.
Transformers consist of an encoder-decoder architecture, where the encoder processes the input sequence and the decoder generates the output sequence. Each encoder and decoder layer in the transformer contains multiple self-attention mechanisms followed by feedforward neural networks.
The following is a very simple representation of a transformer.
The two components in the architecture: the encoder and decoder. The former is the element that receives each component of the input sequence, encoding (hence the name) it into a vector carrying the context information about the whole sequence. This is sent to the decoder whose job is to understand the context and resolve the output meaningfully.
Here is a simple translation of English sentence to German.
Behind the scenes, the job of the decoder is now to calculate the weight for each hidden state received by the encoder. This is done with the decoder's own hidden state, in other words, the output of the previous time step in the model. The aim is to provide greater importance to hidden states with higher weight values, and this is our new context vector. We can then use this context as input to the decoder’s feed forward neural net to obtain the output sequence we are after. In turn the output sequence is used as the hidden state of the decoder that is sued in the calculation of weights in the next time step.