Auto Regression

What does auto-regression mean?

Auto regression refers to predicting the next token in time-sequence models. Given many previous tokens, predict the next one. Given “How are you”, predict “today”. Now you may be wondering, why is it called auto-regression?

Regression is the process of predicting a value y’ for an unseen input x’, given many pairs of (x, y). Auto-regressive model’s name comes from regression, do it does something similar. It takes in a sequence and predicts one output. Naturally, it’s the next token that’s going to be appended to the sequence!