Last Modified : Saturday, April 20, 2024
In simple terms, Context Window is how much information an AI can recall at a time when processing input from user. It's similar to us, for a certain topic, we can only remember few points about it, so does AI.
Context Window can be increased as well, allowing AI to deeply understand user inputs and generate more coherent responses. Context Window is largely used in Natural Language Processing (NLP) in various tasks from sentiment analysis to text generation.
How does context window works for sentiment analysis
In sentiment analysis, for a given sentence context window analyses surronding words to predict a sentiment. For each word, context window has some contextual information about it and based on a sentiment of a word is calculated.
How does context window works for Large Language Models (LLM) for text generation
For LLM's context window is how much information do they understand form user input, the more the context window, the better LLM's understand the user input.
How is context window calculated
Language models don't understand raw text, so we convert this text into tokens, which are nothing but mappings of a words or characters to numbers.
So the number of tokens model can process at once is a context window. For eg, GPT 3.5 has context window of 16k and it was increased to 128k in GPT 4
Does increasing context window increases model performence?
Till some level, as we have seen with GPT 3.5 and GPT 4, increasing context window does improves model performace, but after some levels, LLM models stars to struggle to extract the relevent information. Also with increasing context window, price for processing that much information also increases linearly.
How does RAG helps with context window's limitations?
First what is RAG (Retrieval Augmented Generation)
RAG is a database which contains any kind of information that you want to pass it to LLM's, so insted of passing all the information as user input to LLM's we store some info into database like large documents and LLM's then extract that information to generate response that is RAG.
So insted of passing all the information in context window we store it to database and tell LLM's to extract it from there, then we will have better response from LLM's. Using RAG also reduces the cost of processing context window.
Hope that this blog answered all your questions about context window.