Generation Phase

The prompt for the LLM is now ready, so what remains is to send it over and receive a response. To connect to LLMs, the application uses Langchain's streaming support, which nicely fits with the event streaming used in this application:

answer = ''
for chunk in get_llm().stream(qa_prompt):
    yield f'data: {chunk.content}\n\n'
    answer += chunk.content

The get_llm() function is defined in api/llm_integrations.py. Its purpose is to return the correct LLM integration from Langchain according to the configuration. Assuming you configured OpenAI, the LLM returned is going to be an instance of the ChatOpenAI class from Langchain.