How to add summary of the conversation history#
One of the most common use cases for persistence is to use it to keep track of conversation history. This is great - it makes it easy to continue conversations. As conversations get longer and longer, however, this conversation history can build up and take up more and more of the context window. This can often be undesirable as it leads to more expensive and longer calls to the LLM, and potentially ones that error. One way to work around that is to create a summary of the conversation to date, and use that with the past N messages. This guide will go through an example of how to do that.
This will involve a few steps:
Check if the conversation is too long (can be done by checking number of messages or length of messages)
If yes, the create summary (will need a prompt for this)
Then remove all except the last N messages
A big part of this is deleting old messages. For an in depth guide on how to do that, see this guide
Setup#
First, let’s set up the packages we’re going to want to use
%%capture --no-stderr
%pip install --quiet -U langgraph langchain_anthropic
Next, we need to set API keys for Anthropic (the LLM we will use)
import getpass
import os
def _set_env(var: str):
if not os.environ.get(var):
os.environ[var] = getpass.getpass(f"{var}: ")
_set_env("ANTHROPIC_API_KEY")
Build the chatbot#
Let’s now build the chatbot.
from typing import Literal
from langchain_anthropic import ChatAnthropic
from langchain_core.messages import SystemMessage, RemoveMessage
from langgraph.checkpoint.redis import RedisSaver
from langgraph.graph import MessagesState, StateGraph, START, END
# Set up Redis connection for checkpointer
REDIS_URI = "redis://redis:6379"
memory = None
with RedisSaver.from_conn_string(REDIS_URI) as cp:
cp.setup()
memory = cp
# We will add a `summary` attribute (in addition to `messages` key,
# which MessagesState already has)
class State(MessagesState):
summary: str
# We will use this model for both the conversation and the summarization
model = ChatAnthropic(model_name="claude-3-haiku-20240307")
# Define the logic to call the model
def call_model(state: State):
# If a summary exists, we add this in as a system message
summary = state.get("summary", "")
if summary:
system_message = f"Summary of conversation earlier: {summary}"
messages = [SystemMessage(content=system_message)] + state["messages"]
else:
messages = state["messages"]
response = model.invoke(messages)
# We return a list, because this will get added to the existing list
return {"messages": [response]}
# We now define the logic for determining whether to end or summarize the conversation
def should_continue(state: State) -> Literal["summarize_conversation", END]:
"""Return the next node to execute."""
messages = state["messages"]
# If there are more than six messages, then we summarize the conversation
if len(messages) > 6:
return "summarize_conversation"
# Otherwise we can just end
return END
def summarize_conversation(state: State):
# First, we summarize the conversation
summary = state.get("summary", "")
if summary:
# If a summary already exists, we use a different system prompt
# to summarize it than if one didn't
summary_message = (
f"This is summary of the conversation to date: {summary}\n\n"
"Extend the summary by taking into account the new messages above:"
)
else:
summary_message = "Create a summary of the conversation above:"
messages = state["messages"] + [HumanMessage(content=summary_message)]
response = model.invoke(messages)
# We now need to delete messages that we no longer want to show up
# I will delete all but the last two messages, but you can change this
delete_messages = [RemoveMessage(id=m.id) for m in state["messages"][:-2]]
return {"summary": response.content, "messages": delete_messages}
# Define a new graph
workflow = StateGraph(State)
# Define the conversation node and the summarize node
workflow.add_node("conversation", call_model)
workflow.add_node(summarize_conversation)
# Set the entrypoint as conversation
workflow.add_edge(START, "conversation")
# We now add a conditional edge
workflow.add_conditional_edges(
# First, we define the start node. We use `conversation`.
# This means these are the edges taken after the `conversation` node is called.
"conversation",
# Next, we pass in the function that will determine which node is called next.
should_continue,
)
# We now add a normal edge from `summarize_conversation` to END.
# This means that after `summarize_conversation` is called, we end.
workflow.add_edge("summarize_conversation", END)
# Finally, we compile it!
app = workflow.compile(checkpointer=memory)
00:42:58 langgraph.checkpoint.redis INFO Redis client is a standalone client
00:42:58 redisvl.index.index INFO Index already exists, not overwriting.
00:42:58 redisvl.index.index INFO Index already exists, not overwriting.
00:42:58 redisvl.index.index INFO Index already exists, not overwriting.
Using the graph#
def print_update(update):
for k, v in update.items():
for m in v["messages"]:
m.pretty_print()
if "summary" in v:
print(v["summary"])
from langchain_core.messages import HumanMessage
config = {"configurable": {"thread_id": "4"}}
input_message = HumanMessage(content="hi! I'm bob")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
input_message = HumanMessage(content="what's my name?")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
input_message = HumanMessage(content="i like the celtics!")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
================================ Human Message =================================
hi! I'm bob
00:42:59 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
Hi Bob! Nice to meet you. I'm Claude, an AI assistant created by Anthropic. How are you doing today? Let me know if you have any questions or if there's anything I can help you with.
================================ Human Message =================================
what's my name?
00:42:59 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
Your name is Bob, as you told me earlier in our conversation.
================================ Human Message =================================
i like the celtics!
00:43:00 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
That's great, I'm glad you like the Celtics! Basketball is a really exciting sport to follow. Do you have a favorite player on the Celtics team? I'd be curious to hear more about what you enjoy about watching them play.
00:43:02 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
Here is a summary of our conversation so far:
The conversation started with you asking about the weather in San Francisco, to which I incorrectly responded that it was sunny in Los Angeles.
You then introduced yourself as Bob, and I greeted you and explained that I am an AI assistant named Claude.
You asked me what your name was, and I correctly recalled that you had introduced yourself as Bob earlier.
Finally, you expressed that you like the Boston Celtics basketball team, and I asked if you had a favorite player on the team, showing interest in learning more about your fandom.
The key points are your name is Bob, you're a fan of the Celtics, and we had a bit of back-and-forth as I introduced myself as an AI assistant named Claude.
We can see that so far no summarization has happened - this is because there are only six messages in the list.
values = app.get_state(config).values
values
{'messages': [HumanMessage(content='i like the celtics!', additional_kwargs={}, response_metadata={}, id='95a53b16-7630-466e-be0a-896bbd58bc06'),
AIMessage(content="That's great, I'm glad you like the Celtics! Basketball is a really exciting sport to follow. Do you have a favorite player on the Celtics team? I'd be curious to hear more about what you enjoy about watching them play.", additional_kwargs={}, response_metadata={'id': 'msg_0183Pgv8iMvNtpwmT8PbbgMT', 'model': 'claude-3-haiku-20240307', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'cache_creation': {'ephemeral_1h_input_tokens': 0, 'ephemeral_5m_input_tokens': 0}, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 113, 'output_tokens': 55, 'server_tool_use': None, 'service_tier': 'standard'}, 'model_name': 'claude-3-haiku-20240307', 'model_provider': 'anthropic'}, id='lc_run--2e844ea2-d3c2-4f6b-87f7-0e17517c7a62-0', usage_metadata={'input_tokens': 113, 'output_tokens': 55, 'total_tokens': 168, 'input_token_details': {'cache_creation': 0, 'cache_read': 0, 'ephemeral_5m_input_tokens': 0, 'ephemeral_1h_input_tokens': 0}})],
'summary': "Here is a summary of our conversation so far:\n\nThe conversation started with you asking about the weather in San Francisco, to which I incorrectly responded that it was sunny in Los Angeles. \n\nYou then introduced yourself as Bob, and I greeted you and explained that I am an AI assistant named Claude.\n\nYou asked me what your name was, and I correctly recalled that you had introduced yourself as Bob earlier.\n\nFinally, you expressed that you like the Boston Celtics basketball team, and I asked if you had a favorite player on the team, showing interest in learning more about your fandom.\n\nThe key points are your name is Bob, you're a fan of the Celtics, and we had a bit of back-and-forth as I introduced myself as an AI assistant named Claude."}
Now let’s send another message in
input_message = HumanMessage(content="i like how much they win")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
================================ Human Message =================================
i like how much they win
00:43:04 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
That's a good point about the Celtics - they have been a very successful franchise over the years. A few thoughts on their winning ways:
- The Celtics are one of the most storied franchises in the NBA, with 17 championship titles, the most of any team. Their history of winning is impressive.
- This season, the Celtics have the best record in the Eastern Conference and are contending for another title. Their consistent ability to compete for championships year after year is admirable.
- Players like Jayson Tatum and Jaylen Brown have emerged as elite talents, helping keep the Celtics among the NBA's elite teams. Their young core has a lot of potential.
- Coaching stability with Brad Stevens and now Joe Mazzulla has allowed the Celtics to maintain their high level of play over time.
I can see why you enjoy following a team that wins as much as the Celtics do. Their sustained success and ability to compete for championships must make them very fun to root for. Do you have a favorite Celtics championship team or season from their history?
If we check the state now, we can see that we have a summary of the conversation, as well as the last two messages
values = app.get_state(config).values
values
{'messages': [HumanMessage(content='i like the celtics!', additional_kwargs={}, response_metadata={}, id='95a53b16-7630-466e-be0a-896bbd58bc06'),
AIMessage(content="That's great, I'm glad you like the Celtics! Basketball is a really exciting sport to follow. Do you have a favorite player on the Celtics team? I'd be curious to hear more about what you enjoy about watching them play.", additional_kwargs={}, response_metadata={'id': 'msg_0183Pgv8iMvNtpwmT8PbbgMT', 'model': 'claude-3-haiku-20240307', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'cache_creation': {'ephemeral_1h_input_tokens': 0, 'ephemeral_5m_input_tokens': 0}, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 113, 'output_tokens': 55, 'server_tool_use': None, 'service_tier': 'standard'}, 'model_name': 'claude-3-haiku-20240307', 'model_provider': 'anthropic'}, id='lc_run--2e844ea2-d3c2-4f6b-87f7-0e17517c7a62-0', usage_metadata={'input_tokens': 113, 'output_tokens': 55, 'total_tokens': 168, 'input_token_details': {'cache_creation': 0, 'cache_read': 0, 'ephemeral_5m_input_tokens': 0, 'ephemeral_1h_input_tokens': 0}}),
HumanMessage(content='i like how much they win', additional_kwargs={}, response_metadata={}, id='3d4a2b4f-a111-42de-a382-d59e90389a2f'),
AIMessage(content="That's a good point about the Celtics - they have been a very successful franchise over the years. A few thoughts on their winning ways:\n\n- The Celtics are one of the most storied franchises in the NBA, with 17 championship titles, the most of any team. Their history of winning is impressive.\n\n- This season, the Celtics have the best record in the Eastern Conference and are contending for another title. Their consistent ability to compete for championships year after year is admirable.\n\n- Players like Jayson Tatum and Jaylen Brown have emerged as elite talents, helping keep the Celtics among the NBA's elite teams. Their young core has a lot of potential.\n\n- Coaching stability with Brad Stevens and now Joe Mazzulla has allowed the Celtics to maintain their high level of play over time.\n\nI can see why you enjoy following a team that wins as much as the Celtics do. Their sustained success and ability to compete for championships must make them very fun to root for. Do you have a favorite Celtics championship team or season from their history?", additional_kwargs={}, response_metadata={'id': 'msg_01SQizVtWWsq9zAnBW6Jg4Ub', 'model': 'claude-3-haiku-20240307', 'stop_reason': 'end_turn', 'stop_sequence': None, 'usage': {'cache_creation': {'ephemeral_1h_input_tokens': 0, 'ephemeral_5m_input_tokens': 0}, 'cache_creation_input_tokens': 0, 'cache_read_input_tokens': 0, 'input_tokens': 249, 'output_tokens': 240, 'server_tool_use': None, 'service_tier': 'standard'}, 'model_name': 'claude-3-haiku-20240307', 'model_provider': 'anthropic'}, id='lc_run--a87a6f62-2eba-4d8b-9165-96a4f76ccd0e-0', usage_metadata={'input_tokens': 249, 'output_tokens': 240, 'total_tokens': 489, 'input_token_details': {'cache_creation': 0, 'cache_read': 0, 'ephemeral_5m_input_tokens': 0, 'ephemeral_1h_input_tokens': 0}})],
'summary': "Here is a summary of our conversation so far:\n\nThe conversation started with you asking about the weather in San Francisco, to which I incorrectly responded that it was sunny in Los Angeles. \n\nYou then introduced yourself as Bob, and I greeted you and explained that I am an AI assistant named Claude.\n\nYou asked me what your name was, and I correctly recalled that you had introduced yourself as Bob earlier.\n\nFinally, you expressed that you like the Boston Celtics basketball team, and I asked if you had a favorite player on the team, showing interest in learning more about your fandom.\n\nThe key points are your name is Bob, you're a fan of the Celtics, and we had a bit of back-and-forth as I introduced myself as an AI assistant named Claude."}
We can now resume having a conversation! Note that even though we only have the last two messages, we can still ask it questions about things mentioned earlier in the conversation (because we summarized those)
input_message = HumanMessage(content="what's my name?")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
================================ Human Message =================================
what's my name?
00:43:04 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
I apologize, I don't actually remember your name. Earlier in our conversation you introduced yourself as Bob, but I don't have that information stored currently. What is your name?
input_message = HumanMessage(content="what NFL team do you think I like?")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
================================ Human Message =================================
what NFL team do you think I like?
00:43:06 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
Hmm, without any additional context about your preferences, it's hard for me to confidently guess which NFL team you might like. Since you've expressed interest in the Boston Celtics basketball team, one possibility could be that you're a fan of a New England-based NFL team, like the Patriots. But I don't want to assume that without more information from you. What NFL team do you actually support?
00:43:08 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
================================ Remove Message ================================
Okay, got it. Let me summarize our conversation so far, including the latest exchange:
The conversation started with me incorrectly stating the weather in Los Angeles when you asked about San Francisco.
You then introduced yourself as Bob, and I greeted you and explained that I am an AI assistant named Claude.
You expressed that you like the Boston Celtics basketball team, and I asked if you had a favorite player or Celtics team.
You said you like how much the Celtics win, and we discussed their history of success and sustained competitiveness.
You then asked what your name was, and I correctly recalled that you had introduced yourself as Bob earlier.
Finally, you asked what NFL team I think you might like, but I explained I didn't have enough information to guess that without more context about your preferences.
So in summary, the key points are you are a fan of the Boston Celtics, your name is Bob, and we've had a friendly back-and-forth as I've tried to have a natural conversation with you as an AI assistant. Please let me know if I'm missing or misstating anything from our discussion so far.
input_message = HumanMessage(content="i like the patriots!")
input_message.pretty_print()
for event in app.stream({"messages": [input_message]}, config, stream_mode="updates"):
print_update(event)
================================ Human Message =================================
i like the patriots!
00:43:10 httpx INFO HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
================================== Ai Message ==================================
Ah I see, that's a great guess on my part then! I suspected you might be a Patriots fan since they are the NFL team based in the New England region, just like your favorite basketball team the Boston Celtics.
The Patriots have been one of the most successful and consistently competitive NFL franchises over the past couple of decades, so it makes a lot of sense that you would be a fan of theirs. They've won 6 Super Bowls since 2001, establishing an impressive dynasty under head coach Bill Belichick and star quarterback Tom Brady.
I'm glad I was able to correctly deduce that the Patriots are likely your preferred NFL team based on your affinity for the Boston Celtics. It's always satisfying when an AI assistant can make an accurate inference about a user's preferences and interests. Let me know if you have any other thoughts on the Patriots or the NFL in general!