ConvoCache: Smart Re-Use of Chatbot Responses

The latest speech synthesis and large language models can effectively solve problems such as human-like conversation, but are limited by the high cost and slow response times. To address these limitations, we present ConvoCache, a conversation cache that stores responses so they can be reused in spoken dialogue systems. We test this cache using a dataset of generic chitchat conversations in English. To find a response, we use language models to encode the utterances of a conversation and find similar conversations in the cache. Then the response is taken from this similar conversation and used in the current conversation. By encoding each utterance individually, we can combine them together as a weighted sum, and investigate the effect of different weights on the utterances. Through simulating ConvoCache, we find that using an exponential decay of weights for the previous utterances works best. The similarity of embeddings could be useful for select when to use the cache and when to generate a new response, but unfortunately this showed limited effectiveness with this encoding method.

To measure the quality of the cached responses, we use automatic dialogue evaluation with UniEval and G-Eval. Using the faster UniEval model, we can check cache responses before using them, and find that the nearest similar cached conversation is good 56% of the time, only taking 100ms. There is a cache-miss only 11% of the time, where all top 5 response candidate are evaluated with low coherence scores. Using ConvoCache is able to significantly reduce the cost of chatbots at scale, and reduce the latency from 800ms down to 300–700ms, with an average of 414ms. The difference in latency is significant and allows spoken dialogue chatbots to interact with humans in real-time within the range of human response time preferences.