GPTCache
GPTCache is a library for creating semantic caches for large language models queries. It supports LangChain and llama_index integrations, and can also be used from different programming languages thanks to its Docker server image. This library aims to reduce API costs and improve response times.
GPTCache
72zilliztech · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- You create semantic cache for large language model queries.
- You use LangChain and llama_index integrations in their projects.
- You cache from different languages through the Docker server image.
- ↑ PositiveHacker News
GPTCache: Slash Your LLM API Costs by 10x
“It looks like a game-changer for those working with LLM services. By caching query results, it effectively cuts down the number of requests and token count sent to the LLM service, leading to a substantial reduction in overall costs. If you're leveraging LLMs for your projects,…”
- • NeutralHacker News
Show HN: GPTCache – Redis for LLMs
“Smart: > However, using an exact match approach for LLM caches is less effective due to the complexity and variability of LLM queries, resulting in a low cache hit rate. To address this issue, GPTCache adopt alternative strategies like semantic caching. Semantic caching…”