fluximp
Optimizing token usage for my AI coding agent: https://github.com/fluxmoot/fluximp Its slow going. When I started this near the beginning of the year the ratio between input and cached tokens costs for the interesting open weight models was less than 10 and provider side cache hit rates weren't very good, so it wasn't expensive to break the prefix to remove old things from the context and do rolling summarization to keep the prompt size down. Now some DS v4.1 Flash providers have a ratio of 33 to 50 and >99% cache hit rates. That makes the cost of prefix cache breaks a lot higher and its definitely harder to get ahead on total cost* of the usual approaches to context management for DS v4.1 Flash specifically. * - the number of requests is reduced by a factor of ~3 relative to oh-my-pi on the same benchmark; the task is long enough to require compaction once when using oh-my-pi