Google claims its new AI software cuts inference latency by over 90% — but there's a catch
Google’s new open-source TPU Sync aims to speed AI inference by moving KV-cache data between memory tiers, cutting time to first token by up to 90%.
Entities: Google