Google claims its new AI software cuts inference latency by over 90% — but there's a catch

Google’s new open-source TPU Sync aims to speed AI inference by moving KV-cache data between memory tiers, cutting time to first token by up to 90%.

The New Stack · Amanda Caswell · https://www.facebook.com/thenewstack

Entities: Google