I cached it. Spark evicted it. We are no longer friends. Anyway, here's what Spark caching actually does under the hood: https://luminousmen.substack.com/p/spark-caching-explained-what-really