In the rapidly evolving landscape of artificial intelligence, machine learning models—particularly large foundation models—are increasingly evaluated across a multitude of axes that reflect their real-world utility. Beyond the traditional accuracy measures tailored to a specific task of interest, stakeholders now routinely examine metrics that capture safety, calibration, and fairness properties. However, this growing complexity in evaluation is running headfirst into a foundational problem regarding how these models are tested. The classical approach to these problems assumes that the test set has been strictly held out from the training process, an assumption that is becoming increasingly difficult to justify in the era of large foundation models trained on massive, web-scale datasets. Addressing this mounting methodological crisis, recent research has formalized a notion of evaluability without relying on the traditional held-out test set assumption. The core question driving this line of inquiry is whether it remains feasible to evaluate black-box models in settings where test samples may have inadvertently or deliberately been used during the training phase. This problem has been studied under various settings, capturing different levels of access an evaluator might have to the model in question. Read Also: Researchers Introduce "Minions," a Distributed Virtual GPU Cloud Harnessing Idle Apple Silicon for Machine Learning Inference Trajectory-Shaped Discrete Flow Matching Breakthrough Shatters Speed and Quality Bottlenecks in Language Generation To understand the scope of this challenge, researchers first analyzed the most constrained and natural setting: one where the evaluator only looks at the output of the model on the given test set without any additional interaction. Through rigorous analysis, the study fully characterizes the class of distributions where accurate evaluation is still possible under these conditions, establishing tight upper and lower bounds on the sample complexity of evaluation. The implications of this first finding are profound for the AI community. The results demonstrate that black-box evaluation in this strictly observational setup is feasible if and only if the underlying data distribution is close to having a small support. When models are evaluated on vast, highly diverse distributions common to modern foundation models, simple observation of test set outputs becomes fundamentally insufficient if data contamination has occurred, highlighting the severe limitations of traditional evaluation pipelines in the modern era. Moving beyond purely observational methods, the researchers then considered more interactive evaluation algorithms—specifically those that can query the model on additional, strategically chosen inputs. This exploration revealed fascinating connections to self-correctors, a concept well-studied in classical program testing. By leveraging these connections, the study shows that even a small amount of query access can significantly increase the power and reliability of the evaluator, particularly for natural distributions and concept classes that would otherwise be intractable to assess under data contamination. Related readings and updates. Speech foundation models have recently achieved remarkable capabilities across a wide range of tasks, transforming how machines understand and process human audio. However, their evaluation remains disjointed across different tasks and model types. Because different models excel at distinct aspects of speech processing, they naturally require different evaluation protocols, creating a fragmented ecosystem for developers and researchers alike. To address this fragmentation, related work has proposed a unified taxonomy that tackles a central question in the field: Which evaluation is appropriate for which model? The newly defined taxonomy attempts to bring order to the chaos by establishing three orthogonal axes to categorize speech model assessments, providing a structured framework for researchers striving to benchmark audio-based foundation models accurately and consistently. Meanwhile, the practical deployment of foundation models continues to accelerate across the technology sector. At the 2024 Worldwide Developers Conference, Apple introduced Apple Intelligence, a personal intelligence system integrated deeply into iOS 18, iPadOS 18, and macOS Sequoia. Designed to bring advanced AI capabilities directly to consumer devices while maintaining strict privacy standards, Apple Intelligence represents a major milestone in how generative models are packaged and deployed for end-users. Apple Intelligence is comprised of multiple highly-capable generative models that are specialized for users’ everyday tasks, designed to adapt on the fly to their current activity. The foundation models built into Apple Intelligence have been carefully fine-tuned for practical user experiences, such as writing and refining text, prioritizing and summarizing notifications, creating playful images for conversations with family and friends, and taking in-app actions to simplify interactions across various applications. As these on-device and server-based foundation models become ubiquitous, the underlying questions of robust evaluation, data integrity, and performance verification—such as those addressed in recent evaluability research—will only grow more critical for the broader artificial intelligence community. Post navigation Apple Silicon Breakthrough: Running Frontier-Class AI Models Locally with MLX and the M5 Ultra Mac Studio