Apple Researchers Reveal New AI System That Can Beat GPT-4
Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.
In a research paper (via VentureBeat), Apple outlines a new system for how large language models tackle reference resolution, which involves deciphering ambiguous references to on-screen entities, as well as understanding conversational and background context. As a result, ReALM could lead to more intuitive and natural interactions with devices.
Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple's ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.
ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.
ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.
Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.
Popular Stories
Apple has begun selling the Hikawa Phone Grip and Stand, a new limited-edition iPhone accessory designed with accessibility in mind.
Designed by LA-based Bailey Hikawa to celebrate the 40th anniversary of accessibility at Apple, the grip uses magnets to securely snap onto any iPhone with MagSafe.
Apple says it can be removed with ease, and doubles as a stand with two different viewing...
iOS 26.2 is currently in beta testing. The upcoming update includes a handful of new features and changes on the iPhone, including a new Liquid Glass slider for the Lock Screen's clock, offline lyrics for Apple Music, and more.
In a recent press release, Apple confirmed that iOS 26.2 will be released to all users in December, but it did not provide a specific release date.
Keep reading...
Apple's annual four-day Black Friday through Cyber Monday shopping event is returning on Friday, November 28 through Monday, December 1 in many countries, including the United States, Canada, Australia, New Zealand, France, Germany, Italy, Spain, the United Kingdom, Belgium, the Netherlands, Sweden, Thailand, and others.
During the shopping event, customers can get an Apple gift card with...
iOS 27 will reportedly have two major elements: quality improvements and new AI features.
In his Power On newsletter today, Bloomberg's Mark Gurman said that iOS 27 will be similar to Mac OS X Snow Leopard, in the sense that Apple is focused on improving "quality and underlying performance" over adding new features.
Gurman said there is one exception to this rule, though, as he expects...
On this week's episode of The MacRumors Show, we talk through all of the new features and improvements expected to come to next year's iPhone 18 Pro and iPhone 18 Pro Max models.
Subscribe to The MacRumors Show YouTube channel for more videos
Apple's next-generation iPhones are less than ten months away and we already have a good idea about what to expect based on corroborated leaks, rumors,...
Apple's first foldable iPhone is expected to launch alongside the iPhone 18 Pro models in fall 2026, and it's shaping up to include two standout features that could set it apart from the competition.
The book-style foldable will reportedly feature an industry-first 24-megapixel under-display camera built into the inner display, according to a recent JP Morgan equity research report. That...
We're only a few days away from Black Friday, which will take place on Friday, November 28 in 2025. As always, this will be the best time of the year to shop for great deals, including popular Apple products like AirPods, iPad, Apple Watch, and more. In this article, the majority of the discounts will be found on Amazon.
Note: MacRumors is an affiliate partner with some of these vendors. When...
Black Friday is just over a week away, and iPad deals have finally started to flood in at retailers like Amazon and Best Buy. Below we're tracking discounts on every current generation iPad, including lowest-ever prices on M3 iPad Air and M5 iPad Pro, plus steep markdowns on iPad and iPad mini.
Note: MacRumors is an affiliate partner with some of these vendors. When you click a link and make a ...
Apple's iPhone development roadmap runs several years into the future and the company is continually working with suppliers on several successive iPhone models at the same time, which is why we often get rumored features months ahead of launch. The iPhone 18 series is no different, and we already have a good idea of what to expect for the iPhone 18 Pro and iPhone 18 Pro Max.
One thing worth...
