Apple Researchers Reveal New AI System That Can Beat GPT-4
Apple researchers have developed an artificial intelligence system named ReALM (Reference Resolution as Language Modeling) that aims to radically enhance how voice assistants understand and respond to commands.
In a research paper (via VentureBeat), Apple outlines a new system for how large language models tackle reference resolution, which involves deciphering ambiguous references to on-screen entities, as well as understanding conversational and background context. As a result, ReALM could lead to more intuitive and natural interactions with devices.
Reference resolution is an important part of natural language understanding, enabling users to use pronouns and other indirect references in conversation without confusion. For digital assistants, this capability has historically been a significant challenge, limited by the need to interpret a wide range of verbal cues and visual information. Apple's ReALM system seeks to address this by converting the complex process of reference resolution into a pure language modeling problem. In doing so, it can comprehend references to visual elements displayed on a screen and integrate this understanding into the conversational flow.
ReALM reconstructs the visual layout of a screen using textual representations. This involves parsing on-screen entities and their locations to generate a textual format that captures the screen's content and structure. Apple researchers found that this strategy, combined with specific fine-tuning of language models for reference resolution tasks, significantly outperforms traditional methods, including the capabilities of OpenAI's GPT-4.
ReALM could enable users to interact with digital assistants much more efficiently with reference to what is currently displayed on their screen without the need for precise, detailed instructions. This has the potential to make voice assistants much more useful in a variety of settings, such as helping drivers navigate infotainment systems while driving or assisting users with disabilities by providing an easier and more accurate means of indirect interaction.
Apple has now published several AI research papers. Last month, the company revealed a new method for training large language models that seamlessly integrates both text and visual information. Apple is widely expected to unveil an array of AI features at WWDC in June.
Popular Stories
While the iPhone 17 Pro and iPhone 17 Pro Max are not expected to launch until September, there are already plenty of rumors about the devices.
iPhone 17 Pro's alleged design via Front Page Tech
Below, we recap key changes rumored for the iPhone 17 Pro models as of March 2025:
Aluminum frame: iPhone 17 Pro models are rumored to have an aluminum frame, whereas the iPhone 15 Pro and iPhone...
If you pay for iCloud storage on your iPhone, Apple has a new perk for you, at no additional cost.
The new perk is the ability to create invitations in the Apple Invites app for the iPhone, which launched in the App Store last month.
In the Apple Invites app, iCloud+ subscribers can create invitations for any occasion, such as birthday parties, graduations, baby showers, and more. Anyone ...
Apple today seeded the release candidate versions of upcoming iOS 18.4 and iPadOS 18.4 updates to developers for testing purposes, with the software coming a week after Apple released the fourth betas.
iOS 18.4 and iPadOS 18.4 can be downloaded from the Settings app on a compatible device by going to General > Software Update.
With iOS 18.4, Apple is adding the Priority Notifications...
Apple today announced that AirPods Max with a USB-C port will be gaining support for lossless audio and ultra-low latency audio with a firmware update next month, alongside the release of iOS 18.4, iPadOS 18.4, and macOS 15.4.
For context, audio files are typically compressed to keep file sizes smaller. There are lossy compression standards like MP3 and AAC (Advanced Audio Codec), which...
We're not getting new Siri Apple Intelligence features in iOS 18.4 as expected, but the upcoming update does have quite a few new additions that will be worth upgrading for. We've rounded up the five best features to look forward to, and if you're not running the beta, you can expect to get access to these in early April.
Priority Notifications
If you have an iPhone or iPad that supports...
Apple will launch its long-rumored foldable iPhone next year with a ~$2,000 premium price tag attached, expects well-connected Bloomberg reporter Mark Gurman.
Gurman's comments on Apple's launch plans for its first foldable device appeared in the Q&A section of his latest Power On newsletter. Earlier this month, the reporter said Apple's foldable iPhone could be arriving "as early as 2026,"...
Apple is expected to release iOS 18.4 to the general public as soon as next week, following more than a month of beta testing.
Apple's website says some iOS 18.4 features will be released in "early April," so the update should be out as early as Tuesday, April 1.
Apple this week seeded the iOS 18.4 Release Candidate, which is typically the final beta version, barring the discovery of any...
Leaker Jon Prosser today shared a mockup of what he says the Messages app will look like in iOS 19, demoing an interface with rounded, translucent bubble-shaped navigation buttons at the top and softer, rounder corners for the keyboard and word suggestions.
Jon Prosser's Messages app mockup
The return button, a button for going back to the Messages list, and the FaceTime button have a deeper...
