Apple Preparing for Upcoming Siri Onscreen Awareness Feature With New iOS 18.2 API for Developers

Apple is working on more advanced Siri functionality as part of its Apple Intelligence feature set, and to prepare, it has been providing developers with App Intent APIs so apps will be ready for the new capabilities.

Apple Intelligence General Feature 2
With the latest wave of betas, Apple has a new API that lets developers make onscreen content in their apps available to ‌Siri‌ and ‌Apple Intelligence‌. From Apple's documentation:

When a user asks a question about onscreen content or wants to perform an action on it, Siri and Apple Intelligence can retrieve the content to respond to the question and perform the action. If the user explicitly requests it, Siri and Apple Intelligence can send content to supported third-party services. For example, someone could view a website and use Siri to provide a summary by saying or typing a phrase like "Hey Siri, what's this document about?"

In the iOS 18.2 beta, ChatGPT integration with ‌Siri‌ allows users to ask questions about photos and documents, such as PDFs and presentations, and get information about them. You can, for example, ask ‌Siri‌ "what's in this photo?" and ‌Siri‌ will take a screenshot and hand it over to ChatGPT. ChatGPT then relays what's in the image, and the same feature works for PDFs and other documents.

It does not seem that the iOS 18.2 ChatGPT integration is the onscreen awareness functionality that Apple has planned for ‌Siri‌, but it could be related. Apple describes onscreen awareness as the ability for ‌Siri‌ to understand and take action on things on the screen. If someone texts you an address, for example, you'll be able to say "Add this address to their contact card," and ‌Siri‌ will do it. This functionality is not available in iOS 18.2, and ChatGPT is limited to assessing screenshots, but it is somewhat confusing.

Onscreen awareness, like personal context and in-app actions, is a feature that Apple has planned for ‌Siri‌, but it's probably not something that we're getting this year. Many of the ‌Siri‌ features are coming in a future version of iOS 18, and Bloomberg's Mark Gurman has said we can expect to see them in iOS 18.4, an update set to be released in the spring of 2025.

While there are multiple ‌Siri‌ features that won't be coming until next year, Apple is providing developers with APIs in advance so that developers have several months to prepare and so the features are ready for the public when the updates actually come out.

Related Roundups: iOS 18, iPadOS 18
Related Forums: iOS 18, iPadOS 18

Popular Stories

iOS 18

Apple Says iOS 18.4 Will Be Released in April With These New Features

Wednesday February 26, 2025 7:15 am PST by
In a recent press release, Apple confirmed that iOS 18.4 will be released in April. From the Apple News+ Food announcement:Coming with iOS 18.4 and iPadOS 18.4 in April, Apple News+ subscribers will have access to Apple News+ Food, a new section that will feature tens of thousands of recipes — as well as stories about restaurants, healthy eating, kitchen essentials, and more — from the...
Generic iPhone 17 Feature With Full Width Dynamic Island

Latest iPhone 17 Series CAD Images in Line With Redesign Rumors

Friday February 28, 2025 2:51 am PST by
Apple is expected to embrace a new camera system design for some models in its upcoming iPhone 17 series, and the latest purported CAD images don't deviate from what we have been hearing lately about Apple's new lineup. If you do not like the sound of an iPhone with a Google Pixel-style camera bar, look away now. Seasoned leaker Sonny Dickson shared the following images in a post on X...
iphone 16e usb c feature

Apple Provides Reason for iPhone 16e's Lack of MagSafe

Friday February 28, 2025 4:39 am PST by
Apple has offered a reason why the iPhone 16e doesn't include MagSafe, one of the more notable omissions from its latest entry-level smartphone. According to Apple representatives who spoke to Daring Fireball's John Gruber, MagSafe is not included in the iPhone 16e because "most people in the iPhone 16e's target audience exclusively charge their phones by plugging them into a charging...
apple intelligence black

These New Apple Intelligence Features Are Coming in iOS 18.4

Friday February 28, 2025 3:17 pm PST by
iOS 18.4 was supposed to bring new Apple Intelligence Siri features, but Apple ended up needing to pull those capabilities from the update to continue testing. There are fewer new Apple Intelligence additions now, but there are still some new features that will make the update worth installing when it comes out in April. Priority Notifications Apple introduced Priority Notifications back at ...
Generic iOS 19 Feature Mock Light

iOS 19 Rumored to Include These New Features for Your iPhone

Saturday March 1, 2025 11:00 am PST by
iOS 19 is still around three months away from being unveiled, but there are plenty of rumors about the upcoming update. Below, we recap iOS 19 rumors so far. Redesigned Camera App A leak earlier this year allegedly revealed a redesigned Camera app coming with iOS 19. On his YouTube channel Front Page Tech in January, Jon Prosser shared a video showing what the new Camera app will...
cook trump

Trump Responds to Apple Keeping Diversity Policies

Wednesday February 26, 2025 6:32 am PST by
In an all-caps post on Truth Social today, U.S. President Donald Trump said Apple should fully end its diversity, equity, and inclusion (DEI) policies. Tim Cook meeting with President Trump in 2017 "APPLE SHOULD GET RID OF DEI RULES, NOT JUST MAKE ADJUSTMENTS TO THEM," he wrote. Trump's post comes one day after Apple held its annual shareholders meeting, during which a majority of...
apple watch ultra snow

6 Features Coming to the Apple Watch Ultra 3

Tuesday February 25, 2025 9:00 am PST by
The Apple Watch Ultra 3 is expected to launch later this year, arriving two years after the previous model with a series of improvements. While no noticeable design changes are expected for the third generation since the company tends to stick with the same Apple Watch design through three generations before changing it, there are a series of internal upgrades on the way. By the time the ...
airpods pro purple

Here's When AirPods Pro 3 Are Rumored to Launch

Monday February 24, 2025 9:14 am PST by
According to a post on X today from a leaker known as Kosutami, Apple plans to launch AirPods Pro 3 in May or June this year. The leaker also claimed that an AirTag 2 will launch around the same time. Kosutami is best known as a collector of prototype Apple hardware, but they have occasionally shared accurate information about Apple's future product plans. For example, they accurately...
apple c1

How Fast is Apple's First-Ever 5G Modem? The Results Are Surprising

Friday February 28, 2025 10:08 am PST by
iPhone 16e reviews are now out, and Apple's custom-designed C1 modem has been put to the test. The results so far are quite surprising, as the C1's speeds are not as slow compared to Qualcomm modems as originally expected. While the C1 does not support ultra-fast mmWave 5G in the U.S., it appears to offer comparable 5G performance to Qualcomm's Snapdragon X71 modem found in the iPhone 16,...

Top Rated Comments

smoker68x Avatar
17 weeks ago
Is this going to be a one time universal permission to Siri, or an individual access per app like camera/microphone/location?
Score: 4 Votes (Like | Disagree)
Fraserpatty Avatar
17 weeks ago
This is the type of Siri functionality that I have been waiting for. I hope Apple doesn’t disappoint.
Score: 3 Votes (Like | Disagree)
dannys1 Avatar
17 weeks ago

but it is somewhat confusing.
Is it?
Score: 2 Votes (Like | Disagree)
deeside81 Avatar
17 weeks ago
Siri still doesn’t know my Scottish accent. It’s absolutely useless and doesn’t work with me. After 10 years they’ve had plenty of time to fix Siri not understanding accents. If I put an American accent on it understands me clearly. Rubbish.
Score: 2 Votes (Like | Disagree)
alexjholland Avatar
17 weeks ago
Siri still gets confused when I try to turn off the lights.

Forgive me if I'm cynical.
Score: 2 Votes (Like | Disagree)
svish Avatar
17 weeks ago
Good to know. Should make Siri much more useful.
Score: 2 Votes (Like | Disagree)