Future Apple devices will share and edit 3D AR images
Instead of passively displaying virtual Apple AR Objects “Apple Glass” can allow users to share 3D data and manipulate it in editing applications.
Apple’s countless patents and patent applications for Apple AR already include some for displaying 3D data from other devices. However, two new patents show that Apple wants users to be able to selectively share or retrieve objects and also edit them.
“Viewing shared 3D content from other devices” is a recently disclosed patent that proposes that head-mounted displays (HMDs) could use “three-dimensional (3D) content from a separate device.”
Apple’s patent is unusually blunt on current systems that attempt to do this. “Existing computer systems and applications do not adequately facilitate the sharing and use of 3D content to provide and use CGR environments on electronic devices,” he says.
What Apple wants to see is an HMD capable of displaying “3D content corresponding to data objects received in CGR environments”, regardless of their origin. Instead of all users seeing AR or VR versions of objects generated by their own hardware, they should be able to see what other people have done.
“For example, a first user, who is at home using a first device,” says Apple, “may receive a data object for a sofa from a second user who is in a retail store looking at the sofa.”
“In this example, the second user uses a second device to create or identify a data object corresponding to the sofa,” he continues, “for example, using a camera from the second device to create a file that includes a 3D model of the sofa or to identify a file or the data storage address of a file that includes a 3D model of the sofa.
Patent detail showing (top) a real world environment and (bottom) the same space with a shared 3D object in it
Just as you can now place furniture from Ikea’s database in that company’s augmented reality app, here a friend from the local store might send you a 3D version of the sofa.
The patent deals mainly with different ways of capturing and transmitting that object. Include details on how to accompany the object with text, such as an email message, identifying the source.
“ the first user can reposition the sofa relative to the real-world tables in the captured images, “says Apple,” and then physically move the device around the room to see the sofa from different vantage points within the room. ”
Manipulate shared 3D objects
“Method and system for 360 degree head-mounted display monitoring between software program modules using image or video texture sharing” is a second new related patent. This is about how users can receive 3D information in a file and then manipulate it correctly.
“Conventional 180 or 360-degree videos and / or images are stored in flat storage formats using equirectangular or cubic projections to represent spherical space,” Apple explains. “If these videos and / or images are edited in conventional graphics or editing applications, it is difficult for the user to interpret the experience of the final result when the video or images are distributed and presented in a dome, cubic or spherically mapped projection within of a head-mounted virtual reality screen. ”
“Editing and manipulating these images on these flat projections requires special skill and a lot of trial and error,” continues the patent. “Also, it is not an uncommon experience to realize, after manipulating composite or spherical-edited images or videos, that subsequent shots are misaligned or that the stereoscopic parallax points do not match naturally.”
Detail of the patent showing part of the rendering to correctly edit shared 3D objects
Apple’s proposal is to “continuously acquire orientation and position data” from an HMD and “simultaneously render a monoscopic or stereoscopic view representative of that orientation on the head-mounted display, in real time.”
So a user gets a 3D image, perhaps shared by another user, and the HMD alters its screen to match the user’s position. It also dynamically modifies the image that is displayed “when the media manipulation application modifies the image data.”
The first patent is credited to five inventors, including Norman N. Wang. Their previous related patents include one on “generation of high-fidelity, refined 2D and 3D textures.”
Timothy Dashwood, who invented the second patent, is also credited with a previous one related to the stereoscopic representation of virtual objects in 3D.