Apple enters the augmented reality fray with ARKit for iOS

https://techcrunch.com/2017/06/05/apple-enters-the-augmented-reality-fray-with-arkit-for-ios/

by salimmadjd • 9 years ago

169 100 9 years ago

tuyguntn• 9 years ago

How does it compare to Google Tango? Tango needs special hardware, but ARKit is not. Even if it required, there would be a definitely device (iPhone or iPad). Traction wise Apple could win, but what about performance and accuracy wise?

asadlionpk• 9 years ago

For now it's more like how Snapchat's latest world filters work (try them out). Basic environment tracking, good enough for a smooth experience, considering you keep your camera steady.

Tango takes the tracking to the next level. Maybe future iPhones will have that kind of IMU and other sensors, allowing Apple to expand their AR API.

Eridrus• 9 years ago

I was pretty impressed with their live demo.

I tried Pokemon Go briefly, and their "AR" was a complete joke compared to this.

paul92905• 9 years ago

I thought we might be seeing an AR app built into the Camera or Map app.

Nope just a kit for developers to build games only or does the ARkit allow access to the camera app? I'm dying to see innovation from Apple like in the camera app when you open it there's an option to view how things look in the past(100 years ago what did this spot in New York City look like.. where you can take a pic of yourself in the past).

I'm sure there's oodles that can be done with an AR Camera.

Apple and the others to me seem to be slow to innovate.

basisword• 9 years ago

I might be wrong but it sounds like 3rd party devs can build those things using ARKit.

aylmao2• 9 years ago

Yeah, and that's the whole point of it. There's too many applications of AR for Apple to build alone.

Someone2• 9 years ago

Even ignoring that you'll need multiple photos to do the 3D reconstruction you need for augmented reality, that 100 year old photos may be lacking in resolution, and giving a few decades of leeway either way for that "100 years ago" I doubt even Google has enough data to have a reasonable chance of having a photo for random locations you might point your camera at.

paul92905• 9 years ago

Does not have to be exact and a real photo from that time in that spot or a tad far away and or an arial shot can be used to create renderings.

mendeza4• 9 years ago

I think this is a good thing they are not baking AR into the camera like Facebook is doing with the Camera Effects Platform.

This allows developers to invent future "camera" apps that could be better to use than the regular camera.

huangc104• 9 years ago

How does AR work without integration with the camera? They showed a demo of dev AR camera view where the camera identified "surface".

eridius2• 9 years ago

AR requires camera access, but I'm rather confused as to why you think it would be integrated with the built-in Camera app. How would that work?

paul92905• 9 years ago

Open the camera app and at the bottom click AR... an overlay appears with options like..

- Time Machine(idea noted above)

- Daylight View (if currently nighttime)

- and other AR apps devs or Apple creates.

htormey• 9 years ago

ARKit was one of the most interesting features announced this WWDC. Can't wait to play around with some demo code and see what it's like.

synaesthesisx4• 9 years ago

They're paving the way for their revolutionary consumer AR eyewear (which may be delayed now apparently). Although maybe we'll see tech from some of their recent AR acquisitions (PrimeSense/Metaio) integrated into the upcoming iPhone, possibly in the form of an advanced depth/surface mapping sensor of sorts.

What's fascinating is Apple has a large team focusing on HCI & AR - what "pinch-to-zoom" and multitouch did for smartphones is what Apple will need to successfully pull off the "big bang" for consumer AR.

scarlac• 9 years ago

which may be delayed now apparently

I'm curious, what have you read/seen that makes you think this unknown product is delayed before being announced?

threeseed• 9 years ago

There was an AMA on Reddit from a Foxconn employee who looks to be pretty spot on about other products:

https://www.reddit.com/r/apple/comments/6ezhwm/iama_foxconn_...

Rough bill of materials of Project Mirrorshades - Apple Iris. Note Iris is Siri backwards. Kopin NED Acetate frame Polarisized or prescription lens with Zeiss smart optics Bone conduction modules Microphones (noise cancellation) Light sensor Accelerometer for step tracking and head movement app navigation Magnetometer for navigation Capacitive Pavel Ceramic battery Apple chipset Charging circuit BL5 Induction module

Crystal, champagne and black. We only saw transparent reader lens but I understand they can procure polarised or prescription lenses Cellulose acetate Injection mold frames into an aluminum mold. Colors are added and tumbled for finish. P3 frame design

Two sizes, men's and woman's

Usually 4x the BOM cost; it would indicate around $600 mark. Crystal is just clear. But clear without artefacts is difficult.

rubyn00bie• 9 years ago

I'd guess articles like this one, but I feel like I've seen a specific reference at some point too but couldn't find it with a quick search:

https://www.macrumors.com/2017/03/27/apple-stepping-up-ar-ey...

synaesthesisx4• 9 years ago

Many were speculating that these glasses would be Apple's first major new product release at their new headquarters/"Spaceship", coinciding with the 10th anniversary iPhone launch. Robert Scoble has been mentioning the Zeiss partnership since CES, and recently someone from Foxconn/Hon Hai had allegedly leaked some details about a Google-Glass-like device with a near eye display in the bill of materials. Supposedly they won't be ready this year due to unique design challenges, but who knows for certain...

I work closely with people in the AR industry and everyone knows Apple has been investing heavily in developing this wearable device. Augmented reality is the next major fundamental platform shift, and Apple is hoping it will play the role the iPhone did as the catalyst for the smartphone/mobile era - but this time for consumer AR.

What Apple announced today is simply the beginning - they are providing tools so they can establish the foundation for "the largest AR platform in the world" (as mentioned at WWDC today).

Side note: There's a video from the 90's or late 80's showing an Apple concept for augmented reality glasses - it features a woman translating French or something while reading a restaurant menu. I can't seem to find it but it was on YouTube at some point...

izacus• 9 years ago

There's an AR fray? Google threw their weight behind VR on mobile, who else is in the AR space?

mattbettinson• 9 years ago

Microsoft?

cjCamel• 9 years ago

Hololens. Probably the most powerful AR device available atm.

aylmao2• 9 years ago

Microsoft, Snapchat, Google, Facebook.

Facebook made a big deal in F8 out of adding cameras to all its applications. They're in the AR game big time.

https://www.nytimes.com/2017/04/18/technology/mark-zuckerber...

DonHopkins• 9 years ago

Magic Leap is conspicuously but rightly missing from your list, because they're a fraud that's nowhere near releasing a real product, and whose bullshit is besmirching and dragging down the reputation of AR.

https://www.youtube.com/watch?v=w8J5BWL8oJY

wtfishackernews• 9 years ago

Google's Tango is AR

moron4hire3• 9 years ago

Google has at least three AR products: Glass is still in business for enterprise, Tango tablets and phones are finally out, and future versions of Daydream are supposed to be capable of inside-out tracking via an AR tracking system.

Microsoft has the HoloLens, which technology they have also cannibalized for a series of headsets they are dubbing "Mixed Reality" for the depth camera pass-through they are performing.

HTC is supposedly developing their own AR/inside-out system.

Leap Motion, Intel, Qualcomm, Meta, and Magic Leap are still considered in the running, with varying levels of believability.

And then there are a slew of different companies that are doing AR solutions with either the standard webcam (e.g. Facebook, Snap, Vuforia, Wikitude) or their own, standalone camera (e.g. Zed, Occipital).

X86BSD• 9 years ago

I don't count Google. Google is an unreliable product company. They kill half their products and the other half are absolute tire fires. You may like android, but it's ugly AF compared to iOS which is far from perfect. Gmail needs some major work on the interface. Speaking of interfaces have you seen their tv app? Kill me now.

I have no love for Google. I certainly don't trust them enough to purchase a product they make for fear of them killing it. Plus they mine everything they can get access to for who knows what ends. "So for those reasons, I'm out".

MS.... Meh. No one has come to expect great things from them.

The market will decide who wins though.

runesoerensen• 9 years ago

More information and docs here https://developer.apple.com/arkit/

stesch2• 9 years ago

craigching• 9 years ago

I thought 6S+ ran A9?

wsinks• 9 years ago

Thank you for this! Perfect link.

Stanleyc233• 9 years ago

There was also a very brief mention of a "depth api" before they started talking about the actual ARKit. is that endpoint documented anywhere?

huangc104• 9 years ago

here? https://developer.apple.com/documentation/arkit

Stanleyc233• 9 years ago

nah I think its something for normal photos, not immersive content.

Camera: Updated video and image compression, improved Portrait Mode with support for low light, plus a new Depth API for developers.

from https://venturebeat.com/2017/06/05/apple-announces-ios-11/

masklinn• 9 years ago

Nah, it was during the camera bits, they noted that the dual camera lets them add depth querying/manipulation and they're providing an API for it.

josecastillo• 9 years ago

I think you're referring to this, depth data mapping from the dual camera in the iPhone 7+:

https://developer.apple.com/documentation/avfoundation/avdep...

roymurdock2• 9 years ago

Studied the market for AR development solutions recently. A few thoughts:

- Probably using a lot of Metaio tech, a leading AR dev solutions provider Apple acquired/shuttered in May 2015

- Unfortunate naming. Sounds extremely similar to ARToolkit [1], the leading open source/free AR dev solution.

- Markerless is the future of AR, so good on Apple for getting this markerless tech out to compete with Google Tango and Microsoft Hololens. The status quo is currently marker-based recognition using "tags" or QR codes. It's also much harder than marker-based, or "on body" (Snapchat, Facebook AR Studio) recognition. I'd say Google Tango has some of the most impressive SDKs out there for tracking, area learning, and depth perception, but it does require specialized smartphone hardware from Lenovo or Asus, which significantly limits its utility and mkt penetration at this point. Google was just too early with Glass and got burned, but it needs a stronger hardware platform, which is something that Apple can deliver.

- The Unreal-engine AR video demo was cool and the graphics seemed decent for real-time rendering on an iPad, but the real future of AR applications (read: $$$) will come from industrial applications on wearable devices or head mounted displays such as the Hololens, Vuzix, Daqri, or Epson Moverio devices. Examples of industrial/enterprise AR applications: Remote help, complex assembly, pick & pack, line monitoring, materials handling, systems training, etc. This is the market Microsoft and PTC (Vuforia, ThingWorx) are targeting, not the consumer gaming/advertising markets, which use more "basic" forms of AR. This is not to say Pokemon Go/Snapchat do not generate a lot of revenue, but it's very debatable if they can be considered true AR applications.

- Other vendors in the AR development solutions space include: Aurasma (HPE), Blippar, Catchoom, EON Reality, Kudan, Pikkart, Wikitude, among others.

- In conclusion, Apple is currently stuck between a rock and a hard place. It's somewhat late to market with ARKit, will have to compete with Facebook, Google, and Snapchat for consumer AR-oriented developer mindshare, and will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family.

- The best way for Apple to alleviate this jam is to release its own AR glasses/headset, which it is widely rumored to be developing.

[1] https://artoolkit.org/

mendeza4• 9 years ago

For AR, to win, you do not have to be the first one to market to win, its who can make the best platform for developers will win. As an AR developer, previous platforms like Vuforia, ARToolkit, and even Tango are either limiting or very hard to work with.

moron4hire3• 9 years ago

What? Vuforia is extremely easy to work with. Are you sure your difficulty issues weren't just Unity being a garbage heap of bad UI and poorly documented processes?

stesch2• 9 years ago

Other platforms don't limit you to just one operating system.

chrisdun• 9 years ago

| Probably using a lot of Metaio tech

Almost certainly - Metaio presented realtime lighting estimation at InsideAR (their dev conference) in late 2014. It's also been shown by Hans Kaufmann's research at TU Wien in Austria [1] although this has tighter links with other vendors than Metaio.

| Other vendors in the AR development solutions space include

I've had the pleasure of dealing with most of these folks within the last few years. They all have their own specialisms around certain tracking methodologies, pricing models and other features. Evaluate them with your own purposes in mind, and choose the right tool for the job.

| will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family

In this market, Vuforia really does have it fairly cornered - it seems to be a chunk of what PTC are chasing, although they're far from turning a blind eye to B2C.

Edit: Grammar fail.

[1] https://www.ims.tuwien.ac.at/research/virtual-and-augmented-...

Jare• 9 years ago

As you say, the Wingnut demo had nice content, but the whole experience looked unbelievably bad. Like, "original Wii ads with people jumping behind their couches" bad. If they showed it as a tech demo for integrating VFX in real time preview when shooting a movie I would have bought it 100%.

asmithmd1• 9 years ago

Just today PTC announced free trial access to their AR tools:

https://studio.thingworx.com/

Currently locking the AR to the real world requires a QR code-like mark but they are demoing AR with no mark. Not having a mark requires 3D data of something in the field of view:

https://www.youtube.com/watch?v=-S-4oJ8C_fo

skip to minute 19 to see the markless demo

figomatj• 9 years ago

Thingworx 8.0 will release, and Thingworx Studio will be upgraded. The new version of Thingworx studio can recognize the Thing Object by Thing shape just like demo video, not only by QR Code-like mark.

pducks32• 9 years ago

There's also Sturfee which focuses on outdoor spaces which is a very hard problem for simple AR toolkits like this. Depth is hard and so most of these players cannot do outdoor AR. sturfee.com DISCLAIMER: I work at Sturfee and it's really great.

sandover• 9 years ago

In conclusion, Apple is . . . somewhat late to market . . . The best way for Apple to alleviate this jam is to release its own AR glasses/headset, which it is widely rumored to be developing.

I'm not sure this line of reasoning holds up. You mention 3 markets:

1. consumer

2. enterprise

3. the "market" for developer mindshare

Apple doesn't especially care about being early or late to any of these markets. Let's take them one by one.

1. There is currently no real consumer market for AR, so Apple is not late. And I don't see any reason why Apple would not be content to wait and see if one develops. It's not clear how releasing an improved version of Google Glass saves Apple from any kind of "jam". People are not clamoring for this type of product, because its value is highly unclear.

2. There is currently no real enterprise market for AR, so Apple is not late. Apple has no history of making industrial equipment. Are you saying Apple is about to take the unusual step of starting a whole new sales channel, targeting a whole new class of buyers, to try to get first-mover advantage in a new and unproven market helping warehouse workers do line monitoring and "remote help"? It seems unlikely.

3. Apple has a huge, strong developer base it is in no danger of losing. Again, it's not clear how offering an AR SDK gets them out of a jam.

If I had to guess as to why they are playing in this area, it may be to help seed the ground with more iOS devs familiar with this tech, so that in the event that the market does develop, they are in a good position to harvest ideas, talent, and apps.

roymurdock2• 9 years ago

1. Consumer AR market - mobile phone AR-enabled applications - education/gaming/entertainment/enhanced retail. Niantic, Blippar, Catchoom, Snapchat would all beg to differ that this market does not exist.

2. Enterprise/industrial AR market - wearable AR-enabled applications - I've already listed them. This market is very small currently...but it exists. I've talked to engineers and AR tool vendors in this market, creating applications for Vuzix, Epson devices. My argument is that this portion of the market will see a lot of growth over the next 5 years. Hard to explain Microsoft's play with Hololens, or PTC's ThingWorx Studio tracking tech away.

3. They are in a jam with AR application developers specifically, not all developers as a whole. You're right they have a large, strong mobile developer base, but AR developers will move away from mobile/tablet-focused platforms (iOS) to Android/Windows in order to develop for AR wearables.

They're playing in AR because they're looking for the new iPhone...a new form factor to deliver new applications and jump start sales. AR glasses could be that form factor if they can execute like Jobs did with the smartphone.

oliyoung• 9 years ago

In conclusion, Apple is currently stuck between a rock and a hard place. It's somewhat late to market with ARKit, will have to compete with Facebook, Google, and Snapchat for consumer AR-oriented developer mindshare, and will have no play at the industrial/enterprise market if it limits its tools to development for the iOS family.

See also portable music, phones, tablets …

It's Apple's SOP, and if they've shown anything in the last 10-15 years, entering markets late is not a barrier to them competing

artfiend• 9 years ago

Do you know if Apple's new markerless AR tech automatically scales the digital object; or does the user resize the object to suit themselves?

MBCook6• 9 years ago

The documentation says that it can do rough measurements and automatically scales things correctly. They pointed that out in the demo where the coffee cup was roughly the right size for a coffee cup instead of 5 feet tall or something like that.

huangc104• 9 years ago

As a developer, this was the most exciting thing for me from today's WWDC opening speech. Can't wait to try this out myself. The demo they did with the AR game was really neat.

satysin2• 9 years ago

It seemed quite mature for a v1 release. Then again it is hard to judge from an Apple demo but it did look nice.

huangc104• 9 years ago

I think there must have always been some way to do AR (Pokemon GO and other games) but I believe it always required a team to accomplish this and possibly work with engineers at Apple.

As an independent developer, I hope the AR kit will make it much easier for me to create something fun and possibly integrate with Siri and other kits. This is what I'm hoping for :D

I know VR is hot right now, but I've always had the thought AR > VR.

agumonkey2• 9 years ago

Barely related but what happened to HealthKit ? are ambitious Apple Kits failing ?

dpkonofa• 9 years ago

HealthKit is still going. They announced some big wins with HealthKit on the Apple Watch but they weren't included in the keynote. More than likely, there wasn't enough time considering the keynote without any HealthKit announcements was already longer than normal.

agumonkey2• 9 years ago

Thanks. Last month some medtech guy reported about different efforts and said HealthKit vanished. The fact Apple didn't mention made me think he was right. Now I'll wait for more.

aSig• 9 years ago

Is anyone able to find the ARKit sample code that was mentioned during the keynote?

tarentel• 9 years ago

I believe it was briefly mentioned that it wouldn't be available until this fall.

rfinman• 9 years ago

You can download the beta iOS 11 and xCode 9 and test it out. It is pretty impressive

enos_feedler• 9 years ago

Does anyone have resources for "State of the art" algorithms and research that is used to do the sensor fusion (camera frames + motion sensor data)? I found this but not sure its the most up to date. I'm also looking for more detail in terms of code:

http://user.it.uu.se/~thosc112/pubpdf/holsgs2006.pdf

steinomri• 9 years ago

It seems that ARkit is actually using SLAM as the core technology (in parts, the same technology is being used in autonomous cars and navigating drones). Getting it to work on mobile phones is quite challenging and had been the subject of much research in recent years, some good starting points (with code): ORB-SLAM - https://github.com/raulmur/ORB_SLAM2, https://www.youtube.com/watch?v=T-9PYCKhDLM DSO - https://github.com/JakobEngel/dso, https://www.youtube.com/watch?v=C6-xwSOOdqQ VINS - https://github.com/HKUST-Aerial-Robotics/VINS-Mobile, https://www.youtube.com/watch?v=0mTXnIfFisI&feature=youtu.be

(the last one being released just a few days ago...)

Of course there are a lot more resources out there and a lot of the cutting edge research is involved with deep learning techniques (starting with e.g. https://github.com/alexgkendall/caffe-posenet).

You may also want to watch out for this conference http://cvpr2017.thecvf.com/ scheduled for the end of this month and is usually a good source of cutting edge research...

quantumwannabe• 9 years ago

I find it funny that Apple is just now releasing an AR SDK as when the iPhone first came out there were a lot of AR apps available. Most of them were pretty boring, just games like shooting spaceships or POI explorers, but they didn't look that much more primitive than the apps demoed today. I wonder why they didn't jump on AR back then.

MBCook6• 9 years ago

I remember playing with those on my 3G or my 4. It was pretty impressive what people were able to do with those little phones.

I'm sure part of it is just the ability to do useful things, part of it may simply be that they have such had room on the chips that they can do it without draining battery life horribly. I remember those old apps clearly used a lot of processing power.

Apple enters the augmented reality fray with ARKit for iOS

Related Stories