Unity ARKit By Example: Part 4

Exploring image detection and tracking (and object tracking).

This article is part of a series starting with Unity ARKit By Example: Part 1.

Image Tracking

In this example we will use the imaging tracking API from Unity-ARKit-Plugin; we will use its ability to detect and track images in the physical world. Tracking was added in ARKit 2.0 and released with iOS 12.

In particular, when the target image is in the camera viewfinder, the application will place and keep a 30 cm cube over the image.

Contrary to the previous examples, we will keep all the assets (there a number of them) for the application in a separate folder; Assets > Examples > ImageTracking. We start by saving the Hello World scene as ImageAnchor into the application folder.

We then remove the Cube, Canvas, and EventSystem GameObjects from the scene.

We drag the the target image, e.g., jpg, into the application folder.

note: On many of my sample projects I use cat images; I just find them fascinating animals.

We use the Unity menu, Assets > Create > UnityARKitPlugin > ARReferenceImage, to create an image reference; saving in the application folder, e.g., with name cat. The reference image, in addition to defining the Image Texture (the target image of the cat), it defines the Physical Size, e.g., 0.3 is 30 cm, in the physical world.

We use the Unity menu, Assets > Create > UnityARKitPlugin > ARReferenceImagesSet, to create an a collection of reference images; saving in the application folder, e.g., with name ImageAnchor (for lack of a better name). In this case, it has a Size of 1 and with Element 0 referring to the image reference we created.

To enable the application to track the images (actually one of them) defined in the images reference set, we add it to the ARCameraManager (GameObject) > Unity AR Camera Manager (component) > Image Tracking > Detection Images (property). We also ensure the property, Maximum Number of Tracked Images, is set to 1.

As with the previous article, we need to create the 30 cm Cube prefab in the application folder.

Finally, we need to create an empty GameObject, e.g., ImageAnchor, and a script, ImageAnchor, in the application folder. We add the new script as a component to the new GameObject.

Observations:

• We subscribe to three Unity-ARKit-Plugin image anchor events; added, updated, and removed
• For the most part, this script is self-explanatory; except it turns out the removed event is never fired. When the image is out of view of the camera viewfinder, an updated event with the isTracked property equal to false is triggered

In order for the script to operate, we need to assign the image reference (cat) to the public referenceImage property. Likewise we need to assign the Cube prefab to the public anchorPrefab property.

The complete set of objects in the application’s folder are:

note: In hindsight, it might have been more sensible to use different names for the different types of objects.

We now follow the instructions in the section The Build Process of the article Unity ARKit By Example: Part 1 to build and load the application to an iOS device. A couple of things to keep in mind:

• We can leave the bundle identifier as we had before
• Make sure to only include the ImageAnchor scene in the build

Now running the application on an iOS device, we can see the application working.

note: We have to build and load the application to an iOS device to test the application as ARKitRemote does not support image detection and tracking.

Object Detection

Turns out that object detection is very similar to image tracking; the primary difference is that object detection makes use of a point cloud as opposed to an image as the target.

The following video does a great job of explaining the general workflow (the additional complexity is generating the point cloud target); and given the image tracking example above one can mostly build the equivalent object detection example.

For example, I followed the instructions to create a point cloud of a dumbbell and then made the application create a cube when it detects it.

The following are the assets we end up with:

• Cube: The prefab to generate
• Dumbbell: The point cloud of the dumbbell
• DumbbellDetector: The script (below)
• ObjectAnchor: The scene
• RefObj: The reference object
• RefObjsSet: The reference objects set

Observations:

• This code is very similar to the image tracking example; basically we are responding to object instead of image events
• The objects are not tracked in the same way images are, e.g., moving the dumbbell around does not move the GameObject
• Found the object detection to be fussy; have to get the object in the same lighting for the detection to work
• For whatever reason, could not get a public ARReferenceObjectAsset to be visible in the component inspector; instead exposed a string (referenceObjectName)

Wrap Up

Looking through the documents and tutorials, there are a number of features that we did not explore:

• Plane detection (horizontal and vertical)
• ARWorldMap; persistence and/or sharing of AR experiences
• Face detection and tracking

Figure you can explore these on your own.