Thomas und Philipp haben sich in der Hackweek mit Augmented Reality für Webapplikation auseinandergesetzt. Ihr Ziel war herauszufinden was heute schon möglich ist und wohin die Reise geht. 

Welche Browser unterstützen den W3C WebXR Standard? Sind Technologien wie [A-Frame](https://aframe.io/) und [ThreeJS]( https://threejs.org/) schon produktiv einsetzbar? Wo finde ich geeignete Tutorials und Dokumentationen?

Diese und weitere Fragen werden in ihrem Blogeintrag erläutert und können [hier nachgelesen](https://lambda-it.ch/blog/augmented_reality_webapps) werden.


medium_image_detection_ducky.png

small_image_detection_ducky.png

thumbnail_image_detection_ducky.png

image_detection_ducky.png

medium_philipp.jpg

small_philipp.jpg

thumbnail_philipp.jpg

philipp.jpg

Software Entwickler

Regelmässig erstellen wir Reviews zu Projekten in den verschiedensten Entwicklungsphasen. Ob ein Entwicklungskonzept oder der finale Sourcecode - unsere Reviews sind verständlich und zeigen auf, wo Probleme entstehen könnten. Detaillierte Lösungsansätze für die potentiellen Probleme runden die Reviews ab.

lambda_illustrationen_v0101_lambda_coaching_ohne_bg.svg

Review

Wir lieben es, unser Wissen weiterzugeben. Wir sind aktive Blogger, Speaker und «Open Sourcer». Zudem geben wir unser Wissen auch gerne «gebündelt» weiter - an Einzelpersonen, Teams oder an ganze Firmen. Die folgenden «Unterrichtsformen» haben sich für uns bewährt:

- Workshops
- Inhouse Schulungen
- Hackathon

Coaching

Wir sind Teil der Uphill Academy, ein Zusammenschluss von drei Berner Firmen sowie nationalen und internationalen Coaches, um Trainings und Workshops zu organisieren. Wir engagieren uns aktiv in der Organisation und der Durchführung von Trainings und Workshops.

Uphill Academy

Mit unserem Wissen und unserem internationalen Netzwerk finden wir die optimale Lösung für Ihre Fragen

Know-How für die Zukunft

Augmented Reality für Webapplikation – ein Erfahrungsbericht

![pokemon_go.png](https://imgs.xkcd.com/comics/pokemon_go.png)  

However, before getting into AR you should be aware that it can be more complex than first expected.
It is very tempting to jump right in, because there are many frameworks for AR and different examples from blogs and other learning materials.
AR can become complex very quickly and it is recommended to brush up your math skills. Specifically, a good knowhow of vectors and matrices and how to work with a virtual environment is recommended.

## Preparation

There is already quite a range of web-frameworks that try to facilitate creating an AR-project in a browser. We used the following selection of frameworks to get a first impression:
-	A-Frame
-	Three.js
-	Unity
-	React360
-	Babylon.js
-	Argon.js

After putting them into practice, we concluded that some of them are not compatible with our workflow or were not mature enough.
We decided to stick with A-Frame and Three.js as they were a solid choice to get promising results. Since A-Frame is built on top of Three.js they both use the same concepts.

**Three.js** is an extensive library which is used as the base for a lot of AR Frameworks. It provides the core functionality to create an interactive 3D scene in a browser. 
Three.js abstracts WebGL which is a very low-level system that only draws points, lines and triangles. It handles aspects like scenes, lights, shadows, materials, textures, 3D math and the core functionality that you'd have to write yourself if you were to use WebGL directly.

While using Three.js, it became clear that this library is still on a very low level. The loading of 3D models and their placement must be configured meticulously.

In addition, Typescript integration has proved to be complicated and only partially successful. For these reasons, we decided to primarily focus on A-FRAME.

**A-FRAME** immediately caught our attention because it uses web components and can be integrated with other frameworks like Angular and React.

We first integrated some WebXR AR examples into an A-Frame component. Unlike Three.js, A-Frame simplifies the more tedious configurations and allows for a faster and more pleasant initialization process.

### Browser and Device Compatibility

The apparent problem for AR is the poor browser compatibility with the established WebXR standard. The only browsers which support most of the features require modern iOS and Android devices. We were not able to find an exact list of supported devices or required functionality. Luckily, Mozilla has created a Firefox based browser for iOS which tries to implement most WebXR features by emulating them. The App is freely available on Applestore: [WebXR Viewer]( https://apps.apple.com/us/app/webxr-viewer/id1295998056).

### Debugging

We quickly realized that it’s hard to debug a scene on the webxr-viewer as it does not allow remote debugging. As a solution to this we used [Vorlon]( http://www.vorlonjs.io/). With Vorlon it is possible to send console logs and errors to a server. These log entries can then be viewed with a browser on a different device. While it worked fine for quick debug sessions, it required a manual reload for each new build.

### Getting started

A good source to start with AR, is [https://immersive-web.github.io/webxr-samples/](https://immersive-web.github.io/webxr-samples/) and if WebXR Viewer is being used [https://webxr-iOS.webxrexperiments.com/](https://webxr-iOS.webxrexperiments.com/).
By using these examples, we were able to get a first grasp of what is possible and how AR can be applied in web applications.

After experimenting with the basic features of A-Frame, we tried to apply our existing knowledge from different projects. One framework we recently used in other Projects was D3. This library helps to create interactive charts, by allowing the manipulation of HTML Documents based on data. As A-Frame works with Web Components, the integration with D3 was rather simple. An example on how to integrate D3 with A-Frame:

**Scene**
```html

<a-scene>
    <a-entity id="default_angle" camera position="0 0 0" look-controls wasd-controls>
        <a-entity
            cursor="fuse: true; fuseTimeout: 500"
            material="color: black; shader: flat"
            geometry="primitive: ring; radiusInner: 0.02; radiusOuter: 0.03"
        ></a-entity>
    </a-entity>
</a-scene>
```

**D3 Example**
```typescript
// A typical D3 selector used to create a bar chart with A-Frame box components
// using some arbitrary data and settings
const boxData = d3.select('a-scene').selectAll('a-box').data(data)
boxData
    .enter()
    .append('a-box') 
    // At this point it is possible to set the different A-Frame box element attributes
    .attr('color', (d, i) => colorScale(i))
    .attr('width', barWidth)
    .attr('depth', 1)
    .attr('height', d => d.value / 100)
    .attr('position', (d, i) => `${i * (barWidth + 0.1) - (data.length * barWidth) / 2} 1 -8`);
```

![ar_d3_charts.png](https://lambda-it.ch/api/uploads/ar_d3_charts_032efbe739.png)

Going forward we focused on the possible options to interact with the scene. In the current standard there is only one way to trigger an interaction event which is a hit test.

With this event it is possible to obtain a point on a vertical or horizontal surface. The orientation and position are delivered in a Vector4. Once we had the possibility to obtain a specific point on a surface, we tried to load a custom model and display it at this position.

One of our goals for these experiments was to find out if it is possible to interact with the model via touch events. But we had to give up on this because time issues, so we focused on experimental features to find out which upcoming features there are.

As previously mentioned, there are several different implementations of experimental features depending on which browser and device is being used. Since we were using the WebXR viewer for iOS we tried the experimental image detection feature which looked promising.

The image detection was surprisingly fast and reliable. It can be used to bind an anchor point for a scene to a specific image. These images can be everything from a QR code to something else. This way it is possible to have different images with their associated anchor points which will get updated once the image position changed.

**Example of an image detection event callback**
``` typescript
// Run experimental image detection
this.session
    .nonStandard_activateDetectionImage(someImageTagName)
    .then(anchor => {
        ...
    });
```
![image_detection_ducky.png](https://lambda-it.ch/api/uploads/image_detection_ducky_e197d30f41.png)

# Conclusion

* There are currently many different implementations of the WebXR standard because the specifications are still changing rapidly.
* Advanced tasks such as image or depth detection are not yet possible as this is not yet specified.
* Not ready to be used in production, because the majority of devices do not support it. 
* It is not possible to access the video output directly, which means that it is not possible to analyze the image material further and obtain additional information.

Fokus - D3.js

Aktuelles zu D3.js

Augmented Reality für Webapplikation – ein Erfahrungsbericht

A brief journey in Augmented Reality for Web Applications

Fokussiert auf D3.js:

Thomas Schneiter

Software Entwickler