Mobile Cesium VR and AR - convert HTML5 device orientation into cesium camera controls?

I'm trying to control the position and orientation of the camera with the HTML5 position (GPS) and HTML5 device orientation. This will allow us to create mobile VR and AR applications using Cesium.

Maybe someone can give me a hint about what I'm doing wrong. Note that the GPS position is stored in _position.

function onDeviceOrientationChanged(eventData) {
    var x = eventData.beta;
    var y = eventData.gamma;
    var z = eventData.alpha;
    
    // Convert HTML5 orientation data into Quaternion format
    var euler = new THREE.Euler( x, y, z, 'XYZ' );
    var q = new THREE.Quaternion();
    q.setFromEuler(euler);

    // Convert HTML5 GPS position into Cartesian format
    var home = Cesium.Cartographic.fromDegrees(_position.coords.longitude, _position.coords.latitude, 5.0);
    var destination = Cesium.Ellipsoid.WGS84.cartographicToCartesian(home);

    var cart3 = Cesium.Cartesian3.fromElements(q.x,q.y,q.z);
    var cross = Cesium.Cartesian3.cross(destination, cart3);

    // Camera.lookAt(eye, target, up)
    myCesium.scene.camera.controller.lookAt(
            destination,
            cross,
            Cesium.Cartesian3.UNIT_Z);
}

Any help or thoughts appreciated.

It looks like you already have the orientation so you can set it directly without using the lookAt function. Try:

var xRotation = Cesium.Matrix3.fromRotationX(x);

var yRotation = Cesium.Matrix3.fromRotationY(y);

var zRotation = Cesium.Matrix3.fromRotationZ(z);

var rotation = Cesium.Matrix3.multiply(xRotation, Cesium.Matrix3.multiply(yRotation, zRotation));

camera.position = destination;

camera.right = Matrix3.getRow(rotation, 0);

camera.up = Matrix3.getRow(rotation, 1);

camera.direction = Cartesian3.negate(Matrix3.getRow(rotation, 2));

Regards,

Dan

Dan,

That certainly helped! I played with the multiplications and the negating (not my area of expertise) and I at least have it rendering. But it’s not quite right. Hoping you can take a quick peek at it if you have a smartphone.

If you simply copy the below code into an html file (mobile.html?), and drop it next to a Cesium directory (version b28) and serve it up to a Smartphone (over Wi-Fi in my case), it should take about 10 seconds to load and then start rendering tiles over San Diego. As you move the phone, it should change the Cesium camera angle accordingly.

In theory, it should work on iPhones, but I have not tested it on one. I’m using a Nexus 4 running Android 4.4 and a Chrome browser. All the code is simple Cesium and HTML5. The function that needs help is called onDeviceOrientationChanged.

Any help would be greatly appreciated. I intend to publish the code once we figure this out so everyone will benefit.

Pat.

<title>Cesium Mobile</title>

<script src="Cesium/Cesium.js"></script>

<script>

    var widget;

    function load() {

        "use strict";

        var cesiumContainer = document.getElementById('cesiumContainer');

        widget = new Cesium.CesiumWidget(cesiumContainer, {

            imageryProvider: new Cesium.OpenStreetMapImageryProvider(),

            terrainProvider: new Cesium.CesiumTerrainProvider({

                url: 'http://cesiumjs.org/smallterrain',

                credit: 'Terrain data courtesy Analytical Graphics, Inc.'

            }),

            contextOptions: {

                webgl: {

                    alpha: true

                }

            }

        });

        // MANUALLY SET OWN POSITION

        var position={};

        position.coords={};

        position.coords.latitude=32.71; 

        position.coords.longitude=-117.16;

        position.coords.altitude=100;

        setCameraPosition(position);

        // ENABLE THIS TO USE GPS SENSOR TO SET OWN POSITION AUTOMATICALLY

        // BROWSER WILL PROMPT YOU FOR PERMISSION

    //    if (navigator.geolocation) {

    //        navigator.geolocation.watchPosition(setCameraPosition);

    //    }

        // THIS ALLOWS YOU TO USE THE PHONE TO CONTROL THE CESIUM CAMERA VIEW

        if (window.DeviceOrientationEvent) {

            window.addEventListener('deviceorientation', onDeviceOrientationChanged, false);

        }

        window.addEventListener('resize', onResize, false);

        window.setTimeout(onResize, 60);

    }

    var onResize = function() {

        onResizeScene(widget.canvas, widget.scene);

    };

    // Resize handler

    var onResizeScene = function(canvas, scene) {

        var width = canvas.clientWidth;

        var height = canvas.clientHeight;

        if (canvas.width === width && canvas.height === height) {

            return;

        }

        canvas.width = width;

        canvas.height = height;

        scene.camera.frustum.aspectRatio = width / height;

    };

    // =====================================================================

    // POSITION SENSOR

    // =====================================================================

    function setCameraPosition(position) {

    //    console.log("flyTo(" + position.coords.latitude + ", " + position.coords.longitude + ")");

        var home = Cesium.Cartographic.fromDegrees(position.coords.longitude, position.coords.latitude, position.coords.altitude);

        var destination = Cesium.Ellipsoid.WGS84.cartographicToCartesian(home);

        widget.scene.camera.position = destination;

    }

    // =====================================================================

    // ORIENTATION SENSOR

    // http://www.w3.org/TR/orientation-event/

    // x = beta [-180, 180] (aka pitch)

    // y = gamma [-90, 90] (aka roll)

    // z = alpha [0, 360] (aka yaw)

    // =====================================================================

    function onDeviceOrientationChanged(eventData) {

        var x = eventData.beta;

        var y = eventData.gamma;

        var z = eventData.alpha;

        x = Cesium.Math.toRadians(x);

        y = Cesium.Math.toRadians(y);

        z = Cesium.Math.toRadians(180 - z);

        var xRotation = Cesium.Matrix3.fromRotationX(x);

        var yRotation = Cesium.Matrix3.fromRotationY(y);

        var zRotation = Cesium.Matrix3.fromRotationZ(z);

        var rotation = Cesium.Matrix3.multiply(yRotation, Cesium.Matrix3.multiply(xRotation, zRotation));

        Cesium.Quaternion.normalize(rotation, rotation);

        widget.scene.camera.right = Cesium.Matrix3.getRow(rotation, 0);

        widget.scene.camera.up = Cesium.Cartesian3.negate(Cesium.Matrix3.getRow(rotation, 1));

        widget.scene.camera.direction = Cesium.Cartesian3.negate(Cesium.Matrix3.getRow(rotation, 2));

    }

</script>

<style> 

  @import url(Cesium/Widgets/CesiumWidget/CesiumWidget.css);

  #cesiumContainer {

    position: absolute;

    top: 0;

    left: 0;

    height: 100%;

    width: 100%;

    margin: 0;

    overflow: hidden;

    padding: 0;

    font-family: sans-serif;

  }

  body {

    padding: 0;

    margin: 0;

    overflow: hidden;

  }

</style>
<div id="cesiumContainer"></div>

Were you eventually able to make it work Pat? The current code that takes care of device orientation in VR mode is not working as expected. Maybe your insight could put it straight.

Unfortunately not. Really need a Quaternion/Euler kind of person on this.

Doesn't the Cesium cardboard example show how to convert a position and orientation to setting the camera?

I find the code to be pretty obfuscated at best. The present DeviceOrientationCameraController.js code is buggy and I have opened an issue: https://github.com/AnalyticalGraphicsInc/cesium/issues/4135

I also found a good primer to understanding quaternions and matrix in this light from this document: https://dev.opera.com/articles/w3c-device-orientation-usage/

I hope to try it out over the next few days. I shall keep posting wherever I get at.

I've modified very slightly the Cardboard example to respond to a VR device orientation. However since I do not have a VR device I'm using Chrome with the WebVR API Emulation at:https://chrome.google.com/webstore/detail/webvr-api-emulation/gbdnpaebafagioggnhkacnaaahpiefil?hl=en

Put this example at: https://github.com/workingDog/webvrcesium/blob/master/jscriptonly.html
in the Cesium/Apps folder, launch Cesium (node server.js) and point your browser to
to http://localhost:8080/Apps/jscriptonly.html

In Chrome go to

      View->Developer->Developer Tools

then in the "developer tools" top bar, click on the ">>" and select "WebVR"

The WebVR emulation will then be displayed.
Type "e" at the keyboard to go into "rotation" mode and use the mouse to turn the rotation widget.

This is just to say that the orientations seem to be ok for me.

I tried these samples, but I'm getting error at widget.scene.camera.right = Cesium.Matrix3.getRow(rotation, 0, new Cesium.Matrix3());

Uncaught TypeError: Cannot set property '0' of undefined

Does this works in Cesium1.30? Anyone have latest code on this?