How to use the best VR headset with the simplest cloud API ?

What are we trying to do ?

Our goal was to develop and deploy a VR software, on the Oculus Quest, that uses firebase API to store and process our data. Our needs were a little more complex, but for this technical post we will try to:

  • Connect to Firebase from the Quest
  • Upload an image from the quest
  • Save the image information in Fire-cloud Non SQL DB
  • Use Google Image processing API to analyse the image

Why doing this ?

We love firebase as a cloud platform, it’s easy to use and implement from any client. We also have access to the entire google cloud platform from our firebase setup.

We also love the Quest, arguably the best VR headset so far (August 2020).

For this example we will use Unity 3D as a development plateform, because we also love Unity 3D, but it could have been Android Studio or anything else that could deploy software on the Quest.

The Problem

Firebase is very easy to use with Unity 3D, you can find a lot of tutorial and a very well documented SDK:

https://firebase.google.com/docs/unity/setup

It’s relatively easy (and free) to setup a basic development platform on Firebase, get all the access keys and access it through the firebase SDK in Unity. That was our initial approach and it was working well, from the editor and from and Android device (a Galaxy S9).

But, as soon as we get it on the Quest, in VR, it was impossible to connect to Firebase.

It took us sometime (and a lot of adb logs from the Quest) to realize that

The Firebase Unity / Android SDK will only work on devices with Google Play Service installed and the Quest is not one of them.

The Solution

We did a lot of research and tests on how to use Firebase/Google cloud on the Quest and finally end up redeveloping out own Unity/C# SDK using the Firebase and Google Cloud REST API access.

A – External Tools in Unity 3D

We used a few assets and external tools in order to achieve this:

  • Best HTTP/2 from the assets store:

https://assetstore.unity.com/packages/tools/network/best-http-2-155981

You can use any HTTP library in Unity, for this example we need a library that allow use to simply change HTTP headers, create any HTTP request method (GET/POST/PUT/PATCH/DELETE) and embed our data (JSON and RAW data). Our favorite for this is is Best HTTP/2

  • JOSE JWT library

https://github.com/dvsekhvalnov/jose-jwt

We do need to create and request JWT tokens in order to use the google REST API. Our favorite library for this is JOSE-JWT and we installed it using NUGET in Visual Studio and doing a copy of the dll in the Unity Project.

  • Bouncy Castle C# encryption library

http://www.bouncycastle.org/csharp/

One of our main challenge dealing with google public and private key in Unity / C# was the encryption format. Google is using a OpenSLL/PEM format and that we need to XML/X509 Certificate/XML format for the asymmetric encryption. The solution was to use Bouncy Castle for this in order to use JOSE-JWT to request the Bearer Authentication token.

We will compile a specific post to follow on this topic of Cloud Authentication / Encryption

  • JSON Serializer / Deserializer

https://assetstore.unity.com/packages/tools/input-management/json-net-for-unity-11347

All models in Firebase / Google cloud are using a JSON format, we basically. need a Serializer/Deserializer HTTP request data in C# objects. Newtonsoft Json.Net is perfect for this

B- The Security Challenge

This was the biggest challenge of all. How to connect from C#, not using firebase Unity SDK, but the REST API.

1- Create and download a service account: How to use a Firebase service account, download the keys and certificates.

To generate a private key file for your service account

  1. In the Firebase console, open Settings > Service Accounts.
  2. Click Generate New Private Key, then confirm by clicking Generate Key.
  3. Securely store the JSON file containing the key.

We use the following C# / Model in order to use this JSON key file

[Serializable]
public class GoogleKey
{
    [SerializeField]
    [JsonProperty("type")]
    public string type { get; set; }

    [SerializeField]
    [JsonProperty("project_id")]
    public string project_id { get; set; }

    [SerializeField]
    [JsonProperty("private_key_id")]
    public string private_key_id { get; set; }

    [SerializeField]
    [JsonProperty("private_key")]
    public string private_key { get; set; }

    [SerializeField]
    [JsonProperty("client_email")]
    public string client_email { get; set; }

    [SerializeField]
    [JsonProperty("auth_uri")]
    public string auth_uri { get; set; }

    [SerializeField]
    [JsonProperty("token_uri")]
    public string token_uri { get; set; }

    [SerializeField]
    [JsonProperty("auth_provider_x509_cert_url")]
    public string auth_provider_x509_cert_url { get; set; }

    [SerializeField]
    [JsonProperty("client_x509_cert_url")]
    public string client_x509_cert_url { get; set; }
}

2- Use this service account to generate a JWT request and retrieve the JWT bearer: How to deal with the PEM/OpenSLL public / private key in Unity C# version 4.X. How to generate the JWT token and create an HTTP request in Unity

private void RequestJWTBearerToken(Action<string> callback)
    {
        RsaPrivateCrtKeyParameters _rsaParams;
        object _rsaParamsLocker = new object();

        StreamReader sr = new StreamReader(GenerateStreamFromString(firebaseAuthData.currentGoogleKey.private_key.Replace(@"\n", "\n")));
        Org.BouncyCastle.OpenSsl.PemReader pr = new Org.BouncyCastle.OpenSsl.PemReader(sr);
        _rsaParams = (RsaPrivateCrtKeyParameters)pr.ReadObject();

        Dictionary<string, object> payload = new Dictionary<string, object>
            {
                { "iss", JWT_ACCOUNT},
                { "sub", JWT_ACCOUNT},
                { "scope", JWT_SCOPE},
                { "aud", GOOGLE_OATH2_POST_URL},
                { "iat", secondsSinceEpoch(DateTime.UtcNow)},
                { "exp", secondsSinceEpoch(DateTime.UtcNow.AddSeconds(3600)) }
            };


        string token = "";

        try
        {
            token = Jose.JWT.Encode(payload, Org.BouncyCastle.Security.DotNetUtilities.ToRSA(_rsaParams), JwsAlgorithm.RS256);
        } catch (Exception e)
        {
            Debug.Log("ERROR THAT " + e.Message + " / " + e.StackTrace);
        }

        HTTPRequest request2 = new HTTPRequest(new Uri(GOOGLE_OATH2_POST_URL), HTTPMethods.Post, (uri, response) =>
        {

            callback(response.DataAsText);

        });

        request2.AddField("grant_type", GOOGLE_OATH2_POST_PARM);
        request2.AddField("assertion", token);
        request2.Send();
    }

3- Use the Authentication bearer in all our HTTP request

    HTTPRequest request = CreateRequest<string>(path, HTTPMethods.Get, (res) =>
    {
        if ((res != null) && (res.Data != null)) {
            callback(res.Data);

        } else if ((res != null) && (res.Error != null))
        {
            callback("ERROR " + res.Error);
        } else 
        {
            callback("ERROR (unknow) on GET");
        }

    });

    FirebaseAuthManager.Instance.GetJWTBearer((res) => {

            request.AddHeader("Authorization", "Bearer " + res.accessToken);
            request.Send();

    })

admin

This is a paragraph.It is justify aligned. It gets really mad when people associate it with Justin Timberlake. Typically, justified is pretty straight laced. It likes everything to be in its place and not all cattywampus like the rest of the aligns. I am not saying that makes it better than the rest of the aligns, but it does tend to put off more of an elitist attitude.

Categories

Join Our Community
Be part of what’s next in art, design, development, web culture & more.

Recent Comments

    Archives

    Categories