Delivering 360 Video from Device Assets

In my previous post I wrote about the 360 video viewer plugin I am experimenting with using Flutter, a cross-platform development framework, and a practice clip I edited within After Effects. While there is plenty of support for 360 images within Flutter, I was surprised to only find a single plugin to support 360 video. The plugin uses a Google SDK for Cardboard VR. Although Google has since released a newer SDK for cardboard, I only need limited functionality – to view a 360 video, so I am hoping this plugin will suffice for me.

The plugin uses Google VR SDK

In my previous experiment, there was a fairly serious limitation, in that I had to call the 360 clip via a URL.

The plugin only supports 360 video via a URL

360 video files can be quite big, the short ones I have been experimenting with are around 10Mb and even on a fast wifi connection there was a significant delay while the video is downloaded in its entirety before the 360 video player starts playing. This would provide a poor user experience in terms of the wait. I could potentially add some gameplay functionality into this waiting time; but it’s still a big ask – to download large files; especially if the user is outdoors and using mobile data, which may be limited or even expensive to the user.

I spent some time analysing the plugin code as well as the underlying Google VR SDK to see how much work would be involved should I want to implement, myself, a feature to allow local, on-device files to load, rather than via a url. I found that this would not be a trivial project, so I sought an alternative approach.

I could not expect my users to access large video files via their mobile data so I had to think up a new solution. Rather than develop new functionality for the plugin or build my own implementation of a VR SDK plugin, I decided to experiment with setting up a mini http server within my app and thus serving the 360 video asset from a local server url.

Dart, the programming language which underpins the Flutter framework includes built in HTTP server functionality via its HTTPServer class.

Dart’s HttpServer class

Using this approach I would be able to create a lightweight http server within my app from which to serve the 360 videos and satisify the 360 video viewer’s play-from-URL requirement. I even found an existing Flutter plugin which uses this technique and maps http requests directly to a specific assets folder.

Flutter plugin implementing the http server class for a Flutter app scenario

I built a test app combining both the local assets server and the 360 viewer along with my test 360 clip and the video now loads much quicker using the locally served file. This means I can now consider either bundling all my app’s 360 content into the app build itself, or downloading them to local filespace as and when the user has wifi access. Thus, overcoming the problem of slowness and using a user’s mobile data for accessing these clips from the web.

The 360 clip now loads instantly when the button is clicked.

Flutter Adoption Retrospection

Although I’ve built several mobile apps previously using web technologies Cordova and Ionic, I was never fully satisfied with the quality of the end products. In most cases, this was because of the un-native ‘look’ of the web based ‘hybrid’ apps. I recall experiencing much frustration using native plugins too.

After my web-mobile experiences, I spent a fair amount of time learning Kotlin. Although I really enjoyed the language itself, I was surprised at how long it would take to build basic functionality like an interactive user interface. I was also concerned that this approach would mean either focussing purely on Android or mastering Swift, the language behind IOS apps as well as building everything twice from the ground up.

As I was using Unity, a cross-platform game engine, for some AR and VR experimentation, I briefly considered this toolkit for general mobile apps too. However, at that time, I was not happy with the UI (user interface) kits available.

I eventually decided to use Google’s cross-platform UI toolkit Flutter for a a project to build an educational app for GCSE students. The toolkit was still relatively new but I appreciated the cross-platform features – one codeset which could build to both native Android and IOS and even web, Mac and soon Windows. The programming language, Dart, which is used for Flutter was easy to adopt; quite similar to Kotlin, which I had been learning previously. Flutter comes with full support for Material design a visual design framework for app UI, so it didn’t have the problems I’d encountered with Unity in that respect.

All in all, my first Flutter experience was highly enjoyable and rewarding. The most striking part was just how fast I could pick up a completely new framework and language and start pushing iterations of my app to my test devices. The hot reload feature which allows developers to see their coded functionality immediately on a physical device really speeds everything up.

After the project was delivered to the client, I took part in a Flutter competition organised by Google, to build a creative clock interface. There was much buzz about this competition on social media and a true sense of developer community surrounding the Flutter eco system.

My ‘Metropolis’ inspired Art-Deco Clock built with Flutter

I went on to build further apps with more and more advanced features, integrations with Google Firebase, Machine Learning as well as take part in the annual Flutter Hackathon, at which I formed a team which got through to the final stages with our Retro CB project.

Retro Style CB build during the 2020 48 hour Flutter Hackathon

My latest experiments included an app called AISpy using artificial intelligence (AI) to identify objects aswell as collect data for machine learning training purposes.

Flutter app using Tensorflow applied Machine Learning & Computer Vision

Limitations

There is a highly impressive library of official and community run plugins and modules for Flutter. And, even if I need something new for a project, this would be an opportunity for me to build my own plugin for the community. The area’s in which I have found some limitation, though, are surrounding augmented reality. Particular, the implementation of ARCore Cloud Anchors; so this could prove to be an opportunity for me to work on this functionality, should I require it for my new major project.