Delivering 360 Video from Device Assets

In my previous post I wrote about the 360 video viewer plugin I am experimenting with using Flutter, a cross-platform development framework, and a practice clip I edited within After Effects. While there is plenty of support for 360 images within Flutter, I was surprised to only find a single plugin to support 360 video. The plugin uses a Google SDK for Cardboard VR. Although Google has since released a newer SDK for cardboard, I only need limited functionality – to view a 360 video, so I am hoping this plugin will suffice for me.

The plugin uses Google VR SDK

In my previous experiment, there was a fairly serious limitation, in that I had to call the 360 clip via a URL.

The plugin only supports 360 video via a URL

360 video files can be quite big, the short ones I have been experimenting with are around 10Mb and even on a fast wifi connection there was a significant delay while the video is downloaded in its entirety before the 360 video player starts playing. This would provide a poor user experience in terms of the wait. I could potentially add some gameplay functionality into this waiting time; but it’s still a big ask – to download large files; especially if the user is outdoors and using mobile data, which may be limited or even expensive to the user.

I spent some time analysing the plugin code as well as the underlying Google VR SDK to see how much work would be involved should I want to implement, myself, a feature to allow local, on-device files to load, rather than via a url. I found that this would not be a trivial project, so I sought an alternative approach.

I could not expect my users to access large video files via their mobile data so I had to think up a new solution. Rather than develop new functionality for the plugin or build my own implementation of a VR SDK plugin, I decided to experiment with setting up a mini http server within my app and thus serving the 360 video asset from a local server url.

Dart, the programming language which underpins the Flutter framework includes built in HTTP server functionality via its HTTPServer class.

Dart’s HttpServer class

Using this approach I would be able to create a lightweight http server within my app from which to serve the 360 videos and satisify the 360 video viewer’s play-from-URL requirement. I even found an existing Flutter plugin which uses this technique and maps http requests directly to a specific assets folder.

Flutter plugin implementing the http server class for a Flutter app scenario

I built a test app combining both the local assets server and the 360 viewer along with my test 360 clip and the video now loads much quicker using the locally served file. This means I can now consider either bundling all my app’s 360 content into the app build itself, or downloading them to local filespace as and when the user has wifi access. Thus, overcoming the problem of slowness and using a user’s mobile data for accessing these clips from the web.

The 360 clip now loads instantly when the button is clicked.

Testing 360 Video with Flutter App Development

Following on from my 360 video experimentation in Adobe After Effects I wanted to test out options for embedding my 360 videos within a a mobile app using Flutter.

I found a number of existing packages available for 360 images, however, running 360 video is less common. I found one package called video_player_360 which acts as a plugin for the Google VR SDK for both iOS and Android allowing videos to be played via a remote URL.

I have a couple of reservations about this package. For a start it using a Google plugin which is now archived. Although just for using the 360 video functionality, this should not be a problem. Although, I will be worth me looking at the new SDK which Google has released as a replacement to see if this has sililar video player functionality and maybe I can build a new Flutter plugin using this updated Google VR SDK.

New Google VR SDK for Adnroid and iOS

The second issue I have with this plugin is that it only works with remote URLs. This means that a user must be connected to the internet and given the nature of 360 video, these can be quite big files even for a short clip. In my test I was using a 12MB file hosted on Github. My phone was connected to my office WiFi and it took a good 5 seconds to load up. Once it did load, the video worked nicely on my mobile device (Google Pixel 3). I was able to move the phone about and experience the immersive effect of the video.

Testing 360 Video within a Flutter mobile app

360 Film Experimentation

A feature of my app concept is to use footage from within local towns. I would like to use 360 film if possible, for several reasons.

  1. It will provide a more immersive user experience
  2. It will provide opportunity to hide clues to the game within the virtual environment which are a little more challenging to find
  3. It will generally make for a more unique and innovative feel to the app

There are several areas I need to explore and experiment with in order to test the feasability of this idea.

I would like to explore the possibility of shooting studio footage – for example of character acted scenes, using a green screen which would be integrated into the town footage. This would mean I could reuse the same studio footage in multiple towns and therefore offer the app to a greater number of users. As my theme for the app is Film Noir, I could also use other props and footage such as period vehicles and so on.

I have began to experiment with 360 footage and stock character footage using Adobe Premier and Adobe After Effects.

Inital 360 video footage in colour

I started by opening a short 360 video from a local woodland in Adobe Premier. I then applied some filters to create a black and white affect.

360 Footage with Black & White Filter

Next I took the footage into Adobe After Effects to add rain and a more darker ambience. The rain and darkness will be a common theme throught my ‘Film Noir’ style app.

My next tasks was to test superimposing footage filmed on a green screen. For testing purposes I grabbed a free clip from Pixabay. Using the Keylight plugin provided with After Effects a transparency is created allowing the content to intergrate within the existing scene.

Keylight helps to convert the green screen footage into a transparency
Keylight Transparency

In order to make the 2d footage work within the 360 environment, I had to convert the layer to a 3d layer and use VR tools within After Effects to anchor down the layer within the 360 sphere so that it remains in place even when the viewer navigates around the 360 environment.

Here is the 360 version on Youtube:

Conclusions and Next Steps

I found the experiments with 360 film quite promising. There are a few areas I need to iron out though. The output from After Effects is producing an ill formatted sphere. This can be seen in the Youtube video above. I am not sure why this is happening. The view is fine within After Effects so I think it is a problem within the export settings. The VR controls in After Effects are still quite daunting to me. But I’m pleased I’ve been able to apply some effects like the ‘Noir’ filter; the rain and mist effects as well as bring in the extra character.

My next steps are to iron out the export issues and make another short clip using new 360 footage from my town which I intend to capture this coming weekend. I will also film some test acting scenes using a greenscreen and try getting together a solid example of a clip suitable for use in my app.

I would then like to test the integration of 360 film into a Flutter app. To see what controls are available.

One consideration that I am now aware of, is that the output file sizes are quite big. This could be an issue with a mobile App. I will seek to keep the clips as short as possible and try to loop and use Flutter for the audio rather than the video. This way I can keep the video files as compact as possible. It will be a good idea to look at managing downloading forthcoming videos for the gameplay and removing previous ones, so that the app is not a huge download size and doesn’t become more and more bloated either.