So one of the things we believe strongly at JuiceBox is the need to be building our games to be ubiquitous. With Android grabbing the biggest market share gains in 2012 we see the market for application developers as increasingly fragmented and increasingly frustrating. As if targeting two platforms wasn’t a challenge enough, the actual spec range of android devices being brought into the market are pretty diverse – for instance while the specs for an IPhone 4 generally refer to one well-known build – the Galaxy S3 has 12 different model variants, some with half the RAM as others. We think in order to compete as an app developer in 2013, you need to be in both app stores (tabling Win 8 for the moment) – and to be in both app stores without doing dual development you need to be either targeting a virtual machine or cross compiling the same code base to both platforms.
For our first title, Honorbound – we’ve selected AIR as our platform with Starling as a rendering framework. A decision that was born both out of familiarity with the product and with Adobe’s continuing emphasis on making the platform a viable one for mobile gaming. Of the solutions out there, it also happens to be one the few with a viable web strategy as well (via Flash Player) – though Unity 3D’s own Flash target is gaining maturity.
While Flash is a very successful and widely used solution for Web Gaming – AIR seems to have faced a bit of friction in gaining inroads into the mobile space. Some of the biggest mobile titles to-date to successfully launch on AIR have been Zynga’s Ruby Blast, Amanita Design’s Machinarium and SpryFox’s Triple Town. While these are all quite successful titles (and fun in their own right) – it hasn’t quite matched the larger adoption of Unity 3D. Ongoing mobile game development on AIR is somewhat controversial – we think this is both because of this slow adoption and a perception that the AIR VM yields inferior performance against other solutions for gaming workloads.
But – being stubborn, I wanted to see the proof so we put together a series of very basic benchmarks for AIR on Starling and Unity 3D and ran them on a series of devices. This post shares the process, and the results.
For this test, we pulled together a small variety of hardware from our test pool. Here’s the devices we used:
- IPhone 5
- IPad 3
- IPad 1
- Galaxy S3 GT-I9300 – http://en.wikipedia.org/wiki/Samsung_Galaxy_S_III#Model_variants
- Galaxy 7 Inch Tab – http://www.samsung.com/us/mobile/galaxy-tab/GT-P3113TSYXAR
We tried to get a decent variety of devices a developer may aim to target, and a few android devices to see if there were any performance characteristics that changed radically between the two platforms.
Our first title is a 2D title, so our benchmarks reflect this. The benchmarks are a variety of flexing pure render throughput and mixing tweens and dummy FLOPs to simulate possible game logic or bone movements/animations. In each benchmark, a sprite is rendered to the screen and it’s texture changes in a defined order from 1 to 10. These are done using the Starling MovieClip class and a Tile Animation script on a plane in Unity 3D. In both examples, a texture atlas is used to source the texture. The dummy FLOPs is a tight loop of multiplying SINs and COSines together – In both platforms, I double checked that these were not optimized out, and I think the results hold that.
- Benchmark 1 – 10 Animating Sprites
- Benchmark 2 – 50 Animating Sprites
- Benchmark 3 – 250 Animating Sprites
- Benchmark 4 – 500 Animating Sprites
- Benchmark 5 – 10 Animating Sprites with random tweened movement
- Benchmark 6 – 50 Animating Sprites with random tweened movement
- Benchmark 7 – 250 Animating Sprites with random tweened movement
- Benchmark 8 – 500 Animating Sprites with random tweened movement
- Benchmark 9 – 100 Animating Sprites with random tweened movement with 25 Dummy Flops per frame
- Benchmark 10 – 100 Animating Sprites with random tweened movement with 100 Dummy Flops per frame
- Benchmark 11 – 100 Animating Sprites with random tweened movement with 225 Dummy Flops per frame
- Benchmark 12 – 100 Animating Sprites with random tweened movement with 625 Dummy Flops per frame
- Benchmark 13 – 100 Animating Sprites with random tweened movement with 400 Dummy Flops per frame
- Benchmark 14 – 100 Animating Sprites with random tweened movement with 1225 Dummy Flops per frame
- Benchmark 15 – 100 Animating Sprites with random tweened movement with 2500 Dummy Flops per frame
In each case, the scene was rendered for 10 seconds, and the FPS score represents the the average over that time frame.
The code for the benchmark is available here:
The code’s pretty scratchy, as benchmark code tends to be. As always, very interested in feedback here – and and if I’ve done something horrific let me know (I consider myself decently strong in Flash and extremely amateur in U3D).
Here’s some prelim results, again Y axis is FPS averaged over 10 seconds of rendering. Also, keep in mind a potential confirmation bias as my current project is in AIR (though I don’t think the results show any :))
NOTE – There’s a labeling error here. The charts below incorrectly refer to a Nexus 7. These results actually match up to our Galaxy 7.
So – a few thoughts on the results, as they were somewhat surprising to me
- For GPU Exclusive Workloads – AIR is on par to slightly better than Unity 3D. Infact on the Galaxy 7 (which has no GPU) – AIR crushes it as the scene gets more complex. We suspect this is PROBABLY attributable to a solid software rasterizer in Stage3D.
- For Mixed Workloads – U3D dominates AIR on all devices. I suspect there’s a few factors here, cross-compilation helps, and I think the render core for U3D better utilizes the multiple cores available in higher end devices. This is somewhere
- For AIR Mobile – We were pretty shocked at how quickly performance degraded when just adding a few dozen FLOPs – really speaks to the need to be vigilant about what your simulation logic is doing every frame – and if possible, move more CPU intensive operations to alchemy.