2D Gaming Mobile Performance – Starling + AIR Vs. Unity 3D

Greasy test devices

Some greasy test devices

So one of the things we believe strongly at JuiceBox is the need to be building our games to be ubiquitous. With Android grabbing the biggest market share gains in 2012 we see the market for application developers as increasingly fragmented and increasingly frustrating. As if targeting two platforms wasn’t a challenge enough, the actual spec range of android devices being brought into the market are pretty diverse – for instance while the specs for an IPhone 4 generally refer to one well-known build – the Galaxy S3 has 12 different model variants, some with half the RAM as others. We think in order to compete as an app developer in 2013, you need to be in both app stores (tabling Win 8 for the moment) – and to be in both app stores without doing dual development you need to be either targeting a virtual machine or cross compiling the same code base to both platforms.

For our first title, Honorbound – we’ve selected AIR as our platform with Starling as a rendering framework. A decision that was born both out of familiarity with the product and with Adobe’s continuing emphasis on making the platform a viable one for mobile gaming. Of the solutions out there, it also happens to be one the few with a viable web strategy as well (via Flash Player) – though Unity 3D’s own Flash target is gaining maturity.

While Flash is a very successful and widely used solution for Web Gaming – AIR seems to have faced a bit of friction in gaining inroads into the mobile space. Some of the biggest mobile titles to-date to successfully launch on AIR have been Zynga’s Ruby Blast, Amanita Design’s Machinarium and SpryFox’s Triple Town. While these are all quite successful titles (and fun in their own right) – it hasn’t quite matched the larger adoption of Unity 3D. Ongoing mobile game development on AIR is somewhat controversial – we think this is both because of this slow adoption and a perception that the AIR VM yields inferior performance against other solutions for gaming workloads.

But – being stubborn, I wanted to see the proof so we put together a series of very basic benchmarks for AIR on Starling and Unity 3D and ran them on a series of devices. This post shares the process, and the results.

The Hardware

For this test, we pulled together a small variety of hardware from our test pool. Here’s the devices we used:

We tried to get a decent variety of devices a developer may aim to target, and a few android devices to see if there were any performance characteristics that changed radically between the two platforms.

The Benchmarks

Our first title is a 2D title, so our benchmarks reflect this. The benchmarks are a variety of flexing pure render throughput and mixing tweens and dummy FLOPs to simulate possible game logic or bone movements/animations. In each benchmark, a sprite is rendered to the screen and it’s texture changes in a defined order from 1 to 10. These are done using the Starling MovieClip class and a Tile Animation script on a plane in Unity 3D. In both examples, a texture atlas is used to source the texture. The dummy FLOPs is a tight loop of multiplying SINs and COSines together – In both platforms, I double checked that these were not optimized out, and I think the results hold that.

  • Benchmark 1 – 10 Animating Sprites
  • Benchmark 2 – 50 Animating Sprites
  • Benchmark 3 – 250 Animating Sprites
  • Benchmark 4 – 500 Animating Sprites
  • Benchmark 5 – 10 Animating Sprites with random tweened movement
  • Benchmark 6 – 50 Animating Sprites with random tweened movement
  • Benchmark 7 – 250 Animating Sprites with random tweened movement
  • Benchmark 8 – 500 Animating Sprites with random tweened movement
  • Benchmark 9 – 100 Animating Sprites with random tweened movement with 25 Dummy Flops per frame
  • Benchmark 10 – 100 Animating Sprites with random tweened movement with 100 Dummy Flops per frame
  • Benchmark 11 – 100 Animating Sprites with random tweened movement with 225 Dummy Flops per frame
  • Benchmark 12 – 100 Animating Sprites with random tweened movement with 625 Dummy Flops per frame
  • Benchmark 13 – 100 Animating Sprites with random tweened movement with 400 Dummy Flops per frame
  • Benchmark 14 – 100 Animating Sprites with random tweened movement with 1225 Dummy Flops per frame
  • Benchmark 15 – 100 Animating Sprites with random tweened movement with 2500 Dummy Flops per frame

In each case, the scene was rendered for 10 seconds, and the FPS score represents the the average over that time frame.

Bench-App in action. It's pretty seizure inducing

Bench-App in action. It’s pretty seizure inducing

The code for the benchmark is available here:

https://github.com/juiceboxgames/client-bench

The code’s pretty scratchy, as benchmark code tends to be. As always, very interested in feedback here – and and if I’ve done something horrific let me know (I consider myself decently strong in Flash and extremely amateur in U3D).

Results

Here’s some prelim results, again Y axis is FPS averaged over 10 seconds of rendering. Also, keep in mind a potential confirmation bias as my current project is in AIR (though I don’t think the results show any :))

NOTE – There’s a labeling error here. The charts below incorrectly refer to a Nexus 7. These results actually match up to our Galaxy 7.

10 Animating Sprites

10 Animating Sprites

50 Animating Sprites

50 Animating Sprites

250 Animating Sprites

250 Animating Sprites

500 Animating Sprites

500 Animating Sprites

10 Animating Sprites with random tweened movement

10 Animating Sprites with random tweened movement

50 Animating Sprites with random tweened movement

50 Animating Sprites with random tweened movement

250 Animating Sprites with random tweened movement

250 Animating Sprites with random tweened movement

500 Animating Sprites with random tweened movement

500 Animating Sprites with random tweened movement

100 Animating Sprites with random tweened movement with 25 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 25 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 100 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 100 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 225 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 225 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 625 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 625 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 400 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 400 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 1225 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 1225 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 2500 Dummy Flops per frame

100 Animating Sprites with random tweened movement with 2500 Dummy Flops per frame

Conclusions

So – a few thoughts on the results, as they were somewhat surprising to me

  • For GPU Exclusive Workloads – AIR is on par to slightly better than Unity 3D. Infact on the Galaxy 7 (which has no GPU) – AIR crushes it as the scene gets more complex. We suspect this is PROBABLY attributable to a solid software rasterizer in Stage3D.
  • For Mixed Workloads – U3D dominates AIR on all devices. I suspect there’s a few factors here, cross-compilation helps, and I think the render core for U3D better utilizes the multiple cores available in higher end devices. This is somewhere
  • For AIR Mobile – We were pretty shocked at how quickly performance degraded when just adding a few dozen FLOPs – really speaks to the need to be vigilant about what your simulation logic is doing every frame – and if possible, move more CPU intensive operations to alchemy.

Comments

  1. A few thoughts on your Unity project…

    – You should cache your transform lookups:

    Transform t = myGameObject.transform;

    This is because, under the hood, transform is a getter that does a GetComponent() (or something like that) and you don’t want to pay that cost every time through the Update() function.

    – Your material uses the “Diffuse” shader. This means you’re using dynamic lighting, which I don’t think was your intent (does Air even support dynamic lights?) Instead, you should probably use an Unlit shader (i.e. “Unlit > Texture”) which gives the same effect of blotting a texture onto a plane, only without paying the cost of lighting.

    – You’re using a plane with 121 triangles. Unless you need to do per-vertex lighting, you should use a simple quad / 2-triangle plane instead.

    – Your animation technique doesn’t support using shared materials, which breaks dynamic batching of draw calls (see http://docs.unity3d.com/Documentation/Manual/DrawCallBatching.html) which is probably the biggest performance killer on the iPad 1.

    — Rob

  2. “if possible, move more CPU intensive operations to alchemy” as if it would help at all ) I myself and also somebody else a year before tested alchemy performance vs pure as3 on mobile and found the ratio to be in favor of as3. native extensions might help, but not alchemy.

  3. The interesting thing is people can debate on the implementation of the code and whatever else but there is a lot of misinformation out there when it comes to AIR performance. People keep comparing apples to oranges, all over the internet. GPU accelerated content keeps getting compared to software rendered flash/AIR content. So this is a welcomed comparison of apples to apples using GPU acceleration. Adobe AIR is clearly holding its own being that it is a VM and it is a viable solution for high performance cross platform game development. Why jump ship to JS or Unity when we (AS3 devs) have all the tools, generous community, and 10+ years of experience on a platform that has seen some really great advancements recently? Only real reason I can come up with is Adobe’s lack of public commitment to the platform. Which is clearly due to market pressure and trying to please stock holders. But I will be sticking with AIR, it just works!

    “Men lie, woman lie, programmers to, but numbers never do” – :)

    I am building GameBuilder Studio on top of Adobe AIR and it works great! (a future 2D Unity type tool for AIR) http://gamebuilderstudio.com

  4. thanks, very informative. i wish someone do similar test for 3d using away 3d or any other as3 engine with unity3d. some simple cubes or skeleton deformations etc…
    any ways, huge thanks

  5. Can you also compare with nativa application?

  6. KanedaFr says:

    now just compare results with Haxe NME/OpenFL …

  7. You should test with OpenFL and NME too

  8. Anonymous says:

    Very interesting post thanks for getting the time to do this. Its been no secret that Actionscript performance is an issue for Flash for being the best solution for 2D games cross platform although even at the minimum FLOP value that is 625 and the second test is 10,000 cos and sin function calls which sounds like a big jump please excuse me in case I missed something. But I wonder if its just the sin and cos function calls which are slow or any two function calls, which is a much bigger issue?

  9. Hermes David Montes de Oca Segovia says:

    i have also made several benchmarks with unity vs air + starling and for pure gpu stuff Stage3D is fast really fast, but for severe computation stuff ActionScript 3 sucks, thats why for 2D Unity may be slower but once you involve physics, unity goes on unharmed like nothing happened and air crawls to really bad performance.

    flash port of Box2D is basically unusable on mobile with really poor performance, nape is much superior but you have to be very careful with your code to avoid bottlenecks, also if you are not careful with air + starling you can sky rocket the memory usage to unhealthy levels very fast

    i love air + starling, its very simple and minimalistic, but to get decent performance on most devices on slightly complex projects you have to be an optimisation guru..

    and unity may be an overkill for most 2D stuff, but in the end if you can deal with it, its a much more stable platform to develop on, less fear to break stuff hahahaha

  10. I try to edit the air version,and run on ipad2, found 2 problem of your code:
    1) the first benchmark only~52 fps,beacuse start the air need ~1second init
    2) benchmark15 so slow because the program hvn’t release the object from benchmark 1~14.
    When I change the order become 15,9,8,5,4,1 ,then:
    benchmark 15 fps 58.7
    benchmark 1 fps 30.7

Share your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 47 other followers