I haven’t been a full-time iPhone developer for a long time (who has?), and an ebook developer for even less. Before that, I worked for a (primarily) online agency with a long and distinguished (and continuing!) history of creating interactive experiences with a very strong focus on narrative and story. Up until today, I didn’t think there was much common ground between the two: after all, a book is a book and a game is a game, right? However, in trying to map out the differences between the two, I’m starting to think that all the stuff I’ve learned about presentation, narrative flow and appropriate modes of interaction might actually come in useful when trying to figure out where to go next with ebooks on a device that has many more capabilities than a simple ebook reader.

My background as an interactive developer means that I tend to want to make stuff that’s as immersive and interactive as possible, playing with the available technology. When I’m making ebooks, however, I’m trying to make something that’s as much like a real book as possible, down to spending days over perfecting a page-turn effect. How to reconcile the two? Is it by making ebooks that make heavy use of hypertext, breaking a book down into a non-linear, user-directed stream of fragments? Maybe with some whizzy graphical effects, flying from fragment to fragment (I’m pretty sure I remember some monstrosities along these lines spewing forth out of CD-ROMs back in the early-90s day)? Of course it isn’t. It does raise a few interesting questions though: once a book stops being a slab of dead tree and becomes a bitstream, how does it retain its “bookness”? How much can you meddle with the experience of reading a book before it stops being a book? Should we even be meddling at all? That pretty much all existing ebook readers have gone to great pains to emulate the physicality of real books, pages, page turns and all suggests that, at least for the moment, erring on the side of caution in this respect is very much the order of the day. I’d suggest, though, that perhaps this for similar reasons that (according to folklore, at least) early PCs were coloured the same beige as all other office equipment at the time: until people are used to a new, disruptive technology, dress it up like the familiar, old technology.

A big steer on the way I’ve been thinking about this is this wee snippet of a conversation on the always-excellent Shift Run Stop podcast with the author Chris Cleave. The episode can be downloaded here, but I’ll transcribe the relevant bit here for the impatient. You really should listen to the podcast though, it’s awesome.

~33:11
Cleave: “I don’t know how much the concept of authorship is relevant [in the environment of a game] where people can choose.”

(Cleave then goes on to describe the kind of game he’d like to play with his audience, collaboratively buidling a novel which is super-interesting in itself, but probably best discussed another time).

Once you place control of the flow of a narrative in the hands of a player, you hand over at least part of the authorship of that narrative to the player. This is not a bad thing: a well crafted interactive narrative is an immensely satisfying thing to play, arguably because of the role you’re taking in authoring the ongoing story. Valve are particularly good at this. However, it does mean that you can no longer claim that the resulting story has a single author, and here’s the crucial point: making a narrative interactive divides authorship duties between the original author and the player, fundamentally changing the way that it is experienced.

If we accept, then, that the essential defining characteristic of a book (e or otherwise) is the experience of reading a linear, authored text in the form into which it was painstakingly crafted by the original author, where does that leave us with expanding the ebook?

Firstly, it strikes me that at the very least, the capabilities of an iOS (or similar) device have great potential in being able to define the context in which that text is read. There are examples of books that have shipped or released in tandem with CDs. By listening to a custom soundtrack while you read the book, you’re getting a more immersive experience than reading the book alone, although that’s still the core act. Extra meaning and texture has extended the basic reading experience in a way that doesn’t interfere with the structure of the text and the authors intended meaning: preserving “bookness”.

Extending this train of thought, GPS data could be used to make certain portions of a text only visible in a specific spot, or kinds of spot (think: parks, bars, the top of a hill etc…) and I’m sure there’s many, many other ways a smart device can be used to enhance the context in which an ebook is read. This feels like a really exciting area to me.

Secondly, although I don’t really enjoy reading ebooks on an iPhone (more a personal foible than a fundamental flaw in the technology), I have been really enjoying playing through old (and not so old) Z-machine games in Frotz. In a nutshell, these are text-based adventures that you play by typing commands like “walk north”, “take lamp”, “kill troll” etc. Something about them feels ‘right’ on the iPhone in a way that ebook readers don’t; while you’re still reading a lot of (in a lot of cases, very well-written) text, the read / think / interact / read / … rhythm feels more suited to an interactive device, rather than the artificial simulation of dead-tree book rhythm you get with an ebook reader (why are we still simulating pages on a device whose form factor is nothing at all like a page in a book?). With a bit of thought to adapting the flow of text to better suit the nature of the platform on which it’s being delivered, I think some truly satisfying ebooks can be produced which stand up as interesting works in their own right, rather than clumsy copies of traditional books. Six To Start and Penguin Books did some groundbreaking work in this area on the We Tell Stories project in 2008.

Jon Hopkins – Insides
Richard Hawley – Truelove’s Gutter
British Sea Power – Man of Aran
Health – Get Color
Grammatics – Grammatics
Fuck Buttons – Tarot Sport
Moderat – Moderat
Tim Exile – Listening Tree
Mothertrucker – Dark Transmi55ions
Hudson Mohawke – Polyfolk Dance

Back in the day, all the noise was about how everyone would have their own personalised media stream, tailored specifically to the needs and tastes of the individual. Now it’s here, it’s clear that we’ve created devices for keeping us in the same old worn-out grooves, selective exposure raised to the nth degree.

Technology advances. We replaced our newspapers with RSS readers (already fairly effective partisan-mills) and then Twitter, an echo-chamber, mostly a closed loop for people with similar interests to share their similarity, create bubbles to inhabit. Things that allow similarity to glom together breed mediocrity, an average of everything: brown.

I want to be surprised. I want recommendation engines that recommend things from the very edges of my extended network, and outside of it. I don’t want 63% of people who bought this also bought this, I want to see the 1%, curve balls. Last.fm is good for this: I can be listening to “artists similar to Aphex Twin” and get the occasional bit of Bach, or ELO, or Motown because someone, somewhere in the world creates those overlaps on the Venn diagram.

Let’s turn the algorithms inside-out. Let’s get them to throw out the brown bits in the middle, and find the spiky, technicolour bits around the edge. Let’s create tools for surprising ourselves.

We pitched for and won two interactive activities. The first of these activities, and the one which I’ll talk about in this post was named ‘Tricked Out Trucks’. The brief was simple enough: we were to create a tool which would allow visitors to the site to customise an airport truck in 3D. The creator of the best design (as picked by a panel of luminaries) would win that truck for real, all pimped out and delivered to their door. Apparently, the client had come to this concept without ever having seen Den Ivanov & PARK Studio’s Brahma Bus in 2007, and were very surprised when we told them about it :)

We ended up differing from PARK Studio’s (pretty seminal) app in two ways. First, rather than allow freehand drawing, we assumed that most casual visitors to the site either wouldn’t have the artistic skills or inclination to draw an elaborate design, so we provided preset patterns which the user could customise to their liking and decals which they could transform and place freely. Second, we wanted to allow users to make modifications to the shape of the truck by adding body mods: spoilers, wheels and wing mirrors.

The Process, Part I: 3D Models

The first thing we needed were 3D models. After gathering a whole bunch of photos, we asked JS3D to cast their minds back to Quake-era game modeling and provide us with low-polycount models of the truck and mods. We worked on a rule of thumb of 200 – 500 polys per model, which was pretty conservative; we could probably have got away with up to about 1000, but we were treading carefully in unknown territory (this was our first big 3D project), and I wanted to err on the side of caution in the interests of keeping the chances of encountering unpleasant surprises as low as possible. JS3D were really helpful here: they understood what we were trying to do straight away, and were great at suggesting optimisations I’d never have thought of, such as faking complex geometry by using transparent texture maps on simple planes (this is how the wing mirrors and spoilers were achieved). 

They were also really helpful when we ran into that dreaded pitfall of Flash 3D engines: sketchy z-sorting. Working closely with JS3D, we identified the areas of the model which were most affected: areas where polygons overlapped, for example, or were in close proximity to each other, or inside the volume of other shapes, and tweaked the model to minimise those effects. At the time, I wasn’t aware of the alternative depth sorting options available in Papervision and Away3D; these would certainly have been helpful to us, but actually altering the geometry of a model is a fairly bullet-proof way of reducing z-sorting artifacts, and I’d recommend it to anyone who is having trouble with polygon-popup issues, especially given the extra processing overhead of the sorting methods.

The Process, Part II: Decorating the truck

The second big part of this project was working up a system which would allow us to redraw parts of the texture map on the fly, and have that update on the model in realtime. Conversely, for positioning decals, we needed to detect where the user was pointing on the model and accurately draw decals back onto the texture map, based on their state in the interface. We were lucky here in that the truck was a pretty simple, boxy shape, which made manipulating different parts of the texture map pretty easy: essentially, I treated the truck as a box, and the texture as a net of that box. I then created a set of masks corresponding to each side of the truck to allow me to isolate those sides in the texture bitmap, and a matching set of transforms that allowed me to composite new bitmaps in the correct place (and with with properly adjusted scale & rotation) on that texture.

The texture drawn onto the truck model has five layers (in order, from the bottom up):

1. paint
2. pattern
3. body details
4. decals
5. baked lightmap

As the user makes changes in the interface, the layers for the relevant side are redrawn within their mask area, and the newly-recomposited texture is passed to the truck view, where the truck’s material is updated with the new bitmap. By redrawing the texture in specific areas only, the whole cycle could run quickly enough to update the truck view in realtime, most of the time ;)

Getting decals working correctly was quite tricky. In the interface, we presented the user with the option to drag & transform decals across the view of the truck, which would then be added to the texture map. I eventually got this working with a bit of a hack: given that the truck view was being locked off at dead-on 90 angles while the user was positioning their decals, this meant that the view on screen corresponded fairly directly to one of the masked face areas in the texture map. Therefore, if I could get the mouse’s relative position on the truck view, I could translate that directly to a corresponding position on the texture. This position had to be a percentage of the distance from the top left of the view (e.g. (.3, .8) ) rather than absolute pixel values to allow for scale differences between the texture map on the onscreen rendering of the truck. 

I later found that this is called UV mapping by proper 3D programmers, and the ability to get the mouse’s UV coordinates on a texture has been available in both PV & Away for quite a while. Oh well; nothing like implementing something yourself for getting a proper understanding of it!

Afterword: 3D in Flash

At the beginning of the project, whilst still in the prototype stage, I created an abstract TruckView class and wrappers for Away3D and Papervision so I could switch back and forth between the two during the development process for actually rendering the thing. The following are my impressions of the two as I worked through the project. 

I started out with Away3D, because (to my eye) it looked a bit better during my early tests. Then, as I tackled the finer points of interacting with the surface of the truck model, I switched to Papervision, as it seemed to have better interactive features, and better documentation on those features. Finally, however, I switched back to Away3D; in the final stages of the project, we were having some performance and z-sorting issues, and Away3D was hitting consistently higher framerates and looking better whilst doing it. It’s generally my impression that Papervision was a bit stronger at hardcore technical stuff and tends to implement new features more quickly, but Away3D had a nicer API and (crucially) looked nicer.

I should stress that this is just my opinion, and the observations that I made about the strenghths and weaknesses of the two could just as easily be down to my (mis)use of them as to any actual differences. Remember, I started this project as a n00b to both engines, and they make progress all the time; both will certainly be totally different beasts by now from their state last year. That said, we’re currently working on another 3D project which went through a similar evaluation process at the beginning of development, and again we settled with Away. The best recommendation I can make if you’re about to start a 3D project in Flash is to evaluate both and see which best meets your needs.

Last summer, Mathmos asked us if we fancied doing something for the 45th birthday of their iconic Astro Lamp. We felt that a natural fit for this would be a screen saver that simulated the lava lamp.

From a coder’s perspective, this project had two interesting problems to solve. Firstly, we had to consider the physics at work inside a lava lamp in order to create a realistic simulation. Secondly, once you’ve figured out where your blobs are, their sizes and how they’re moving, you need to draw them in a satisfyingly blobby way.

This second post will cover how I went about drawing those blobs. You can read Part I here.

The Interesting Bit II: Drawing blobs

Drawing blobby shapes on a screen is a well established area of computer graphics theory: they’re called metaballs and the technique was invented by Jim Blinn in the early 1980s and popularised through the demoscene in the 90s. As a side note, if you’re unfamiliar with said scene and even remotely interested in creative computing, you owe it to yourself to get acquainted. Here’s an example of a production particularly rich in metaballs:

Anyway, the general theory goes something like this:

If you’ve got a collection of centres of force (say, the blobs’ centres of gravity), and a bunch of stuff attracted to those centres (say, wax), then you get shapes forming as the stuff collects around those centres. At a certain distance from any given centre, its attractive force isn’t enough to hold the stuff in place: this threshold is where the surface of your blobs form. In computer graphics, this is called an isosurface: it’s a surface which runs through space, tracing an edge where the attractive force is constant at a certain value.

If you only had one centre in a system, this method would end up drawing a perfect circle (the stuff is attracted to that single, lonely centre equally in all directions), but as you add more centres to a system, you get blobby shapes forming as the multiple attractive forces alter where in space that boundary falls.

There’s a number of different ways you can achieve this effect in Flash. By far the easiest is cheating it with BitmapFilters, but this wasn’t going to work for our purposes; it’s fine when you’ve only got a small bitmap to render, but we needed to draw metaballs across the whole screen. For that matter, any method which involved drawing a bitmap of the blobs wasn’t suitable for that same reason.

That meant that I’d have to find a method that used vectors to trace the edges of the blob shapes. I started by looking at this method and its AS3 port, but still wasn’t getting the speed I needed (that, and the fact that I couldn’t even begin to understand the Clever Maths at work, so didn’t stand a chance at optimising the code and bending it to my will).

The next method I tried was marching squares. This method works by breaking down your drawable area into a grid of cells and sampling groups of four cells at a time. By calculating which of those cells fall inside of the isosurface, you can both draw an approximation of the shape of the surface across those four cells and figure out what direction your sample window should move in to continue to trace that surface.

You can see an example of the simulation running with a marching squares renderer here. While it was pretty quick, it didn’t give us visuals of nearly a high enough resolution, and experiments fitting a curve through the points generated by the trace didn’t work very well. I decided to prototype my own method in Processing.

The principle is pretty simple: start at a centre of mass, then work outwards until you hit the surface threshold. Once you’re at the surface, take a 90° turn and start to trace the edge of the surface by taking a step forward, then sweeping left and right until you hit the surface again. Repeat until you get back to where you started. I called this the ‘border patrol’ method.

This worked pretty well to a point; it was very fast, and easy to understand. Unfortunately, it was also quite temperamental. Getting the method to recognise when it had got back to its starting point proved tricker than it had any right to be, and it wasn’t very good at turning around sharp, acute corners.

In the end, we simply ran out of time and ended up porting and modifying this method, published by Den Ivanov, which I suspect is a clever variation on marching squares.

This is my biggest regret from the project; I’d really like to have continued to develop the border patrol method, get it bug-free and stable enough to use. However, sometimes you reach a sensible limit where you have to recognise that you’re sinking a disproportionate amount of time and effort into a problem and go with what works, even if it’s not exactly what you wanted.

That said, I’m still proud of the final product, the client were happy and it was a hugely educational project to work on. Maybe one day I’ll go back and finish off border patrol properly ;)

Download the screensaver yourself from virtualastro.mathmos.com

Last summer, Mathmos asked us if we fancied doing something for the 45th birthday of their iconic Astro Lamp. We felt that a natural fit for this would be a screen saver that simulated the lava lamp.

From a coder’s perspective, this project had two interesting problems to solve. Firstly, we had to consider the physics at work inside a lava lamp in order to create a realistic simulation. Secondly, once you’ve figured out where your blobs are, their sizes and how they’re moving, you need to draw them in a satisfyingly blobby way.

In this first post, I’m going to talk about the physical model we created to simulate the motion of blobs in a lava lamp. You can read Part II here.

The Interesting Bit I: Lava lamp physics

When I was in secondary school, I ran across some instructions on the internet (at that point, it might even been on Gopher) on how to build a lava lamp. The recipe called for blobs of vegetable oil suspended in methylated spirits. I built it with my science teacher; it failed. But I’d learned a lot about how they work along the way.

In the base of the lamp, there’s a light bulb which illuminates the lamp and heats the liquid. This causes the blobs to rise to the top where they cool and sink back down again – a textbook example of heat convection. While they’re doing this rising and falling, they’re colliding with one another and glomming together to form new blobs, or breaking smaller blobs off as some regions of a large blob change temperature more rapidly than others.

To simulate this process, I started with a simple setup in APE: gravity, a few circle particles and box to hold them in. I then wrapped the basic particle class to give me particles that could have a certain amount of heat, which the system uses to apply an upwards force to each particle. The strength of that upwards force is dictated by a particle’s heat, which is increased while the particle is near the bottom of the system, but gradually cools over time to the point where gravity becomes stronger than the upwards force and the particle starts to sink.

That gave me masses which would rise and fall inside my simulated lamp, and half of my physical simulation.

The second step was to simulate the glomming together / breaking apart behaviour of the blobs. This is achieved by spawning springs between particles when they get within a certain distance of each other to make them drift together. These springs are quite loose, so as to create gentle motion, and also have a breaking point. The breaking point is important, as it allows particles to unstick from one another as gravity and heat force them in opposite directions.

This worked pretty well, and, once the spawning and breaking points and spring stiffness had been juggled sufficiently, we ended up with a system where nice blobby shapes formed from groups of particles coming together as they drifted past each other on their way up and down.

For extra bonus points, I added a stage where connected particles exchange mass downhill along their springs; this simulates an effect I noticed in lava lamps where blobs appear to flow into one another, and fuels the breaking off effect as the lower blobs become heavier and slow down relative to their rapidly shrinking conjoined siblings. This little touch made the simulation feel that much more realistic, and I’m particularly chuffed with the way it turned out.

Continue reading in Part II.

Today, I added a SWF asset to my main class, like so:

[Embed(source="../../assets/frame.swf")]
[Bindable]
public var Frame:Class;

but upon compiling, mxmlc threw the following error:

Error: The definition of base class MovieClipLoaderAsset was not found.

It would appear that using the [Embed] metatag requires the presence of some classes you’re not going to get by default. 

A quick Spotlight search later revealed MovieClipLoaderAsset.as in the following location:
{Flex SDK}/frameworks/projects/framework/src/mx/core

Adding the framework source folder:
{Flex SDK}/frameworks/projects/framework/src to my project as a new linked library solved the problem and the compiler was happy again.