Background

There has been some discussion about the state and future of ActionScript 3 and the Flash player from the Flash/Flex community recently. Ted Patrick is spurring some of this discussion - kudos to Ted for touching the community and getting a great response - and others at Adobe have also taken part. Several articles have also been posted such as:

• Joa Ebert: This is an outrage
• Nicolas Cannasse: The failure of AS3 ?
• Andre Michelle: AS3 Failure?
• Nate Lokers: Flash Scheme, Lisp, and jQuery-like $ugar
• Peter Elst: Making the case for ActionScript
• Keith Peters has also been talking on Twitter about this.

I have come to Flash somewhat reluctantly, being a fan of open software and environments. I have done a share of Microsoft work in the ancient past, but I generally enjoy a more open environment such as Java, Ruby, Python, HTML, JavaScript, et. al. I use Flash (and selected it as a technology of choice when I was architecting for a massive new release of our market leading debt collection software at Ontario Systems) primarily because of its ubiquity, power, and consistency across platforms. Flash reigns because of these three factors and they are the most necessary places for Adobe to concentrate its efforts.

From the Java fan club to the latest dynamic cult

However, ActionScript 3 as a language is less than exciting to me, as I have sought to avoid the drudgery of Java in the past and have yearned for more dynamic alternatives in Ruby, Python, Objective-C, and even Lisp. It was for this reason that I began working on Red Sun, a project to port Ruby to the Flash VM. My initial work on Red Sun centered on generating SWF files from Ruby. I implemented a SWF parser and creator with support for creating and modifying the ABC sections and other critical sections of the file format. The SWF format is tag based, so I simply ignored and passed through sections my tool didn't comprehend. The ABC (ActionScript Byte Code) section was fully parsed and could be modified and written out. The tool could also generate a new SWF from scratch and translate simple Ruby code into ABC functions.

This route was extremely illuminating to my understanding of the Flash player, but ultimately I found it was too difficult to actually complete a reasonable implementation of Ruby in time for my presentation at RubyConf 2008. So, in a flurry of activity that I had between jobs I instead ported the MRI Ruby interpreter from C to ActionScript 3. This task was completed and using Red Sun you are able to run a significant portion of Ruby in the Flash VM. The major limiting factors remain:

1. Dependence on Ruby 1.9 to parse and compile Ruby code into Ruby bytecode
2. Poor coverage of the Ruby standard library (implemented in C and only small portions ported to ActionScript 3)
3. Little production testing and likely large memory leaks remain

The interest in Red Sun was decent for the conference, but really I don't think anyone has used it. I have received sporadic questions about getting it running but that is all. I know that much of this is my own fault due to lack of promotion or compelling examples, so I am not really frustrated or anything. I learned enough in that work about Ruby and Flash and writing languages to make the effort more than worthwhile.

There is a big difference in the Ruby culture of openness and the closed source Flash player culture. Also, Ruby is driven by developers, albeit creative ones, and not designers like the Flash world. Few full time Ruby developers are trying to generate interactive graphics experiments for the web or enterprise RIA solutions. I think in general the allure of Flash is small for Rubyists.

Flash Forward!

So what does my experience lead me to look forward to in the future of Flash? It is a difficult question, as there are countless frustrating limitations of ActionScript 3 in relation to Java or C#, not to mention dynamic languages and even JavaScript. As others have pointed out, ActionScript 3 is in a limbo between the dynamism of JavaScript and the structure of Java. The ActionScript 3 prototype is hampered to near-uselessness but we are missing important object oriented basics such as method overloading, true enums, covariant return types, etc.

In terms of my current "day job," I care about advancing ActionScript 3 so it is easier to use, and upgrading the capabilities of the Flash VM so we can do our job more effectively. Primarily, what I find as major issues are related to working with Flex and its love of private variables and deep class hierarchies. Flex 4 is addressing some of this and I think it will be a big improvement.

What seems to be more important than changing ActionScript 3 are the broader issues such as performance, a global error handler, and possibly threading. Although everyone that begs for threading needs to consider the complete and utter breakage that will occur if it is enabled. If you think it is difficult to write asynchronous code, it is much more difficult to write proper multithreaded code. I see Flash and Flex code that will regularly break when multithreaded that used to make me cringe, having come from an inherently multithreaded server environment and also time working with Java. The Flash VM will be forced to break locks to keep from stalling the VM and that could easily cause major issues.

Speed improvements both to the AVM2 and to rendering would be one of the biggest attractions to keeping developers and designers on Flash. On a competing platform I have built effect mockups and received them from designers that had to be scaled way back for performance. This was extremely irritating and destroyed our vision. We were used to designing for Flash and this platform just couldn't keep up with our creative vision. I hear people ditching Flash for Silverlight for this very reason. C# does have some nifty language features, but you can still argue for Flash in general until performance kills you (which is rare, but happens).

Personal aspirations with Flash

I don't expect Adobe to create a language that I love, given the current direction, but fixing a few issues and spending their time on performance could really be a boon for the life of Flash. Adobe has put a lot of effort into MXML and it's easy and flexible binding is a great tool for building a DSL. Degrafa and Axiis are prime examples of the power of MXML. If Adobe can push the performance of the ActionScript generated by the compiled MXML it could be a saving grace.

I have been researching the possibility of creating an expressive language that is also efficient and interactive for visual experimentation. I then to think that the Flash player is the best environment for it, since sharing via the web is so ubiquitous. Processing already exists for offline rendering and I don't have any hope or desire to compete with it, but I don't think it works so well for integrating into the web (it does offer the best applet experience I've seen, however). The major downside of working in Flash is performance and device access. There are popular workarounds for accessing devices in AIR, but if my work will run in AIR and use a custom language I would rather just build a native executable.

The bottom line

I will favor Flash as long as the install base stays as high as it as and as long as it is fast enough to be compelling above Silverlight, JavaFX, and JavaScript solutions. The community can work around language frustrations and even build better frameworks alongside Flex or as replacements to Flex. Adobe can benefit from all of this excitement as long as the player remains the dominant technology.

When you're building your interface, you cannot predict how users will behave. You put common links on the side, support breadcrumb navigation, and build a flexible menu system. But the users still have to jump through hoops to move from task to task. Perhaps they regularly jump to the home page or main menu and then trek back down through the system to find the tool they need next.

Users are forming Desire Lines in your UI. They are telling you what order they typically perform tasks in, but you cannot see them. Unfortunately, these lines are not as obvious as the dead grass and dirt that characterize them on university campuses. As a UI designer you must gather the data and observe the trends. Then you can pave the desire lines with concrete and help your users get where they really want to go.

http://en.wikipedia.org/wiki/Desire_lines

A desire line is a path developed by erosion caused by animal or human footfall. The path usually represents the shortest or most easily navigated route between an origin and destination. The width and amount of erosion of the line represents the amount of demand. Desire lines were used in early transportation planning, prior to the advent of computerized models.

They are manifested on the surface of the earth in certain cases, e.g., as dirt pathways created by people walking through a field, when the original movement by individuals helps clear a path, thereby encouraging more travel. Explorers may tread a path through foliage or grass, leaving a trail "of least resistance" for followers.

The lines may be seen along an unpaved road shoulder or some other unpaved natural surface. The paths take on an organically grown appearance by being unbiased toward existing constructed routes. These are almost always the most direct and the shortest routes between two points, and may later be surfaced.

Desire lines can usually be found as shortcuts in places where constructed pathways take a circuitous route.

Many streets in old cities began as desire lines which evolved over the decades or centuries into the modern streets of today.

If users could leave desire lines in our applications, then these would instruct later users, as well as designers, in how users find their paths across our applications. Don Norman calls desire lines social signifiers. They are signifiers that are created by the behavior of social groups. Our pristine application interfaces aren't trodden by the feet of countless travelers, but would they be better interfaces if they were?

Perhaps interface designers can look for ways to instrument their user interfaces to report this data and improve constantly. Certainly this is an accepted technique for observing users' paths through a website and it should be adopted and supported by user interface frameworks across all platforms.

Custom cursors offer tactile feedback to users of an application about what operations are possible on the underlying component as well as the state of the application. The busy cursor encourages the user to wait for an operation to complete while the hand cursor informs them that this user interface component will respond to their mouse clicks. However, there are unique problems with the custom cursor implementation in the Flash player.

The Flash player utilizes a rendering pipeline that is affectionately referred to as the elastic racetrack. The racetrack represents an infinite loop of processing in which one lap around the racetrack represents a single frame of animation. The Flash player is deliberately single threaded and therefore if an operation takes up a significant portion of time the racetrack can lengthen, the framerate will decrease, and users will notice the application becoming unresponsive.

Most rendering pipelines have an issue such as this, however the Flash player's single threaded nature and the general difficulty of deferring work from one frame to the next causes this to be particularly noticeable. A typical desktop application may also enter a long period of single threaded processing, however this behavior does not negatively impact the mouse cursor. The reason is that most desktop applications modify the system cursor which is not rendered by the application's render loop, but is handled by the operating system in a more robust fashion. Flash player, however, renders its custom cursors using its own render loop - so a long running operation will cause the mouse cursor to stick, jump, and disappear.

This is obviously a very jarring and frustrating experience to most users. Due to the typical behavior of desktop applications, most users will interpret the jumping or nonexistent cursor as an application crash rather than simply a long running operation. This is particularly complicated in the Flash player as the busy cursor is also rendered by the Flash render loop and also suffers from these limitations.

There are two ways to address this problem. The first is to pay careful attention to the processing time of any code in your application. This is a common suggestion, but one that is not trivial for most developers. This is complicated further by the issue that even if an operation is determined to be taking too long there are limited options available to the developer. The developer can first try to optimize the operation as much as possible so that it does not take so much time. This is the most direct approach and often leads to good results. The next approach is to try to spread the operation over multiple render loops. This is often nontrivial and not obvious. This can be further aggravated by the delayed and queued rendering behavior of most Flex components. If a Flex component is not provided with its data during its first frame of existence it may be rendered during that frame in a partially complete manner, which can often cause significant visual jarring in the application when it resolves to its correct state.

This visual jarring can be reduced by causing the component to wait to render itself until all of its data is available, but this is not a common practice and it is not immediately obvious how to achieve this using the Flex framework.

The performance lag can also come from other sources such as an extended service call (which should not itself slow down the framerate of the Flash player), parsing large amounts of XML or other data, or rendering complicated graphics. The Flash player parses XML data when it is returned from a service call into XML objects and with significant data sizes this can cause a lurch in the Flash player. The Flash player also typically renders the entire visual representation during one frame and complicated visuals - which may perform adequately after their initial rendering - can cause a pause.

The second way to address this problem is by altering the use of custom cursors in the Flash player. Since the movement of the cursor is governed by the framerate of the Flash player this can be beneficial even when there are no significant performance barriers. Users will be accustomed to their cursor moving at near the refresh rate of their monitors - usually above 60 frames per second, or 60 Hz. Flash movies and Flex applications typically run at framerates of 24 Hz, although developers can override this setting. If you are developing an application where smooth animation and cursors is a necessity, it would be wise to raise the framerate somewhat, however, problems have been reported with server calls when the framerate is set too high.

Thankfully, there is a compromise possible between abandoning custom cursors altogether and using them in the typical manner. This compromise is to decorate the system cursor rather than replacing it completely. Decorating the system cursor involves attaching a Shape or Sprite to the current mouse position and avoiding operations which set the Flash cursor directly. This decoration may at times lag behind the system cursor, but it will at least allow the user to move their mouse with confidence across your application.

Decorating the system cursor can also be a significant help when the user is performing precise manipulations such as in image editing or layout. Since the system cursor will still render at full speed the user can be confident in their motions, even though the Flash player may lag in updating the user interface.

Observing users is, as always, one of the best ways to identify usability problems with your application, and should be relied upon to focus your efforts on important problems. However, attention to detail will always be appreciated by users even when they cannot specifically identify the problem they are experiencing. Unhappy users are more likely to simply quietly disappear than to register their complaints, and you may lose them quickly if they feel your application is unresponsive. Most users will equate an unresponsive application with a broken application. If your application is involved in selling anything to users this can lead them to question whether your application is safe enough to enter their financial information into and result in abandonment.

Touch interfaces are increasing in popularity and ubiquity since the introduction of the iPhone and other mobile devices. Touch screens deliver a larger interactive surface and direct manipulation of the user interface to users. The direct manipulation leads to an intuitive interaction experience, but the touch has a particular Achille's Heel that the mouse doesn't suffer from - the user's finger is blocking their view of the screen.

There are several methods for addressing this difficulty, some of which are employed currently on touch-enabled phones. The iPhone, for example, will popup hints when you hold your finger over a letter on the keyboard or on a link on a webpage. However, this interaction is used sparingly. In the presentation of the new Palm WebOS the application menu bar "Wave" is drawn just above the user's finger so the user can see what they are selecting.

Context menus are a particular case in which the touch interface obscures the data being selected. Context menus also suffer from a typical problem in the transition from large, high-resolution desktop displays to a mobile device - they are too large and rely on vertical scrolling. Hierarchical context menus particularly aggravate this situation by requiring more vertical scrolling.

A better context menu could be based on a radial popup menu. The available options could popup in a circle around the user's finger. This allows all of the options to be visible at once, and the user can move their finger horizontally and vertically to make a selection. The interface can register a selection based on the direction of the user's finger, whether or not the finger is actually hovering over a menu item. This allows the user to move their finger outside of the radius of the context menu and make a selection without obscuring the text describing the function.

Hierarchical context menus can be implemented similarly with successive levels of detail opening out of the radial menu when the user moves their finger over that section of the menu. The speed of opening the hierarchy must be closely controlled and adjusted based on user tests to keep users from making unintentional selections.

Canceling the selection can be facilitated by the user moving their finger back to the center position that originally initiated the context menu.

Corey Haines stopped by and paired with me on Monday, 12/13/2008, this week. We had a great time talking and hanging out, as well as working on the Red Sun Ruby Flash VM for several hours. We implemented the core trappings for exception handling which was a major missing feature in the VM. We had a long talk about software craftsmanship and the journey that we take in our line of work.

I think the most important lesson that we felt was that a range of experience and interest is beneficial for a craftsman. We both traced some of our best learning and growing experiences back to interacting with key individuals. I also saw again how my path as a developer has been rather erratic as I've worked in significantly different environments with very different goals and deliverables. Overall, I think this has benefited me as a developer as I've been exposed to people with very different mindsets over the years.

Corey posted the video interview on his pair programming tour site earlier today. It was very interesting pairing with someone when the majority of the work was of the kind that was new to the pair. I haven't done a lot of pairing in my career (unfortunately). It's probably obvious to you if you've paired much, but even though Corey probably doesn't know ActionScript 3 well, and has not used C significantly, he was able to point out errors that I would have missed until either compile or execution. It also helped me to explain what I was doing as I went along. It forced me to have a reason for the things that I did.

The downside of this project in particular is that it doesn't lend itself well to Test Driven Development. I know some may argue that everything can be TDDed, but this is a unique kind of project. I am essentially re-implementing Ruby 1.9 (MRI/YARV) in ActionScript 3. This means that I copy, line-for-line, the C code into ActionScript, translating the idioms as I go. This means, as I explained/rationalized to Corey, that I can write large chunks of code without tests - even code that isn't yet running. The benefit of doing this is that when another part of the system gets implemented, those earlier parts will just work, assuming I did it right. Also, the goal for testing is to ruby RubySpec against Red Sun, and at that point I will have lots of failing tests to fix!

When I started working on Red Sun I didn't do this. I only implemented the functions and parts of functions that seemed essential at the moment. This led to significant slowdown and confusion later in the project. My partially complete method was now passed different arguments and data such that it exercised the alternate, unimplemented code path. I found it better to implement as much as possible of each function whenever I encountered the function, and to stub broken paths with verbose error messages or exceptions to point them out should they later execute.

I was very happy that Corey came and I am planning on posting some more insights from our discussions over the next few weeks. I was very happy that we got exceptions implemented. I intentionally skipped all of the exception handling code before RubyConf 2008 because I was afraid of getting stuck in it and having a broken implementation. I haven't implemented all of the various control flow interrupting operations (such as redo, break, and next), but raise and rescue are working pretty well. If you're interested in Red Sun, you can view the video of Red Sun Introduction by Confreaks from RubyConf 2008.

Thanks again for coming Corey! I hope there can be more people who take this journey and they can stop by Muncie if they do.

After my post about Don Norman's Signifiers, not Affordances article, I had an interesting discussion about the relationship between signifiers, affordances, and features. The importance of signifiers was again reiterated by several people as the key to driving user satisfaction and user capability within an application. Features can only be accessed if the user has an understanding that the feature exists, and signifiers are the agents you as a designer can use to communicate with your users.

I have always been interested in the idea of affordances in user interfaces since I first heard the term. So naturally I was very interested to read an article by Donald Norman - the original researcher to bring the term from psychology to design - about the importance of signifiers instead of affordances.

The term affordances was coined by psychologist J. J. Gibson to explain the possible actions that a specific agent can take on a specific object. Norman brought the idea of affordances, which are realities that exist regardless of the agent's knowledge of them, to design in the guise of "perceivable affordances." For Norman's design work, an unperceived affordance is essentially nothing.

In his article, Norman espouses the idea that designers should forget about affordances and focus on the signs that affordances exist; these he calls signifiers. His reasoning is straightforward - it's not the fact that this user interface widget can be dragged that is important to design, but the signifier that indicates to the user that the item can be dragged.

I will be presenting Red Sun at RubyConf 2008 this coming Saturday, November 8, 2008. Red Sun is a virtual machine for Ruby written in ActionScript 3 that runs in the Flash player and Adobe AIR.

I will be posting more information as the presentation wraps up. The development for this has been ongoing for a few months at Github in the Red Sun git repository.

The VM is moderately functional. If you can build the AIR project you can play around with it quite easily. I will be releasing better information and instructions soon.

Update: The videos from RubyConf 2008 have been made available to watch by confreaks. You can see all of the videos at RubyConf 2008 Confreaks Coverage and my video at Red Sun: Ruby Flash VM Confreaks Video.

I have published the source code that I demonstrated during my 360|Flex presentation on github. You don't need git to get the source, just click the download button on the page. Again, this is the code for Improving your Programming by Reading the Flex Source Code.

This does not include the class browser. That is an AIR project that I will publish separately.

Github is a source repository system like Google code or Sourceforge, but using git - a distributed source control solution. I recommend git, if you haven't used it, for any side projects or open source projects that you work on. Git allows anyone to branch/fork a project and make any changes they see fit. Then the changes can be merged back in if they are useful. Github supports forking of open source projects so people can, for example, make changes to a project available without needing approval of the original maintainers.

The nice folks at 360|Flex (with sponsorship from Adobe) recorded our presentations in San Jose Aug 18-20th 2008. I'm planning to transcribe the talk and perhaps post some better screencasts for the middle part of the talk. I had many specific examples prepared that floundered somewhat on stage. Also, a friendly attendee suggested that posting about Eclipse/Flex Builder 3 shortcuts would be good, so I'll do that. There are so many things that I planned but didn't include!

The best part of the talk is the first 8 minutes, at 0:45, and 1:01 where I show the Star Wars of Flex development. I also showed a few utilities that I've developed: a class browser, a stack inspection tool (to find method callers), and a performance profiler. I'll be getting these together and posting the source within the next 2 weeks.