Building an Interactive IFS Editor
O, Thou hast damnable iteration; and art, indeed, able to corrupt a saint.
— William Shakespeare, Falstaff, Scene II
I’m a software engineer, working on distributed cloud application management software, but sometimes I like to take a break from that and write code that does something more, well, fun… To that end, I have been working on and off for the last few years on a fractal visualisation system.
I’d like to show off some of the parts I’m proud of, but also some that I think need improved or fixed. I will talk about the design and implementation, and the code that makes it work, and in the process explain why I think it’s useful to have a long-term project like this.
First, let me introduce the project. Iterator is a program that renders IFS fractals. These are Iterated Function Systems, a set of transforming functions that operate on the 2D plane and are repeatedly evaluated on a source image to produce a resulting fractal. The project itself is around five years old now, and I recently released version 1.5.0 and I would encourage you to download the distribution and play with the IFS Explorer. The codebase is just over 6000 lines (not including comments and blank lines) written in Java 8 with HTML documentation — also available as a PDF. If you configure your JVM to trust the GitHub server then an easy way to get started is to use the JNLP launcher.
Rendering an IFS fractal like the one above turns out to be possible as both a recursive series of evaluations or as a stochastic process known as the Chaos Game. My program uses the chaos game method, which is computationally less demanding and has scope for a broader range of visualisations. I have actually implemented an IFS renderer using the lisp-based macro language in AutoCAD, allowing an initial vector image to be copied, scaled, and transformed repeatedly. This is also known as the MRCM or Multiple-Reduction Copy-Machine method, and the PC I was running it on rapidly filled its 512 KiB memory after only a few iterations!
This is a technical article, but not a mathematical one. The references at the end of this article give more detailed explanations of IFS mathematics than I ever could, so I am not going to try and write yet another tutorial on how Sierpinski Triangles and Barnsley Ferns work — please use the same primary sources that I did, it will give you a much better grasp of what is going on. I would particularly recommend the Flake book, Computational Beauty of Nature, which covers not just Iterated Function Systems (in Chapter 7) but also a huge range of other interesting topics such as Cellular Automata or Neural Networks that will provide inspiration for your own projects.
Architecture
The Iterator project started as a single application, IFS Explorer, which allows interactive creation and viewing of IFS fractals, and now includes a stand-alone image renderer and a scriptable animation creation application. These all use the same shared codebase and configuration mechanism, so I will mostly be describing the explorer application in depth, but will point out how the components are used in the other parts.
There are several other IFS applications available, and it is a popular choice as a computer graphics project at universities. I felt that the programs available did not allow easy and intuitive exploration and creation of fractals, as they often required the user to type in a series of opaque matrix coefficients and the UI was simply a canvas displaying the rendered result. From the start, IFS Explorer was designed as an easy to use application that would let anyone create Sierpinski gaskets or Barnsley Ferns just by clicking and dragging on screen with the mouse. This means my choice of language and framework was restricted to one with good 2D graphics and UI component APIs. I chose Java due to my familiarity with the language, having used it since its introduction in the early 90s. The Swing and AWT toolkits are getting a little old, but are simple to use and provide broad platform support. The Java 2D drawing APIs are not as complex or featured as OpenGL, Core Graphics or DirectX but they provide the usual range of primitives including affine transforms, which is sufficient for this application.
The basic functions used in an IFS are affine transforms, which map a point on the plane to another point reversible, after scaling, rotating, skewing, translating or flipping it. Because the affine transform objects are already provided by the Java APIs, I did not need to re-implement the matrix mathematics required for an IFS, which helped immensely getting started. The next choice was whether to use Swing, AWT, JavaFX or perhaps Eclipse SWT as the UI component API. In this instance, Swing seemed the best choice. It is tightly integrated with Java 2D, I have a reasonable amount of experience with it already, and it provides a good cross-platform experience. JavaFX would perhaps be a nice alternative, but it is lacking some of the OSX integration present in Swing, and AWT is generally too primitive and ugly. Eclipse SWT, while powerful, is much more difficult to use, and I have never developed a large application using it. Finally, I wanted to use a minimum of external libraries, so chose Google Guava as a useful utility toolkit, but stopped short of dependency injection frameworks or other tools. Initially I used the 1.7.0 JVM but recently changed to Java 8, to make full use of the latest language features.
So, having run git init iterator in my terminal window, I then had to create the class structure that would make up the application. I will take some liberties with the timeline, and intend to describe the final architecture only, unless the intermediate steps are instructive in some way. For the IFS Explorer, the MVC paradigm was the obvious choice. The collection of affine transforms would be the model, the views would include an editor and a graphical renderer, and the controller would be the Swing GUI application that handled user input and other UI activity. The Swing GUI controller can also be replaced with a simple text mode console application; this is used for the IFS Renderer high resolution single image creator and the IFS Animator scripted animation tool which renders frames showing manipulations of the individual transforms in an IFS based on a DSL describing the changes. The views also include a display of the underlying matrices making up the IFS and editing dialogs for the transforms and the application configuration properties. Configuration is seen as part of the controller, but is decoupled from the user interaction components.
In the explorer application there is an additional part to the controller, the Google Guava EventBus. This is used to pass the model state (i.e. the IFS data) between the controller and the views when the user makes changes. The Guava eventbus is a simple, in-memory pub-sub queue system. The view and controller components register as subscribers to the model data. When a view component is finished editing it publishes the updated model and the rest of the application can then reconfigure and redisplay as appropriate. The diagram below shows some of the interconnections between classes.
The MVC architecture lends itself to encapsulation and has very well defined points of contact between components. It is very common for both desktop and web applications, and in fact the Iterator components as they stand could easily be reused in a web application with a browser based UI running in an HTML5 canvas, with the rendering being done by a REST API driven service using the same mechanism as the IFS Renderer application. In a future release I will probably implement something along these lines. This illustrates the fact that the project is a work in progress, and allows me to learn and build software spanning multiple disciplines, giving me more depth of experience than my day-to-day work. This is an important part of working on a side project. Additionally, it gives insight into the entire project lifecycle, from design to release management and documentation. This should improve your skillset and help you in your regular work.
Implementation
The actual implementation of IFS Explorer proceeded fairly straightforwardly to begin with, and I concentrated on writing the basics of a user interface to edit the IFS transforms. Although the fact that the Java 2D APIs included an AffineTransform class precluded having to write a matrix manipulation library, the 2D geometry aspect of the code is still very much in evidence. In this section I will illustrate some of the code that performs these actions.
The initial setup of the MVC architecture using the EventBus class turned out to be very simple. Each component that required notification of changes implemented the iterator.util.Subscriber interface. This had two methods whose implementation would be annotated with @Subscribe and the whole class would be registered with the event bus, generally using bus.register(this) during construction.
Now, when the editor makes a change to the IFS object, it can call bus.post(ifs) and all registered subscribers will receive a call to their updated(ifs) method and can perform the appropriate actions. This allowed building an MVC application without worrying about composition or coupling of components, since all communication is mediated through the event bus with a minimum of configuration.
This architecture handles changes to the model, but there is more to rendering an IFS than simply the list of affine transforms. While the event bus is ideal for propagating live changes to the model, there is also a requirement to be able to update and apply changes to configuration properties like the rendering style such as from grayscale to colour, or to change the maximum number of threads in the rendering pool.
I required a configuration mechanism that would allow key/value pairs to be set, with varying types for the value, and loaded from files on disk. The .properties format is ideal for this, and Java supports it with a Properties object as part of its core APIs.
The property keys and values can be represented in-memory as a Map<String,Object> with getters and setters providing typecasting when retrieving values. The Guava library provides a ForwardingMap interface that the main iterator.util.Config class implements, this allows the configuration object to act as a map while delegating the put(key, value) and get(key) operations to an internal field. Loading is performed by the Properties object and saving involves a simple loop to iterate through the keys, writing them and their values to a file in the correct format. The values are generally assumed to be either the boxed forms of primitives, Strings or enumerations. There is a cast(value, type) method that handles conversions from String to the required type, although some setter methods also perform conversions, such as those handling colours. The setGradientStart() method is seen below, obtaining the RGBA value as an integer representing only RGB and then converting it to a hex string. The cast() method has an else clause that decodes these hex strings back to an integer and then a Java Color object.
The configuration object is important as a part of the controller, as it provides all of the data required to display an IFS, except the transforms themselves. It is completely independent of the actual view that will display the IFS, however, and contains no user interface code. In fact, for the IFS Renderer and IFS Animator applications this is the only extra class needed for the controller. Other functions performed by this class are the loading of colour maps from palette files, or generating gradients between two colours. The IFS Explorer application also provides a view onto the configuration, via a dialog that presents current values and will allow them to be changed and updated.
The configuration dialog, implemented as the iterator.dialog.Preferences class, ended up being refactored into a generic property editing dialog, implementing a iterator.util.Dialog<T extends Dialog<T>> interface which is both AutoCloseable and a Supplier<T> of the dialog object. You can see that the get() method of the supplier interface simply returns the dialog, and the static show() method uses both this and the abstract showDialog() method to display the dialog and handle exceptions with optional consumers of Throwable.
This is implemented as an abstract base class called iterator.util.AbstractPropertyDialog which extends the Swing JDialog class and adds methods to add fields of different types to it. The fields are typed, and can have restrictions in their formatting based on the Swing AbstractFormatter class, which is extended in the iterator.util.Formatter class to a typed BaseFormatter<T> version that is implemented for various different objects such as DoubleFormatter and OptionalIntegerFormatter and is used throughout the application where numbers need to be displayed as Strings in a consistent fashion.
When a typed field is added to a dialog, it must be possible to set and to retrieve its value, and the methods for doing this are inconsistent across the various Swing components for text fields, drop-down menus and numeric spinners. The Property<T> class is an extension of the Supplier<T> interface, and as can be seen in the code for the Preferences dialog each field has an associated property object.
These objects are in fact returned by the various calls to add the UI components representing the fields, as the dialog is constructed. There are no concrete implementations of the Property interface, rather an anonymous inner class is created each time in the field addition method. This is done with the attach(field) static method in the interface, which is overloaded for each type of component used.
After editing the properties in a dialog and clicking the button to accept the changes, the dialog handler will call the get() method on each Property object and then call the appropriate setter method on the Config object. This transfers the changes to the shared configuration map, and the views can be alerted to update their display.
The gradient fill property is an interesting special case, which required several anonymous helper classes. First, a new method in the base property dialog class addGradientPicker(string) was added to create the field. This used an anonymous extension of the JButton class that overrode its paint() method to display the gradient fill as it would appear, and a new Pair<T> tuple class was created to hold the pair of Color objects representing the start and end colours of the gradient. Rather than having the button perform an Action when clicked, an extension of the MouseAdapter class was created with a suitable mouseClicked(event) method and added as a listener. This class determined whether the mouse was clicked on the left or right side of the button, and opened a custom JColorChooser for either the start or end colour.
Although the Swing colour chooser dialog is customisable, the provided mechanisms are not particularly helpful.
Setting the preview panel to an empty JPanel was required to remove the existing preview, since setting it to null had no effect. The filtering of the chooser panels to determine which was the swatch picker, and then remove it, is necessary because there is no way to instantiate a new set of chooser panels, nor is it possible to modify the array of panels in-place.
The end result is an intuitive and visually attractive way of displaying and editing a gradient fill.
The actual display of the IFS is one of the most interesting parts of this project. It involves some maths, although not incredibly complicated or advanced, mostly a basic understanding of geometry and trigonometry. Initially, the implementation was single-threaded, but the current versions use a multi-threaded approach to gain more performance. This required a good understanding of concurrency and the available tools and primitives provided by Java, and also Guava.
For example, the points being iterated must be accessible by all the threads performing the iteration calculations. I eventually settled on an atomic reference to an array of Point2D objects, available as the AtomicReferenceArray class. In Java 8 this has several useful properties, over and above the usual CAS (compare-and-set) operations. As seen below, I use the updateAndGet(i, updateFunction) to evaluate the application of the transform on a point and atomically set the result and return it. The second version instead returns the original point before performing the update.
This enables multiple threads to execute their application of a (randomly chosen) transform concurrently, without worrying about issues like the X value being set by thread A and the Y value by thread B causing problems and generating incorrect images. Relying on these implementation provided primitives meant that I did not worry so much about correctness, and could trust they would do the right thing. However, managing multiple threads was still an issue, and not something that the JDK made simple.
The problem is to ensure that a pool of N threads can execute concurrently, and will all stop when required as well as being able to increase and decrease the pool size, even while the iteration process is running. Executing a thread will return a Future<?> which can be cancelled, but this is not as useful as it sounds. In fact, I had to design the tasks that would execute concurrently to respond to separate stop signals for individuals or the entire group. This involved maintaining an array of AtomicBoolean latches, one for each thread, that would be checked periodically at the same time as checking for a global stop signal.
As you can see here, the tasks (a Runnable object, which was implemented as a lambda) wait on a latch object to ensure they start at the correct time, then obtain a token value which is an AtomicLong that is used to halt all threads if its value changes. The threads have their Future<?> objects stored according to the task type in the tasks Multimap and the AtomicBoolean cancel objects are stored in the state map, linking them to the futures for their threads. This is used to stop an individual thread by setting the value of its AtomicBoolean to true.
All of the concurrency mechanics are encapsulated in the iterator.view.Iterator class, which only requires access to the Config and IFS objects to execute. It will generate a BufferedImage Java 2D object containing the rendered IFS image. This means that both Swing applications and headless console mode applications can use it to generate images. Additionally, it is not necessary to always execute the image generation tasks concurrently. The iterate() method is public, and the Editor view of IFS Explorer uses this to produce a live view of the IFS while it is being edited.
Looking at the code, you will see that the minimum number of threads is two, yet the editor view executes a single iterator task. This is because there are actually two types of task.
For rendering modes that use a density estimation mechanism, the iterator tasks do not change the BufferedImage pixel, rather they update an array of density values. The second type of task, PLOT_DENSITY, then iterates over that array and sets the pixels in the image appropriately. So for N threads running the LOG_DENSITY_FLAME_INVERSE rendering mode there are actually N-1 tasks of type ITERATE and one PLOT_DENSITY task. Later I will go into much more detail regarding the plotting and iteration mechanisms, and the modes and configuration available.
A non-obvious problem in the display routines was the plotting of grid-lines. It is fairly simple to draw a horizontal line every N pixels, similarly for vertical lines. What is more complex is doing this for a viewport onto the unit square, which may be centered anywhere and have either positive or negative magnification. Obviously the grid spacing must be adjusted with the scale, for two times magnification the lines should be the same number of viewport pixels apart, but will need to be adjusted to have half the virtual distance between lines. Additionally, calculating where to start drawing grid lines and where to stop must be done, as well as where to draw the lines intersecting (0,0) if they appear. The simplistic approach, which I tried first, of iterating from a very negative to a very positive number and only drawing the line if it intersects the viewport (using line.intersects(rectangle) in the Java 2D API) quickly eats up the available CPU at high magnification.
Instead you must calculate how many multiples of the grid spacing are present from the origin to the left margin and then what the remainder or offset is to the first visible line. This code took many tries to get correct, and required essentially empirical checks of where the grid-lines intersected a part of the fractal, and verifying that this remained consistent across magnification and reduction and changes of viewport. The complete algorithm is shown below, although it works as expected, I believe that some simplification is certainly possible.
The IFS model is fairly simple, needing only to hold the lists of transforms with the parameters for each affine transform. The model also contains the name of the IFS (used as the window title) and the window size, which is required because the transforms are defined in terms of screen space pixels and therefore must be resized for rendering at different output resolutions. I decided to use JAXB (Java API for XML Binding) annotations on the model classes, allowing them to be saved as XML files without needing any additional libraries, since JAXB is part of the JDK.
The XML is saved and loaded by creating a JAXBContext for the model and using that to create Marshaller or Unmarshaller JAXB objects. Once any configuration properties have been set, such as for pretty-printing the XML, it is a simple matter to call the marshaller.marshal(jaxbElement, writer) to output the XML. In future releases I may change this to support JSON formatted model data instead, using the Google GSON library which is also annotation based and has equally simple marshalling and unmarshalling methods. The only problem with this change would be maintaining backwards compatibility with older XML saved models, and it may end up being necessary to simply support both formats.
The IFS class also contains some Comparator<C> implementations, one to sort the transforms by z-order for help when selecting overlapping transforms in the editor view, and one to sort any type of Function by id. These are defined as lambdas, with the IFS.Z_ORDER comparison using the Guava ComparisonChain builder, which is a very simple but effective way of building complex comparators, although in this case used for a single check.
There are two parts to the model, the IFS parent class which holds two lists of Transform and Reflection objects, and the CoordinateTransform enumeration which represents functions to be applied after each transform or reflection that will modify the coordinates in some way. All three of these implement the iterator.model.Function interface which is also a UnaryOperator<Point2D> that operates on Java 2D points. This allows the functions to be easily composed, as was seen earlier when looking at how the AtomicReferenceArray of points was updated.
Although the function interface has a getTransform() method returning an AffineTransform, this is only used in the default implementation of the apply(src) method of the operator interface. The coordinate transforms throw an UnsupportedOperationException instead, and implement the apply(src) method directly with their particular transformation.
This example is from the iterator.model.functions.Spherical class, and implements scaling by the inverse of the squared distance from the origin. All of these coordinate transforms are taken from the Fractal Flame paper and their id field is the variation number used in its appendix, where they are defined. Not all the variations are implemented, since some of them use coefficients from the previous affine transform matrix as input to the function, and this would prevent me from implementing them as operators and introduced too much complexity. However, in the future this may be an interesting addition.
The Chaos Game implementation involves maintaining a record of a point in the (X,Y) plane, and repeatedly choosing at random one of the transforms in the IFS model. The transform is then applied to the point, followed by the application of the coordinate transform. Note that there is an identity coordinate transform available if this second application is not desired. In the application, two points are maintained and transformed (using the same function for each) with the second being used to determine the colour, either by mapping the coordinates to a point in an image file (known as stealing if the file used is an image or photograph, or using a file containing a gradient fill generated from the config.getGradientStart() and config.getGradientEnd() colours) or by mapping to a point in HSV colour space, known as ifs-colour mode.
The Task.ITERATE threads simply execute the iteration repeatedly and if the mode does not involve density estimation, they plot a point in the required colour after each iteration. I perform colour correction and scaling in the HSB space, after converting from RGB, since this allows more useful control of the result. The unity() and octet() methods each return a function that clamps its input to a value in the range [0.0,1.0] or [0,255] respectively. Note that variables like hsb and rect are not defined here. This is because they are used in the middle of a loop that will be executed many millions of times. I decided to keep the object creation and destruction overhead outside the loop, so they are defined at the start of the method. The rect variable is used because I want to set some (X,Y) point to the required colour, but Java2D does not have the concept of a point as a displayable object. Instead, I use a one-by-one pixel sized rectangle, and fill it with the chosen colour, or for the live view of the IFS a two-pixel wide rectangle.
For the density estimation rendering modes, the Task.ITERATE threads simply carry out the iterations and update a series of arrays containing the density and the colour of points. The generation of the image is handled by the Task.PLOT_DENSITY thread. The density[] array is updated by incrementing its elements at each iteration, with the array index being calculated from the 2D coordinates. Because the process can involve a very large number of iterations, there is a danger of these density values overflowing, especially when using the Render.LOG_DENSITY_POWER rendering mode that not only increments, but then multiplies the density value to increase it by 1%. Fortunately Guava provides the LongMath.checkedAdd(a, b) method, which throws an ArithmeticException on overflow. This means I do not need to perform checks for negative values. Interestingly, the multiplication does not use this method, instead I convert the long to a double, perform the multiplication, and use Math.min(a, Long.MAX_VALUE) to clamp the result.
I will only show part of the plotDensity() method that performs the work for the Task.PLOT_DENSITY thread here. This code extracts RGB byte values for a particular point, using bitmasks, and then either scales them by a fixed ratio or uses the logarithm of the density at that point to perform colour scaling. This is based on ideas from the _Fractal Flame_ paper, hence the name of the rendering mode as Render.LOG_DENSITY_FLAME. Note that for the inverse mode we can simply subtract the calculated ratio from one. Here we also see how the gamma quantity is used, as the exponent in a call to Math.pow(a, b), to modify the ratio. After scaling the RGB values the colour is converted to HSB space and the brightness component is rescaled and the value component replaced by the calculated density ratio, and the same range clamping is applied as described above.
The colour manipulation code is probably the most important in terms of how the final rendered image appears, and small changes here can have large effects on the output. Unfortunately, the code is more complex than I would like, since it has to deal with conversions between double, float and long that obscure the intent of some calculations. There are definitely still issues present, and more eyes reviewing it would help immensely. For example, colours saturate to white or black too easily and alpha blending sometimes does not operate as desired, causing dithering and artifacts. Any new configuration parameter that affects the output will have to be checked and used here, which requires careful though as these methods are the hottest part of the code and adverse performance impacts would be easily created. I think that this is one of the most interesting parts of the codebase, however, and it is always enjoyable adding a new feature or tweaking a parameter and seeing the visual effects that produces.
Next, I would like to show how the Java 2D graphics APIs were used to display the IFS rendering on screen.
This code encapsulates performing some action using a Graphics2D context object, obtained from a passed in Graphics context and then disposed of after use. This pattern repeats many times in the code, with the graphics context coming from various places, perhaps a Swing component or a BufferedImage object. The action performed is passed in as a lambda, taking the Graphics2D context object as its argument. The exception handler implementing BiConsumer<Throwable,String> object is usually the Output class, which is obtained from the controller, but there are also utility methods that provide simple handlers that either log to System.out or re-throw a wrapped RuntimeException.
For example, here is how a new BufferedImage is filled with a particular colour.
And, during startup, I wanted to use a splash screen with the current version number displayed but without having to create a new bitmap image for each release. The -splash:./lib/splash.png option to the java executable (or SplashScreen-Image in the manifest file of a Jar) allows an image file to be loaded as a splash screen, and I use a PNG with a composite of the editor and viewer displays for this. Java allows access to this image with the SplashScreen.getSplashScreen() method, and the graphics context for the image can then be obtained. I use the same image and text overlay in the About dialog box, so it was simple to extract the code to add the text overlay to a method that accepts a Graphics2D context and re-use it during startup.
The most complex part of the image rendering for the IFS Explorer is actually the interactive editor view. The non-interactive viewer (and the IFS Renderer application) simply execute the iterate() method repeatedly, stopping after some number of iterations if configured, and copy the results from a BufferedImage to either a Swing component or writing to a PNG file. The editor must repeatedly render the IFS based on the current values of the transforms being modified. This is done using a Swing Timer that fires and executes an actionPerformed(event) method every 100ms to repaint the viewport. User interaction updates the properties of the Transform object until the mouse is released, then the transform is added to the IFS list of transforms and the model is pushed out to the eventbus by the controller.
The editing actions are intended to be reasonably intuitive and obvious for anyone who has ever used a paint ot vector graphics editing application. The transforms are displayed as rectangle outlines showing the result of applying their affine transform to a unit square, with grab handles at the corners for resizing and a circular handle at the top centre for rotation. Additionally the keyboard arrow keys can be used for movement, and on OSX trackpad gestures can be used to rotate and resize the transforms.
The resize action performed by dragging a corner of a transform must operate on all four corners, and if shift is held down must maintain the current aspect ratio. This needs to work correctly on rotated and sheared transforms, and to show that resize is possible the cursor should change when the mouse is moved over one of the resize handles. To do this, a Corner enumeration was created to represent the four compass directions Corner.NW, Corner.NE etc.
This represents the X and Y coordinates of the corners of a unit square, and the appropriate Swing cursor to use. The cursor displayed is not necessarily the same as the identity of the corner, in order to be more visually pleasing. Instead, the coordinates of the four corners are obtained, and then sorted first by their X component, then their Y component with the leftmost and topmost corner showing the Cursor.NW_RESIZE_CURSOR cursor and so on. It is simple enough to determine if the mouse is over a corner by transforming the point at the corner of a unit square with the affine transform and determining if a box centered on that new point contains the mouse location.
More complex is the code to decide how to resize the transform. Depending on the corner being moved we must either change the origin (the X and Y coordinates) of the transform or the size (the W and H values) and this must be done in the rotated and sheared reference frame of the transform. To help with this, unit vectors in the X and Y directions are inverse transformed, and their individual X and Y components are then used to calculate the changes to the transform. The case statement below implements this logic, with checks for e.isShiftDown() being used to maintain a fixed aspect-ratio.
The third view component of the IFS Explorer application is probably the simplest. It generates a display of the matrix coefficients for the IFS transforms, and the properties of the reflections. The iterator.view.Details class implementing this is essentially an HTML document renderer. It extends the Swing JTextPane component and uses anHTMLEditorKit with MIME type text/html to display its content. The content is created as CSS and HTML Strings which are then parsed by the Swing editor kit and displayed.
Notice that the CSS rules are split into two groups. The CSS_BRACKET_RULES collection are re-used in the dialog box for editing the matrix coefficients of an affine transform. This dialog is structured to look like a matrix displayed in a textbook, with square brackets on either side of the three-by-two array of numeric input fields. The brackets are formatted using these CSS rules and are actually the borders of a fixed-size HTML div.
One of the hardest things to implement in the UI turned out to be printing. The Swing printing mechanism is very sparsely documented, and there are multiple techniques possible. I wanted to allow users to print both the Viewer and Details components, and the first step is to implement the Printable interface and its print(graphics) method. The easiest printing mechanism to use was that provided by the PrinterJob class. This contains a static method that can be used to obtain an instance and then once the Printable component has been registered along with a PageFormat to use, the job.printDialog() method can be called. This will display the native operating system print dialog, and on successful return the job.print() method can be called to actually print the page. Unfortunately beyond being able to set the print job name, I was unable to work out how to accomplish tasks like setting a header or footer on the page. The printing code is called from a menu action, and is written as a lambda that is passed to the pauseViewer(action) method that will halt the iteration tasks if they are running, and restart them after the lambda has completed.
One aspect of the application that I wanted to ensure was cross-platform support. This is fairly straightforward with Java, since the compiled Jar files will run anywhere with a JVM implementation. However I also wanted to make the UI look-and-feel compatible with the different native platforms. I use a MacBook Pro as my daily work laptop, so most of the customisation has been for OSX, with Windows and Linux being left using standard Swing components for now. Some of the platform customisation happens in the launcher scripts, which are provided in the ./bin directory of the distribution archive. There are three versions; for IFS Explorer they are explorer.cmd for Windows explorer.sh for Linux and explorer.command for OSX.
Additionally I added a dependency on the orange-extensions library for compilation, which provides stubs for the Apple Java extensions and allows building on non OSX platforms. At runtime a check is made to determine the platform OS, and theiterator.AppleSupport class is loaded using reflection if possible. This class implements handlers for the Quit, About and Preferences… menu items which are provided by OSX, and in the UI set-up code these are only created explicitly on other platforms. This library also enables gesture support, which I use for rotation and magnification of transforms in the editor view. This turned out to be surprisingly simple to achieve, and the effect is that of a well-integrated native application.
I hope this journey through the codebase has been interesting, and perhaps also useful. If you want to explore the code on GitHub please do, I would be more than happy to review pull requests and fix any issues discovered.
Summary
So, what have I accomplished, and what else can be done? IFS Explorer achieved the goal of making an easy-to-use fractal creation and viewing application, allowing me to explore and generate a wide variety of Iterated Function System images. The addition of coordinate transforms from the Fractal Flame paper was an interesting exercise in translating equations from a scientific paper into working code, and the resulting images generated by the new feature were well worth the effort. Similarly, the work expended in writing the density estimation and colour management routines was paid back in exceptional and vivid fractals. I found that the simplicity of the editing interface encouraged prolonged intuitive exploration, meaning that images can be designed and iterated on quickly until something suitably eye-catching is arrived at.
The additional programs were made possible by the modular nature of the code and the decision to use the MVC architecture. IFS Renderer, in combination with the ability to save and reuse the entire configuration, makes it possible to generate high-resolution renderings of particular fractals. I hope to use this to create large poster-sized prints of my favourite images. IFS Animator has not proved as useful, but it is able to produce pleasing scripted animations of fractals. This application could do with more work, in particular a graphical user interface to design the animations, since writing the script file for the required changes can be tedious. It would also be good to generate the movie files directly using something like JMF (the Java Media Framework) or one of the Java implementations of ffmpeg, rather than outputting thousands of PNG files.
There is scope for more applications using the same framework as well, for example a random IFS generator that continually creates new combinations of transforms and configuration and renders them, to be used as a screen-saver or background display. Or, the process of generating an image of an IFS could be implemented as a RESTful web service, with a JavaScript front end that would behave in a similar way to the current IFS Explorer UI. This would make the application much more immediately accessible, and the rendering, configuration and model code would be essentially unchanged. A similar process could be used to build an Android mobile application, since Java is one of the supported native languages. The porting process would require careful thought and design of the UI and UX for the new platform, however.
Another interesting project to work on, but one that would be much more complex, is that of extending the IFS rendering to three dimensions. Affine transforms can be easily extended to 3D by increasing the size of the matrix, and Java has 3D APIs, so the framework for rendering should be straightforward enough. The complexity arises when you think about the user interface. Extending the concept used in IFS Explorer, we would need the ability to manipulate a transformed unit cube in space, with at least six degrees of freedom. Probably this would require virtual reality or some other 3D display interface to be really intuitive and usable.
For version 2.0.0 of Iterator I would like to at implement least some of the above improvements to IFS Animator, and also to continue working on the rendering modes for IFS Explorer. Perhaps not moving to 3D, as that would better suit a completely new application, however there are plenty of variations of the coordinate transform function left to implement. As noted, the multi-threaded performance could probably be improved, perhaps using some JNI library to take advantage of Intel CPU vector instructions.
The documentation is another area that needs more work, starting with a description of the animation DSL and also more details of the different rendering modes and transforms with graphical examples of each. The code itself should also be documented, with the JavaDoc output being included in the documentation website. Finally it would be good to make it easier for users to run the applications and this requires some work on the code signing for the JNLP launcher. A Docker image that runs the Linux version of IFS Explorer over VNC would also make it simple to get started.
Lessons
What do I think I have learned from this multi-year project? An important take-away is that it has been fun. I don’t think I would have been able to sustain interest in a project for so long unless it was enjoyable. Secondly, the rendered IFS images are visually stunning in many cases, and the ability to explore the space of possibilities interactively is an enjoyable and interesting process. I have also learned a lot about Java Swing and the Java 2D APIs for creating desktop applications, which has been an interesting change from my day job dealing with web services and applications. The concurrency issues have helped me think constructively about multi-threaded performance, which is a useful skill to have. And the exercise of mathematical skills that had grown very rusty since undergraduate classes at University has helped keep them up-to-date and accessible at work.
I still have some concerns and questions about the performance and the correctness of the multi-threaded code. For example, what are the consequences of the shared state being accessed (such as the density[] array) by multiple threads? Is it really necessary for the thread cancellation checks to use an AtomicBoolean or would a primitive boolean have sufficed? Could use of locking or private ThreadLocal state help speed things up? What about using multiple BufferedImage objects, one per thread, and compositing them together for display?
One thing I think I should have worked harder on in this application is testing. Many times I have implemented changes, only to see the output of the rendered IFS looking completely broken and wrong. The lack of unit tests is partly due to the difficulty of testing a Swing UI, but I expect that individual components could be tested sufficiently if broken out into smaller units, and the existence of classes like the AWT Robot would probably help.
I would definitely recommend starting your own side project like Iterator. Find something that looks fun and engaging and dive in. Applications that generate pretty pictures or produce some other kind of interesting output are probably the best choice, particularly if, like Iterated Function Systems, it is easy to get started and produce something but there is also a huge expanse of complexity and nuance to explore.
As a simpler first step, have a look at the Iterator codebase on GitHub and perhaps think about implementing one of the features I discuss above, or fixing one of the many bugs that are still present. I would be happy to review pull requests and respond to issues. The important thing is to do something that interests you.
References
The following Wikipedia links , papers and books give more background on Iterated Function Systems and their mathematics and implementation. In particular, the Draves and Reckase paper on the Fractal Flame algorithm was very useful for inspiration when implementing rendering modes. The Barnsley book is the classic reference, and I used this as the initial source for the Chaos Game algorithm and the palette based Stealing rendering modes.
- Iterated Function System; Wikipedia
- Affine Transform; Wikipedia
- Construction of fractal objects with iterated function systems; Demko, Stephen and Hodges, Laurie and Naylor, Bruce; SIGGRAPH Computer Graphics, Volume 19, Number 3, 1985
- The Fractal Flame Algorithm; Draves, Scott and Reckase, Eric; 2008
- Superfractals: Patterns of Nature; Barnsley, Michael F; Cambridge University Press; 7 Sep 2006; ISBN 978‐0521844932
- The Computational Beauty of Nature: Computer Explorations of Fractals, Chaos, Complex Systems and Adaptation; Flake, Gary W; MIT Press; 1 Mar 2000; ISBN 978‐0262561273
