Let it snow!

Square Cash makes gift giving a little more festive.

Written by Bob Lee.

If you sent or received Square Cash recently, you might have noticed slight precipitation backdropping our email headers. It all started last week when we decided to do something special for the holidays. I reckon my teammates thought I was joking when I shouted out with glee, “falling snowflakes!” But then I followed up with a working prototype. What says “Happy Holidays!” better than parallax-scrolling, alpha-composited, bokeh snowflakes, amirite?

While the effect is simple, generating it was anything but. Hacking together a proof-of-concept took moments. Profiling and optimizing the rendering speed and file size required orders of magnitude more effort. We didn’t draw this animation in Photoshop. I wrote a couple hundred lines of Java code instead. The challenges were many, but we Square engineers pride ourselves on our ability to push the envelope at every layer of the stack, up to and including email animations.

Most email clients don’t support HTML5. We’re stuck with 90s era technologies, or in this case, animated GIFs. Obviously the animations should look crisp and beautiful, especially on today’s high resolution displays. We render them at 2X resolution — more than 300k pixels per frame. The animations include custom text. This requires us to render them on the fly and poses some fun challenges. Once rendered, the animation needs to download and play quickly, even over slow mobile networks.

To start, I pored over the 25-year-old GIF spec, reading it backwards and forwards, looking for opportunities to cut down the file size without compromising the design. I used pngquant’s best-of-breed Median Cut quantization algorithm to precompute an optimal 16-color palette. Restricting the animation to 16 colors enabled us to encode the images using 4 bits-per-pixel[^1]. It also resulted in slight posterization, promoting color repetition and further improving compression, particularly in the gradient at the bottom of the image. I even tried plotting only the pixels that changed between each frame, but to my surprise, the diffs resulted in more complexity, less repetition, and therefore worse compression than the full frames.

The biggest win came from an optical illusion aimed at reducing the total number of frames. The animation looks like it necessitates 240 frames, but it really only requires 60 (a 75% savings). The animation is composed of four layers of snowflakes, each meant to appear a different distance from the observer. As the layers get further away, the snowflakes get smaller, lighter, slower, and denser, resulting in the illusion of depth. Here are the separate layers ordered closest to furthest:

The trick is only the closest layer travels the full height of the frame. The lower layers travel only a fraction of the height, but they tile and repeat, giving the illusion of continuity. For example, during the course of the animation, the second layer travels only half way down the height of the image, but it repeats twice and looks like it scrolls continuously. The third layer travels one third of the way and repeats three times, and the fourth layer travels one fourth of the way and repeats four times. As you can see, this repetition becomes less evident when you layer the snowflakes on top of each other and animate them at different speeds.

The final animation clocks in at less than 650KB — **0.25 bits/pixel!** To see it in action, send some Cash to a loved one. And check out the final code below. Consider it my gift to you! Happy holidays.

1 /*
2 * Copyright 2013 Square Inc.
3 *
4 * Licensed under the Apache License, Version 2.0 (the "License");
5 * you may not use this file except in compliance with the License.
6 * You may obtain a copy of the License at
7 *
8 * http://www.apache.org/licenses/LICENSE-2.0
9 *
10 * Unless required by applicable law or agreed to in writing, software
11 * distributed under the License is distributed on an "AS IS" BASIS,
12 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13 * See the License for the specific language governing permissions and
14 * limitations under the License.
15 */
16 package com.squareup.franklin.email.image;
17
18 import com.google.common.cache.CacheBuilder;
19 import com.google.common.cache.CacheLoader;
20 import com.google.common.cache.LoadingCache;
21 import com.google.common.io.ByteSource;
22 import com.google.common.io.ByteStreams;
23 import com.google.common.io.Files;
24 import com.squareup.common.locale.ISOCurrency;
25 import com.squareup.common.values.Money;
26 import java.awt.Color;
27 import java.awt.GradientPaint;
28 import java.awt.Graphics2D;
29 import java.awt.Paint;
30 import java.awt.RenderingHints;
31 import java.awt.image.BufferedImage;
32 import java.io.File;
33 import java.io.IOException;
34 import java.util.ArrayList;
35 import java.util.List;
36 import java.util.Random;
37 import javax.imageio.IIOImage;
38 import javax.imageio.ImageIO;
39 import javax.imageio.ImageTypeSpecifier;
40 import javax.imageio.ImageWriteParam;
41 import javax.imageio.ImageWriter;
42 import javax.imageio.metadata.IIOInvalidTreeException;
43 import javax.imageio.metadata.IIOMetadata;
44 import javax.imageio.metadata.IIOMetadataNode;
45 import javax.imageio.stream.ImageOutputStream;
46 import javax.imageio.stream.MemoryCacheImageOutputStream;
47 import org.apache.commons.io.output.ByteArrayOutputStream;
48 import org.w3c.dom.Node;
49
50 /**
51 * Generates animated header images for the holidays.
52 *
53 * @author Bob Lee
54 */
55 public class HolidayHeader {
56
57 /*
58 * Docs for Java's GIF encoder:
59 *
60 * - Plugin notes: http://docs.oracle.com/javase/6/docs/api/javax/imageio/package-summary.html#gif_plugin_notes
61 * - Metadata spec: http://docs.oracle.com/javase/6/docs/api/javax/imageio/metadata/doc-files/gif_metadata.html#gif_stream_metadata_format
62 *
63 * To inspect GIF file structure:
64 *
65 * - Daktari: http://interglacial.com/pub/daktari_gif.html
66 */
67
68 private static final int FRAMES_PER_SECOND = 24;
69
70 /** Random seed used when positioning flakes. */
71 private static final int SEED = 6;
72
73 /** Number of flake layers. */
74 private static final int LAYERS = 4;
75
76 /** Number of flakes in the top layer. */
77 private static final int FLAKES_IN_TOP_LAYER = 8;
78
79 private static final int MAX_FLAKE_RADIUS = radiusFor(LAYERS - 1);
80 private static final int HEIGHT_WITH_PADDING = HeaderImage.HEIGHT + MAX_FLAKE_RADIUS * 2;
81
82 private static final HeaderImage.Color COLOR = HeaderImage.Color.RED;
83
84 /** Constants used for gradient at the bottom of the header. */
85 private static final Color TRANSPARENT_BACKGROUND_COLOR =
86 new Color(COLOR.backgroundColor().getRGB() & 0x00ffffff, true);
87 private static final int GRADIENT_HEIGHT = 200;
88 private static final GradientPaint GRADIENT = new GradientPaint(
89 0, HeaderImage.HEIGHT - GRADIENT_HEIGHT, TRANSPARENT_BACKGROUND_COLOR, 0,
90 HeaderImage.HEIGHT, COLOR.backgroundColor());
91
92 /** Change in y per frame for top layer of flakes. */
93 private static final int DY = 10;
94
95 /** The number of frames required for the highest layer of flakes to cycle once. */
96 private static final int FRAME_COUNT = HEIGHT_WITH_PADDING / DY;
97
98 /** Frames without the amount overlaid. */
99 private static final List<BufferedImage> FRAMES = new ArrayList<>();
100
101 static {
102 for (int frameIndex = 0; frameIndex < FRAME_COUNT; frameIndex++) {
103 Random random = new Random(SEED);
104
105 // Start with a blank red frame.
106 BufferedImage frame = newBuffer();
107 Graphics2D g = frame.createGraphics();
108 g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
109 g.setColor(COLOR.backgroundColor());
110 g.fillRect(0, 0, HeaderImage.WIDTH, HeaderImage.HEIGHT);
111
112 // Draw snowflakes in 3D! We draw the snowflakes in layers, starting with the most distant
113 // layer. Nearer layers are bigger, faster, brighter, and more sparse, just like real life!
114 // To reduce the number of frames–and in turn the file size–lower layers don't scroll the
115 // entire height of the image. We tile them vertically so they still scroll continuously.
116 // For example, layer 3 moves only 1/3 of the way down during the animation, but we tile it
117 // three times so the last frame looks just like the first frame.
118 for (int layerIndex = 0; layerIndex < LAYERS; layerIndex++) {
119 // We add some padding to the top and bottom and draw outside the bounds of the physical
120 // image so snowflakes don't magically appear at the top and disappear when they touch the
121 // bottom.
122 g.setColor(new Color(255, 255, 255, (layerIndex + 2) * 25));
123 int tiles = LAYERS - layerIndex;
124 int tileHeight = HEIGHT_WITH_PADDING / tiles;
125 int radius = radiusFor(layerIndex);
126 int diameter = radius * 2;
127 for (int flakeIndex = 0; flakeIndex < FLAKES_IN_TOP_LAYER; flakeIndex++) {
128 int x = random.nextInt(HeaderImage.WIDTH);
129 int y = random.nextInt(tileHeight);
130 y = y + frameIndex * DY / tiles;
131 for (int copy = 0; copy < tiles; copy++) {
132 g.fillOval(x - radius, (y + copy * tileHeight) % HEIGHT_WITH_PADDING - MAX_FLAKE_RADIUS,
133 diameter, diameter);
134 }
135 }
136 }
137
138 // Draw Square logo.
139 g.drawImage(HeaderImage.LOGO, null, HeaderImage.LOGO_X, HeaderImage.LOGO_Y);
140
141 // Fade to red at the bottom.
142 Paint paint = g.getPaint();
143 g.setPaint(GRADIENT);
144 g.fillRect(0, HeaderImage.HEIGHT - GRADIENT_HEIGHT, HeaderImage.WIDTH, HeaderImage.HEIGHT);
145 g.setPaint(paint);
146
147 FRAMES.add(frame);
148 }
149
150 /*
151 * Note: I tried storing the differences between frames instead of the entire frames, but
152 * the resulting file was actually bigger. This is because the diff images are more complex
153 * than the full images and are therefore less compressible.
154 */
155 }
156
157 private static BufferedImage newBuffer() {
158 // Use 4 bits per pixel (i.e. BINARY, not INDEXED).
159 return new BufferedImage(HeaderImage.WIDTH, HeaderImage.HEIGHT, BufferedImage.TYPE_BYTE_BINARY,
160 COLOR.model());
161 }
162
163 private static final IIOMetadata FIRST_FRAME_METADATA = newFrameMetadata(true);
164 private static final IIOMetadata FRAME_METADATA = newFrameMetadata(false);
165
166 private static ImageWriter newGifWriter() {
167 return ImageIO.getImageWritersByFormatName("gif").next();
168 }
169
170 private enum DisposalMethod {
171 none, doNotDispose, restoreToBackgroundColor, restoreToPrevious
172 }
173
174 private static IIOMetadata newFrameMetadata(boolean first) {
175 ImageWriter writer = newGifWriter();
176 ImageWriteParam iwp = writer.getDefaultWriteParam();
177 IIOMetadata metadata = writer.getDefaultImageMetadata(
178 new ImageTypeSpecifier(newBuffer()), iwp);
179 String metaFormat = metadata.getNativeMetadataFormatName();
180 Node root = metadata.getAsTree(metaFormat);
181 IIOMetadataNode gce = (IIOMetadataNode) findChild(root, "GraphicControlExtension");
182 gce.setAttribute("userDelay", "FALSE");
183 gce.setAttribute("delayTime", String.valueOf(100 / FRAMES_PER_SECOND)); // hundredths of a sec.
184 gce.setAttribute("disposalMethod", DisposalMethod.doNotDispose.name());
185 if (first) {
186 // Use the Netscape application extension to enable looping.
187 IIOMetadataNode aes = new IIOMetadataNode("ApplicationExtensions");
188 IIOMetadataNode ae = new IIOMetadataNode("ApplicationExtension");
189 ae.setAttribute("applicationID", "NETSCAPE");
190 ae.setAttribute("authenticationCode", "2.0");
191 // Loop infinitely.
192 ae.setUserObject(new byte[] { 0x1, 0, 0 });
193 aes.appendChild(ae);
194 root.appendChild(aes);
195 }
196 try {
197 metadata.setFromTree(metaFormat, root);
198 } catch (IIOInvalidTreeException e) {
199 throw new AssertionError(e);
200 }
201 return metadata;
202 }
203
204 private static Node findChild(Node root, String name) {
205 Node child = root.getFirstChild();
206 while (child != null) {
207 if (name.equals(child.getNodeName())) return child;
208 child = child.getNextSibling();
209 }
210 throw new AssertionError();
211 }
212
213 private static int radiusFor(int layer) {
214 return 8 + (layer * 4);
215 }
216
217 private static final LoadingCache<Icon, ByteSource> headersWithIcons =
218 CacheBuilder.newBuilder().build(new CacheLoader<Icon, ByteSource>() {
219 @Override public ByteSource load(Icon icon) throws Exception {
220 return withOverlay(HeaderImage.readImage(icon.fileName()));
221 }
222 });
223
224 /** Creates a header image with the given icon. */
225 public static ByteSource withIcon(Icon icon) {
226 return headersWithIcons.getUnchecked(icon);
227 }
228
229 /** This cache will use ~700MB of memory. */
230 private static final LoadingCache<Money, ByteSource> headersWithAmounts =
231 CacheBuilder.newBuilder().maximumSize(1000).build(new CacheLoader<Money, ByteSource>() {
232 @Override public ByteSource load(Money amount) throws Exception {
233 return withOverlay(newAmountImage(amount));
234 }
235 });
236
237 /** Creates a header image with the given amount. */
238 public static ByteSource withAmount(Money amount) {
239 return headersWithAmounts.getUnchecked(amount);
240 }
241
242 /** Creates a header image with the given image overlaid. */
243 private static ByteSource withOverlay(BufferedImage overlay) {
244 try {
245 int x = (HeaderImage.WIDTH / 2) - (overlay.getWidth() / 2);
246 int y = 200;
247
248 ByteArrayOutputStream bout = new ByteArrayOutputStream();
249 ImageOutputStream iout = new MemoryCacheImageOutputStream(bout);
250 ImageWriter writer = newGifWriter();
251 writer.setOutput(iout);
252 writer.prepareWriteSequence(null);
253
254 boolean first = true;
255 BufferedImage buffer = newBuffer();
256 Graphics2D g = buffer.createGraphics();
257 g.setRenderingHint(RenderingHints.KEY_ANTIALIASING, RenderingHints.VALUE_ANTIALIAS_ON);
258 for (BufferedImage frame : FRAMES) {
259 g.drawImage(frame, null, 0, 0);
260 g.drawImage(overlay, null, x, y);
261
262 // All of our frames share the same color table. Because we explicitly set per-frame
263 // metadata without including a local color table, Java won't emit a local color table.
264 // Java will automatically use the first frame's color table as the global color table.
265 IIOImage ii = new IIOImage(buffer, null, first ? FIRST_FRAME_METADATA : FRAME_METADATA);
266 writer.writeToSequence(ii, null);
267 first = false;
268 }
269
270 writer.endWriteSequence();
271 iout.close();
272
273 return ByteStreams.asByteSource(bout.toByteArray());
274 } catch (IOException e) {
275 throw new AssertionError(e);
276 }
277 }
278
279 private static BufferedImage newAmountImage(Money amount) {
280 Dimensions textDimensions = new Dimensions(440, 218);
281 BufferedImage amountImage = new BufferedImage(
282 textDimensions.width(), textDimensions.height(), BufferedImage.TYPE_INT_ARGB);
283 Graphics2D g = amountImage.createGraphics();
284 AmountImage.drawAmount(g, 0, 0, amount, textDimensions, Color.WHITE);
285 return amountImage;
286 }
287 }

[^1]: It may sound counterintuitive, but using 4 bit-per-pixel can result in a larger file size than using 8 bits-per-pixel. Which encoding is better depends on which bit width produces more repetition in the bytes, and that depends on the nature of the image. The easiest strategy is to try both and see which results in a smaller file.