Cecilia's Drawing Toy: Updated for Little Hands

You learn a lot about proper software design by actually watching users. This is a well-known fact, but sometimes a reminder is helpful.

Cecilia was in town for several weeks, so I got a chance to watch her playing with her drawing toy. I noticed that she preferred to hold the phone tightly with one thumb, which really threw off the motion tracking.

A few changes later, I added support for multiple-touch operations. If little hands need to grab the screen in two places, that's not a problem!

It's also good for finger-painting. :)

As always, the updated version is available on the Play Store if you're not in the mood to mess around with compiling your own.

Cecilia's Drawing Toy

My sister- and brother-in-law recently celebrated the birthday of their second daughter, Cecilia. She's not quite old enough to care about the touchscreen devices yet, but I figured I'd get a head-start on this one. So here you go: Cecilia's Drawing Toy.

Much like its "sister" software, Frances's Tracing Game, this one is available for download on the Play Store, or you can poke through the source code on GitHub. You can select a color and draw with your finger, then to erase, hold the whole device upside-down and shake it (like another drawing toy you might have heard of ;) ).


I was able to re-use quite a bit of the source code from Frances's Tracing Game for this one, so the build cycle wasn't quite as long. I feel like I'm pretty much getting the hang of the drawing toolkit for Android; this one should play more nicely than the first versions of Frances's Tracing Game did with different screen sizes and form factors. It's missing a way to save images right now (but you can do so with the "screenshot" feature on most phones).

If you thumb through the source code, you'll find some goodies. It's all licensed under Apache, so feel free to grab any pieces and use them in your own projects (with proper sub-licensing and attribution). In particular, these modules are broken out and intended for re-use:

• RandomSound: Give it a collection of sound resources and it will select, load, and play random ones continuously. Useful for footsteps, etc. There's some lag between the end of one and the beginning of the next that I should clean up.
• OscillationSensor: Detects back-and-forth jerks in a linear direction. Useful for detecting device shaking.
• FaceDownSensor: Detects when the device's screen is angled towards the floor, and sets a simple flag that the render thread can check.

As always, I dug up some likes and dislikes as I put this together.

Things I Liked

1. Code re-use worked as intended. :) Both the initial layout of the project and the push to the Play Store were a lot simpler with templates ready to go.
2. Instead of wrestling with Windows to build this one, I put together a development environment in a virtual machine running Debian Linux (with my favorite tool suite---emacs, git, and the command-line SDK---installed from packages and ready to go). It turned out to be a smoother development environment, to my mind; I spent less time thrashing about with the command line and more time writing code.
3. Build-test cycle on a connected Android device is super-simple. I just kept "adb logcat" running in one console to spot-check installation failures, had "ant debug && ant installd" as my compilation command, and off I went.

Issues

1. The timestamp on accelerometer events is extremely wonky. As I discussed previously, I eventually ignored them and tagged every incoming sensor event with a datetime as the sensor thread got ahold of it; not accurate, but sufficient for OscillationSensor.
2. The Play Store makes you (hypothetically) put together a lot of images and material for a proper launch. Screenshots for phone, for tablet, for 10" tablet, promo image, icon, big super-awesome icon if you end up in the "Featured" section... It'd be cool if at least some of that could be dynamically generated. I spent more time preparing and uploading PNGs than I did writing RandomSound.
3. 
   

Future Ideas

This'll be a fun one to keep updating, but really, the list of new things is small on this one:

• save the current image when the app quits or is killed
• allow for mixing of your own marker colors
• eraser (besides holding upside down and shaking)
• undo

Enjoy!

If you have a little one in your life, feel free to grab the toy and let me know if they like it! Feedback is welcome on the game's Play Store page, here, or on Plus. Go make something cute. ;)

Quirkiness of Android sensor library timestamps

Update 2014-06-23: Original post doesn't quite work as intended. Details at the end of the post.

I'm working on a sensor library that should give me the last timestamp at which some interesting sensor event happened.

So, the Android sensor library is a little quirky. When you get a sensor event (on a sensor thread, so it is not recommended that you do any high-impact processing there), the event has an associated timestamp. The timestamp is documented thusly: "The time in nanosecond at which the event happened". The grammatically-bizarre English should be your clue that these docs may be slightly off. ;)

It turns out after a bit of Googling (tip o' the hat to StackOverflow, as usual) that the timestamp one receives isn't based off of any particular 0-point defined in the Android OS or the API; it's an arbitrary per-sensor value intended to allow for different measurements from the same sensor to be compared, not for the measurements to be compared to other timestamped events. It's a known issue (bug concerning the documentation; bug concerning the behavior), but as of right now it's the way of the world for Android developers.

If you need to obtain reference to some external timeframe (such as the oh-so-ubiquitous "milliseconds since 1970"), you'll have to put together your own workaround. The solution that's working for me right now is:

1. Initialize your sensor listener with empty offset values dateBase and timestampBase.
2. For each received sensor value (i.e. each call to onSensorChanged), check if your offsets are empty
3. If they are, make a one-time call to populate them:
1. grab a time pretty close to when the sensor event was fired via dateBase = (new Date()).getTime()
2. Record timestampBase = event.timestamp from the new sensor event.
4. To log the milliseconds-since-1970 timestamp of new events, retain (event.timestamp - timestampBase) / 1000000L + dateBase

This calculation is prone to fail if for some reason onSensorChanged gets called too long after the sensor change occurs (it's fine if the difference is on the order of nanoseconds and gets significantly less fine in the 100ms-second range or higher). But it works for my needs.

Update: So it turns out that this approach also doesn't work as desired; if the phone goes to sleep, the sensor clock will skew relative to wall time (I assume the sensor clock isn't "ticking" while the phone is not active). I could detect sleep / wake events, but it becomes a guessing game regarding what scenarios could cause the phone to stop updating the sensor clock. My new plan is to do (new Date()).getTime on every sensor event and take the simple performance hit; for my purpose, I don't need highly-accurate sensing.

Frances's Tracing Game v1.04: On Coordinate Transforms

As one bug dies, another is born. So it goes, so it goes. :)

As  I prepared to publish version 1.03 of Frances's Tracing Game, I took a glance at the statistics and suggestions from the Google Play Store and noticed that many newer Androids couldn't pick up the latest build because I had the minimum SDK version set to 1. I forked the project (Play Store lets you publish multiple binaries under the same app depending on the version of the client's phone), then in the fork I bumped the minimum up to SDK 4 and targeted SDK 11.

That introduced a new oddity: Apparently, my game had been running in a compatibility mode this entire time (due to its claim that it targeted SDK 1); as a result, the screen had been scaled automatically to fit the ancient Android form factor seen in the days of the Nexus 1. When I bumped to SDK4, the compatibility mode went away and my shapes were suddenly far too small to be traced by even clever fingers (screen resolution has improved a lot since the Nexus One days). The Android screen format rules have an awful lot of complexity to them, but to make a long story short: it is best to assume that your client's device could fit just about any rectangular form factor, much like we do with desktop PC programming.

Fortunately, this is an old problem with some old solutions. Many graphics libraries (such as OpenGL) provide a layer of transformations to make it easy to convert from one coordinate space to another. Android's graphics SDK is no exception; the Canvas object supports scaling and translation methods to tweak the underlying matrix that maps pixels in the canvas to pixels on the screen. I'd already used translation previously to center the image; now I just need to use scaling to balloon the image to fit the width of the screen.

Here's the basics of the tweaks I made; if you want to see the full code, it's hosted on GitHub.

Step 1: Tweak the scale

Tweaking the scale is extremely straightforward in Android; there's a pair of scale() methods that wrap the matrix multiplication logic for convenience (note: if you haven't done graphics programming, matrix algebra is basically the bread and butter of shifting coordinate spaces; Wikipedia can get you started on the concepts). I'm using this method, which includes the concept of a pivot point (the point around which the scaling should occur) to scale from the center of my image.

The only interesting part is determining how much to scale. I want to take up about 80% of the space, so I'm solving for S in the equations

final_image_width = 80% * screen_width

final_image_width = S * initial_image_width

This resolves to S = 80% * screen_width / initial_image_width, which I store and use as the scaling factor to rescale the canvas. Since the canvas itself is scaled, the image, line thickness I use for drawing, and pink tracing overlays are all correctly rendered without having to change the rest of the drawing logic.

Step 2: Unscale the touch

The only remaining issue is that the touch events we receive don't go through the rendering logic to be scaled. So for touch events we get from the screen, we need to reverse the scaling operation to get them back into the image coordinate space to determine if they were close to one of the tracing lines. I could have done this by grabbing the canvas's matrix with getMatrix() and inverting it (for a given transformation matrix, inverting the matrix gives you the reverse transformation). But I got a little bit fancy instead; the steps to reverse the scaling of the point (factoring in the pivot) are basically as follows:

1. subtract the pivot coordinates from the point coordinates (this would translate a point on the pivot to (0,0), which is what you want because that point doesn't move under scaling).
2. Multiply the point coordinates by 1/scale_factor.
3. Add the pivot coordinates back to the point coordinates (reversing the translation).

It's mathematically simple enough that I think it's clearer to just represent it directly without the matrix (though just using the matrix inverse might very well have been less code!).

The Result

I am not unhappy. :)

I'm pretty happy with the result. Frances's game looks good on a 7-inch tablet and on a smartphone. I also added a text label for each picture, because she's getting older and is starting to read and write. I passed the labels through the canvas transformation, which leads to some odd scaling that's dependent upon the underlying size of the traced image; I might make some time to clean that up later. 

Feel free to download it to your own device if you'd like; it's free and will never include ads (because what possible use could ads be in a toddler's tracing game other than to junk-up the advertiser's signal with unintended clicks?).

I have no idea if this simple game will hold her interest much longer; it may soon be time to make her something new. And there's still the question of what to make for Cecilia, but there's a bit of time there; she's not very interested in tablets right now because she just realized that when she kicks her legs, her bounce-chair moves. I have to admit, that's pretty cool.

Frances's Tracing Game v1.03

I love three-day weekends. They give you time to think!

I've dusted off Frances's tracing game and finally fixed a niggling bug that had been bothering me. I'd taken a shortcut on centering the images on the screen by putting the image in a view that was embedded within a larger view. That solution had been playing passably, but it had annoying side-effects in Jellybean:

1. Touch-drags that started outside the image didn't register when you dragged inward.
2. As the drag operation happened, an annoying grey rectangle would flicker around the frame of the inner view. I never figued out the root cause of that.

I therefore ditched the two-view solution and did the math internally to set up the images in the center of the view (there's a convenient Canvas.translate method to do the offsetting for me in terms of drawing, and I do the translation of touch events from view coordinates to image coordinates myself).

There's a lesson in this. Sometimes, an operating system changes and it becomes clear that the OS's developers don't have your use case in mind. When that happens, there are a couple of ways you can respond. But when you're working on a hobby project and don't have the clout to steer the OS project itself, it's often easier to adapt to the changes than it is to push back. Don't be afraid to replace someone else's functionality with your own code. But when you can contribute, do; the svg-android renderer instance that I use seems to have issue with some flavors of transform logic, and if I don't find that this has been fixed in later versions, I plan to patch it and submit the changes back to the maintainers.

The new version of Frances's Tracing Game should be available in the app store some time today.

On a related note, there's one more bug to fix... I'm going to have to figure out what to do about the title. Frances herself is growing out of this game, but her younger sister was born this week. Welcome to the world, Cecilia!

Update to Frances's Tracing Game

Frances got a chance this weekend to play with her tracing game some more. She uncovered a bug I hadn't noticed in testing, and I noticed some trouble she was having. So I pushed an update.

Frame bug fix

A bit of inside-ball: under the hood, the traceable image is a custom View embedded in a LinearLayout, with the "gravity" on the LinearLayout set to center the embedded view horizontally and vertically. If Frances starts a trace by putting her finger down outside of the embedded view (i.e. inside the LinearLayout), the Android event system doesn't pass the event to the embedded view, so the trace doesn't get detected even as she moves her finger over the figure. This is perfectly understandable, correct by the UI standards, and 100% wrong for this application. I had to fish around a bit (there are a couple of options that look like they fix this issue and don't quite fit), but in the end the solution was to set the embedded view as the touch listener for the LinearLayout (i.e. embedded view implements OnTouchListener) and translate the coordinates from the LinearLayout's reference to the embedded view's reference using MotionEvent.offsetLocation.

Detect All Points

For simplicity, I'd been using only the first touch coordinate (the output of getX() and getY()) as the tracing location. However, watching Frances play the game, I noticed something about the way she holds the phone: rather than cradling the back, she usually grasps it from the side, which leaves her palm resting on the screen. Naturally, his plays havoc with the touch UI. Fortunately, the fix was simpler than the frame bug: I just read all the touch coordinates and try them all. If you put your palm on the image, that might count as a trace; I'm sort of fine with that, really.

Final Thought: On Users

Watching my niece play this game has been a great motivator for me to fix up the quirks. It's one thing to know you have thousands of anonymous users somewhere out in the world; it's quite another to have a two-year-old sitting in front of you straining her mind to try and understand a concept-breaking bug in the behavior of the program. The frame bug was driving her crazy, but she didn't want her uncle to do it for her; she was insistent on figuring it out herself. If she's going to be a trooper about it, the least I can do is clean up the last 2% of bugs in the toy.

By the time you read this, the latest version should be available on the Play Store. I didn't want to release just a bug update, so there's a couple of new figures in there too. Have fun!

Frances's Tracing Game (or: How I Spent My Summer Vacation)

About the time my niece was first learning to walk, I showed her pictures of our dog on my Android. Within a few weeks, she was asking for the phone every time I stopped by. It was fascinating to watch how quickly she learned the basic controls, such as how to go back and forth in the photo roll and play movies.

I went fishing for some age-appropriate games to put on the phone, and found a couple. One of her favorites is Doodle Toy, a delightful little drawing game. She really enjoyed that, but at some point it went ad supported (not great in a toddler app).

Inspired by the time I taught her to draw circles in Doodle Toy, I played around for a few weeks in a very lackadaisical fashion and put together my first real Android App. It's a very simple tracing game; glide your finger over the black stencil to turn it pink. A little congratulatory sound cue plays when you trace the whole image, and the next one appears. There are only six images; no randomized order, and they loop forever.

Now that the project is done, I have some time to decompress and take stock. Here's a brief tour through the process: What I used, what went smooth, what got bumpy, and what I'd like to do in the future.

The Build Environment

Configuration of an Android project was pretty simple. I started at developer.android.com and basically followed the steps, the key ones being installation of the Android development environment and ant1.8. My primary machine was a Windows Vista box, but for the finishing touches I used a small laptop running Ubuntu Lucid Lynx 10.04. The Ubuntu box was actually a little quicker to set up (with the one exception of the need to switch in ant1.8 instead of the regular ant package; no need to build from source, just a different package). 

In lieu of using Eclipse, I opted for the command line tools and emacs. What I lost in autocompletion and refactoring support I'm pretty sure I made up for in portability, small footprint (I dread the notion of running Eclipse on my tiny laptop), and nearly-nonexistant garbage collection cycles in my IDE. 

The only other issue I ran into was a gotcha that multiple people have reported about the tool suite in Ubuntu: the first time you "ant installd," you're likely to get a permissions error because adb running as a user probably can't access the devices it needs to talk to your Android. The easiest (maybe not so secure ;) ) workaround is to sudo killall adb, then sudo adb devices to force the daemon to run (and stay running) in root. A maybe more secure and less hackish option is given here, but I haven't tried it.

Writing the App

The app is a bit of a hackjob, and it shows... Quite literally, as you can browse the source code here. There's a lack of division of interest (Traceview.java is pretty overloaded to do not just rendering and events, but state change from image to image also). I think there's a fine line one can walk between too much modularity and too much complication; the rule of thumb I use these days is to hack a solution in first, then refactor later. You'll also notice a lack of test cases; the project is simple enough that end-to-end testing takes about thirty seconds, so this hasn't been an issue. Topic for another day: I also find unit testing of essentially visual / event apps a real chore on most architectures, and I haven't yet gone looking to see if there's a clean tool for such things on Android.

One thing I'm pretty happy about is standardizing on SVG for the image format. It really freed me up in the test-and-iterate cycle, as I was able to lean on the svg-edit demo to do all my trace data. I'm pretty enamored with SVG for something this simple; the ability to pop back and forth between visual and text representation of the data turned out to be pretty awesome. I also lucked out and found this awesome SVG to Android Picture converter library, which is Apache licensed. I hacked some specific tweaks into it for my needs, but it was already a 95% fit for my project and saved me the drudgery of hooking up an XML parser (which, I'm told, causes the death of a kitten every time it's done).

Apart from those bits, the only fiddly pieces were the code that senses touch drags and applies them to the image. I originally put a lot of effort into making only the endpoints of a tracing "hot," so you had to trace contiguously. Observation of my niece playing the game showed that it was not only hard to get right, it was irritating to use and placed unneeded restrictions on the player---after all, she's not angling for a high score, she just wants to turn the image pink, and if she wants to do that by squeegeeing with her finger like she's cleaning a window, maybe I should get out of her way! YAGNI.

(There's a future treatise on "Games vs. Toys" here, but it's a story for another day. Suffice to say, I wish I'd remembered the lesson I learned watching her play when I was naming the project, because I should have called it "Frances's Tracing Toy").

Things I Liked

• The documentation on Android is very decent. It walked me through most of the fiddly bits and platform-specific pieces. I walked through the "hello world" cat example and went from there. Anything I couldn't immediately glean from the documentation, I was able to find in the blogging community and stackoverflow.com.
• The test-and-iterate cycle is pretty dang short with my phone plugged in. It's smart about the details, like quitting the app if there's already an instance running.
• The resource abstraction is a pretty good abstraction, and the way resources bind into the code layer is extremely convenient. I ran into a couple of tiny issues where I had behavior in the wrong place in the Traceview code that crashed because it was being run at resource-loading time and tried to touch nonexistant resources, but that was an easy issue to diagnose and solve.
• Publishing flow to the Play Store was pretty much a breeze. I paid my dues and churned out art assets using svg-edit, Paint.NET, and Gimp. The "promotional" shots I screen-grabbed from the phone itself (Thanks, screenshot hard-button!).
• One program, compatible with 1,318 device configurations according to the Play Store. Awesome!

Issues

• The aforementioned device issues on Linux. But that can probably be filed under "Every experience I've ever had plugging a new hardware device into a desktop Linux machine."
• I tried to get cute by centering my trace object using layouts instead of rolling my own offsetting code for the event and rendering handlers. It ought to work, but it got fidgety; I had to force both a re-render and a re-layout of the containing view as well as the child view (it appears that resizing a child doesn't necessarily trigger a repaint for the child's old position rectangle in the parent, though to be honest I didn't track down the exact details). There's still a squeaky little bug in the interactions between parent and child layers, and I'm sadly increasingly convinced that it hasn't been worth it to try and shortcut writing my own centering code by using the layout engine.
• I got very weird behavior when I tried to specify the <uses-sdk android:minSdkVersion="15"/> rule in my manifest (as that was the only system I'd tested on): the toy broke utterly (played sound but drew only a white background and nothing more). This was very unexpected behavior, and I haven't yet tracked down a root cause. As a result, I set my minSdkVersion to "1", and I'm hoping I don't get too many bugreports from people who's Android phone was made by cavemen is a bit older.
• My program is ostensibly compatible with 1,318 device configurations. That's... a lot more than I have access to for testing purposes. I've tested, in fact, on a grand total of one phone. It's a bit freaky to think about all those people who are going to be using this toy on a hardware config for essentially the first time... On the other hand, that's half the fun of PC development, right?

Future Ideas

There's plenty of places I can take this in the future, if I feel like it (and since the code's public, so can you! Just buy me a coke some time if you end up making a mint off of your version ;) )

• More shapes! My niece's parents have already asked for numbers and letters. I could have fun with that. :)
• Fading from a sketch outline to a full-color version when you complete a picture.
• Suppression of the menu bar and soft controls, with a "parent access" secret to get you back to the rest of the phone (personally, I wouldn't recommend letting a toddler have a phone all to herself anyway, but if this offers peace-of-mind to some parents, I could probably make it a thing).
• Usability tweak: Currently, the event listener does the simplest thing and takes the first-index touch point when detecting a touch. But one thing I've noticed about little users is that they aren't great at balancing the phone yet and tend to grip from the side, meaning there's often a little palm touching the screen providing a false reading. I bet I can work around that if I index through all the touch points. :)

Go Have Fun!

I hope you enjoy the game! If you have any comments, feel free to post them on the game's Play Store page itself. If you have comments for me though, feel free to post them here, or come find me on Plus. I'd love to hear from you!