Handling Chrome’s Extra Security for Using the Microphone in Flash

One of my first projects at Mindflash was to create a feature that gave trainers the ability to record narration over their slides within the Mindflash app. The impetus for this was that content like Word documents and PDFs don’t give you the ability to record audio as you’re creating them. Thus if you wanted audio over this type of content there needed to be a solution built into Mindflash. Additionally even though PowerPoint includes built-in audio recording, many of our users struggled to figure out how to effectively use it. We provided help articles to educate them on how to use PowerPoint’s built-in audio recording feature but many users still struggled. Thus, we sought to make something simple for our users to quickly narrate over their slides. Since adding this feature we’ve found that 25% of our customers are using it.

We created this functionality in Flash both because our trainer app at the time was written entirely in Flash and also because there wasn’t (and still isn’t) any way other way for us to add this functionality for all of our supported browsers (back then we supported all the way back to IE8, today we support back to IE9). We spent a lot of time researching and writing up hacks to detect if the user denied the security dialog Flash pops up to ask for permission to use the computer’s microphone and perfecting the workflow so that if she did, the app would respond properly to let her know why she couldn’t use the recorder. Once she allowed permission through the Flash pop-up everything would become enabled because at this point we have permission from Flash to use the microphone and thus record.

This all worked fantastically on all browsers for a while until Chrome added additional security that forced the user to click “Allow” a second time on a notification they bring up below the address bar. See the screenshot for what it looks like below:


There is much to gripe about this notification.

  • It’s in a location potentially far from where the user clicked to trigger it
  • It blends in with Chrome’s address bar so users frequently don’t notice it
  • it doesn’t show up until after you’ve tried to start using the mic.

In my opinion the worst problem is that it doesn’t appear until after you’ve tried to start using the mic. With Flash you can ask for permission before you actually turn the mic on so we’ve set up Mindflash to immediately ask for permission to use the mic as soon as the user opens the audio recorder. This forces the user to determine if they want to allow Flash to have access to their microphone before they’re able to take action to start recording. Thus, in any other browser if they clicked the button to record audio we already have their permission and we’re able to immediately start recording. However, in Chrome as soon as they click the record button the notification center security setting will come up and nothing will actually start recording until they click “Allow.” This is a much more confusing workflow.

Before we figured out how to better handle this, when this notification center security setting came up the app would continue to assume it had access to the microphone and act as though it was recording. This was with good reason as according to Flash the microphone is actually on (this can be checked by verifying the Microphone object’s “muted” property is false) so there was no reason to think we’re not recording. However, even though Flash thinks the mic is on it’s not actually sending any audio data. This is how we were better able handle Chrome’s extra security.

Below is a partial code snippet that shows how we can detect if we’re actually getting audio data once we start trying to record.

The idea here is to constantly check the size of the raw audio data. If two seconds after the user hit the record button we still don’t have even a negligible amount of audio data then we know we’re not actually recording and we’ll show the user something to let them know how they can fix this problem. Once they click the Chrome-specific “Allow” button the recorder works as the user would expect, immediately starting to record their audio. See the screenshot below for how we’ve handled this user experience or give it try for yourself by signing up for a Mindflash trial.


While I haven’t tried it, I’d assume this would work quite similarly for video input. If anyone tries it with video input I’d love to hear how it works so please leave a comment.

How Mindflash’s FocusAssist Works

In a live training environment, generally a real life trainer works with one or more trainees to educate them on the training material. One benefit of this is that the trainer has a good sense of a trainee’s level of engagement. For instance, if a trainee were to fall asleep or be playing with their phone and not paying attention it would be quite obvious in a live training session. Conversely, trainees can show their positive engagement by asking or answering questions in a live training environment. In online training getting this feedback is much trickier.

FocusAssist is a feature we’ve developed at Mindflash to better measure and ensure trainee engagement. It’s a way to help trainers get some of that engagement information while still being able to deliver their course material online, asynchronously. It uses the camera in a trainee’s iPad or computer to determine if the trainee is present and their level of engagement. While FocusAssist does use the trainee’s camera, no video or photographs are saved or streamed anywhere. It simply uses each frame from the camera to determine the user’s level of engagement and throws the image away.

In this post I’ll talk about how we implemented FocusAssist and where we can go from here. For the implementation of FocusAssist we use a third-party framework for the computer vision piece of the feature which finds and tracks the user’s face and eyes. The work we’ve done to implement FocusAssist ties into this framework to have Mindflash’s trainee application respond to the computer vision data intelligently.

A Note on Privacy and Security

We take privacy and security seriously, especially with this feature given its use of the trainee’s camera. The camera input used for FocusAssist is not stored or streamed anywhere. It’s only used for processing on the trainee’s client to determine information about the trainee’s presence and level of engagement. Even this information on engagement level is not identifiable. We never give an individual trainee’s engagement score, but instead only show aggregates once we have engagement scores for five or more trainees.

FocusAssist currently operates in two modes:

  • Measure and require engagement
  • Measure engagement

Requiring Engagement

Requiring engagement is the proverbial stick FocusAssist enables, but it definitely has it’s uses. There are many industries such as healthcare or government in which the organization training an individual absolutely needs to know that the trainee has seen all of the training materials. Previously this meant that training in these areas would need to be done in a live class so that the trainer could mark attendance and ensure the trainees were paying proper attention. With FocusAssist’s require engagement functionality this is no longer the case.

When a Mindflash course has been setup with FocusAssist to require engagement, the course requires that the trainee has an iPad or computer with a camera. The camera must be focused on the trainee at all times. If the camera cannot find the trainee then a modal pop-up will appear over the training material alerting the trainee that the camera cannot see his face. If it finds the trainee but determines the trainee does not meet or exceed the threshold at which it believes the trainee is engaged then it will pause any media in the training material until it can find the trainee and determine he is engaged.

Once FocusAssist can find the trainee and determine he is engaged again the media will automatically resume. When this setting is on, FocusAssist ensures that the trainee sees the training material in full and trainees are not able to skip ahead in the course content. While this is certainly heavy handed it provides a great substitute for a live training environment, allowing trainees to take the course at their convenience.

Measuring Engagement

As a trainee is taking a FocusAssist enabled course, every quarter-second it is checking the trainee’s engagement level. When it does this check it is looking for two primary pieces of information:

  • Do we see a face?
  • Does that face meet or exceed the threshold for what we consider to be engaged?

If we can’t find a face we ignore the data for the purpose of measurement. The reason for this is that it could just as easily be meaningless as it could be meaningful. If we can’t find a face, it could mean the trainee walked away out of boredom but it could also mean that the trainee left the room to use the bathroom or answer the door.

If we can find a face then we utilize the data and factor it into the engagement score. We calculate a separate engagement score for each trainee on each slide in a course. We then aggregate the scores for each slide for all trainees. In our trainer reports, if a slide has an engagement score for 5 or more trainees then the report will show the aggregate to the trainer. If it has fewer than 5, the report informs the trainer that not enough data has been collected. This ensures that trainees cannot be identified by their engagement score.

Block diagram showing how application responds to input from computer vision framework.
Block diagram showing how application responds to input from computer vision framework.

Looking Ahead

At present our algorithms for calculating and aggregating engagement scores are simple. In the future we hope to refine them to make the scores even more meaningful. For example, for a slide in which we know the duration of the content (i.e. a 3 minute video) we could drop the trainee’s score if we know that the trainee only watched 30 seconds of the slide. For all slide types we could do a better job of throwing out outliers. For example, if 10 users were on a slide for 30+ seconds, perhaps we could throw out the score of a user who stayed on the slide for 1 second. The goal of these improvements would be make the score more valuable to the trainer.

Additionally, in the future we hope to be able to get enough data out of the trainee’s facial expression to understand some basic emotions when they’re viewing a slide. Again we’d want to do this in aggregate so as to not identify trainees with this data but imagine if a trainer knew 6 out of 9 trainees showed confusion on a quiz question or that 4 out of 10 trainees appeared bored on a slide.

The end goal of all of this feedback is to help the trainer improve their course material. This then helps future trainees learn and enjoy the training material better. At Mindflash we love thinking of new ways to improve the online training experience. If you have any thoughts on how to improve upon these ideas or others leave a comment or contact us. If you’d like to try out FocusAssist yourself sign up for a free trial of Mindflash.

Dealing with HTML5 Video on Mobile Devices

Web video has never been a cakewalk, but at least in the days when Flash was your only option you could pretty well guarantee it’d work everywhere. This is no longer the case. We’ve gotten to the point where you can easily use H.264 as your only option (with HLS if you’re including your videos in an iOS app) but there are still loads of quirks in playback. Each browser’s implementation of HTML5 video is slightly different – mobile devices throw in additional variance. And of course IE8 and some versions of Firefox (when using H.264 at least) still require Flash for playback.

Fortunately, we’re able to use amazing open-source libraries such as Video.js to smooth out most of this for us, in particular the change in tech from HTML5 to Flash where necessary (looking at you IE8). However even with Video.js I’ve found myself having to work around a lot of variation on mobile devices. Below are some quick notes on what I’ve seen and how I have or haven’t been able to work around it.

iOS Quirks

  • iOS loves to show this incredibly attractive QuickTime logo to let you know it’s a video. It seems to do this when the video is not in a loaded state; thus, you see it on initial load. If you want to hide this bit of ugliness from the user you can push the element offscreen before it starts loading by setting in CSS its position to absolute and its left or top to something way off screen (i.e. -10000px). Once the metadata is loaded you can bring it back and on screen with no QuickTime logo. You need to push it off screen because anything to truly hide it (i.e. display: none or visibility: hidden) will cause the video to not load.
  • Another way to get the QuickTime logo is to let it run to the end of the video. To avoid seeing it after the video ends I’ve set up a listener for the timeupdate event and made the video pause when it approaches the duration. This way the user sees one of the final frames instead of the QuickTime logo when the video “ends”.
  • No autoplay unless embedded in a native app’s custom WebView.
  • On iPhone there’s no way to not go full screen unless embedded in a native app’s custom WebView.

Android 4 Quirks

  • No autoplay unless embedded in native app’s custom WebView.
  • On the default Android 4 browser (not Chrome), you need to tap on the actual video to get it to start. You can overlay another element (i.e. a giant play button) on it and have the click event for it start playback but what you see is it tries to start it and then immediately pauses it. You are able to put a poster on a video but depending on your application’s design this can be kind of confusing as it can just look like an image. There’s no indication to tap on it to start the video. Sadly, we’ve found the best solution to this to be no poster. It lets the Android equivalent of the iOS QuickTime logo show through. Not pretty but at least the user knows to tap on the video to get it started.
  • On the default Android 4 browser (not Chrome) once you’ve started a video in a tab if you leave that tab it can no longer be used again. You must refresh the page. This means if you leave the tab to another tab or leave the browser for the home screen or another app your app basically breaks. I think this is unfixable from a web developer’s perspective as even YouTube works this way. Pretty awful though.

Browser-Level Volume Control Feature Detection
iOS doesn’t allow for you to mute using the HTML5 video api’s mute function. You can call it but it won’t work. Fortunately it doesn’t work in a way that allows us to sniff it out without having to rely on user agents.

The following gist shows how you can check if volume control will work. It’s setup with Video.js in mind but it’ll work with the HTML5 video api as well.

The final key to volume control is that on Android 4 setting the volume prior to playing the video doesn’t work. If you want your volume to persist across videos on your website you’ll need to set it after the video starts playing as otherwise it’ll disregard what you had set the volume to. Thus, wait until you receive the initial ‘playing’ event from the video and then immediately set the volume.

In my experience on both desktop browsers as well as mobile browsers you never hear the volume change (even if what you’re doing it muting the volume) because the change is done so quickly after the browser starts playing the video.

Changing the iOS WebView User Agent to Handle iOS7 Design Changes

iOS7 is here and its design changes are significant. From my perspective though the biggest change for hybrid apps is that the status bar’s space is now allocated to your app. I spent some time thinking about how to deal with this and after a couple iterations I think I’ve found a good solution.

20 More Pixels

One of the nice design changes in iOS7 is that apps can now be truly full screen. They utilize the 20 pixels of status bar at the top of the screen as the app’s content shows through the status bar. Previously the status bar always had a black background. At Mindflash, we’ve found this particularly nice when a user is immersed in a course as the course content now takes 100% of the screen.

This can have some big implications though. At Mindflash we originally designed our app to fit the 1024×748 resolution of the iPad’s screen without the status bar. This isn’t the best thing to do but since every single device on all versions of iOS had this resolution it simplified things in what was a large project. We’ve since fixed this. When we compiled our app in Xcode 5 we found our app to be a tad short. The first obvious solution to fix this was to make the app’s frame 20 pixels smaller when on iOS7. More details on this can be found on Stack Overflow.

This works reasonably well if you have only a single view controller. If you have more than that it gets more complicated. I never got a fully working solution as I deemed it ineffective and looked for something else. The next obvious solution was to handle it by loading some new CSS specific to if you’re running iOS7. It’s easy to do this if all you need to know is whether the user’s running iOS7. You can have your javascript check the browser’s user agent. Unfortunately, we also need to know what version of our native app you’re running. This made our obvious solution more complicated.

iOS7 lets you run apps not compiled in Xcode 5. When it does this, it runs in legacy mode and shows the black status bar making your app’s frame 1024×748 instead of 1024×768. This is great for users because if an app doesn’t have a new release for the new OS ready or the user just doesn’t upgrade their apps, everything should continue to work. Additionally, Mindflash’s iPad app automatically updates the web portion of the app every time a user opens the app and a new version is available.

Thus, it’s likely that we have many users out there on iOS7 running an old version of our app not compiled in Xcode 5 but running our latest web code. In order to determine whether to display the web app in 1024×768 or 1024×748 we need to know both whether they’re on iOS7 and whether they’re on a new enough version of the Mindflash native app.

Appending to the User Agent

It’s easy to check what version of the native app the user is on but how do we communicate that to the web app before it even bootstraps? The answer for us was appending information about the native app version to the existing user agent. I found we could override the user agent in NSUserDefaults easily but we wanted to maintain the existing user agent string and just append our information to this. We found we could do this by creating a temporary webview to pull the user agent before then overriding it with one that has our custom information appended to it.

Now that we have the native app’s version number in the user agent, we can check it and add a class to apply CSS that only affects the app if we’re running iOS7 and on a version of the native app that supports 1024×768 resolution.

The User Agent Hack Is Cool But Why is This Even Necessary?

While having the app show through the status bar looks awesome there are a lot of apps, Mindflash’s included, where some functionality is at the top of the screen. You can’t tap on a button that’s underneath the status bar. Thus, we need to know if the app is showing through the status bar. If it is, we need to make the area where the buttons are 20 pixels taller so there’s nothing tap-able underneath the status bar.

One could argue you could do this by just measuring the height of the browser window. If it’s 768 pixels then we know the app is showing through the status bar. This doesn’t work though if you’re using the same web app for the iPad and the web. If the user’s on the web and has 768 pixels of height you clearly don’t want to do this.

You could add a check of the user agent to see if they’re on iOS in addition to having 768 pixels of available height but that complicates things if you want to allow users to use your web app in Safari without having to get your native app from the App Store. In the end it seemed easier and more reliable to use the version number of both iOS and the native app to determine whether or not the app was showing through the status bar.

Besides, having this information in your user agent makes it easier to check for other things dependent on the native app version. For instance if you add a feature that requires changes to both the native and web apps you can verify whether to activate them on the web app by ensuring the native version is recent enough. In an app like Mindflash’s where the overwhelming majority of changes are done to the web app this has huge advantages because we can ensure that almost all of our new features go out to all users regardless of their native app version.

How We Survived the Recent AWS Outage

When AWS has an outage people really notice. Obviously the tech community explodes. Every company starts tweeting out their downtime status while swarms of developers / ops folks at all of these companies gang tackle the problem to try to get their site back up. Increasingly though, even the non-tech folks notice and are frustrated. On the great Christmas Eve outage of 2012 that took down consumer behemoths such as Netflix, social media was awash in frustrated users, but even the relatively minor outage on Friday caused enough waves for my completely non-technical wife to notice as some of her favorite websites were down.

While AWS is an incredible service it has it’s issues from time to time so it’s best to be prepared. Over the past year the company for which I work, Mindflash, has made significant strides to try to prepare for these sorts of events and during the Friday the 13th outage this paid off.

On Friday Amazon had a partial outage with one of its availability zones (read: one of its data centers) in Virginia. The issue was that while these machines were able to connect to each other just fine, the outside world was unable to reach them.

At Mindflash we run all of our services (except for much of our S3 content – perhaps the subject of another blog post) in Virginia and actually had a substantial number of our servers running in the availability zone that had the problem, but we managed to get away relatively unscathed. How? A combination of efforts over the past year to run with redundancy in multiple availability zones mixed in with some good old-fashioned luck.

To give you a bit of background, Mindflash runs about 5 different types of servers:

  • Public web servers – What explains our product and pricing, allows you to sign up, login, etc.
  • Product web servers – What you see once you log in to the product.
  • API servers – Handle requests from the product to get and store data
  • Conversion servers – Handle the process of taking our trainers’ content and making it into something better for being consumed on the web / mobile devices
  • Background services – Handle scheduled or long-running tasks (aside from conversion) that don’t need to hold up a response to the client.

With a couple caveats we run at least 2 of each of these types of servers and have either load balanced them on ELB so that our traffic is split between each of the same type of machine or set them up to run on a queuing system so if one of the machines goes down it just stops picking up jobs. We do this both to maintain great performance when we have heavy traffic as well as to ensure the site’s uptime.

When we set up our machines for each of these types of servers we ensure that we have them spread out across the availability zones. That way if machines go offline in one of the availability zones we still have at least 1 in another availability zone that’s working. For the servers that run off of a queue system, they’ll stop taking jobs when they go offline because they won’t be able to reach the queue. For those being load balanced, Amazon’s load balancing service, ELB, is able to tell when a server is offline and automatically stop serving traffic to it. Since only one availability zone went out on Friday in theory we should have been 100% good to go.

Well, we weren’t quite. Remember that the outage only affected traffic from outside of Amazon’s network. Thus, the load balancers didn’t think the servers in that availability zone were offline and continued to serve traffic to them. This makes for is a shaky user experience. If you were lucky you’d get one of the servers that was still responding. If you weren’t you’d see errors or the page just wouldn’t load. When we realized this it was easy to fix as we just manually removed the machines in the bad availability zone from all of our load balancers.

There was a huge positive to this though. We’re not 100% there with our redundancy strategy yet so we still have a few servers of which we aren’t running two or more. Even though some of them were in the availability zone with the outage none of them are hit directly by machines outside of Amazon’s network. Thus, everything continued to work since the connection between Amazon’s servers was still intact and working just fine. We definitely caught a break here.

What we’ve learned from this is that it seems our strategy to ensure we have redundancy spread between multiple availability zones paid off where we’ve implemented it. That said, in the areas we haven’t done this though we’re still at risk and during this outage we just got lucky. There’s always more work to do.

In general Mindflash has been very lucky with AWS outages. Even when Netflix was down last Christmas Eve we were still up. While we were still lucky with this one a lot of our success this time was the result of the hard work we’ve put in to prepare for these events.

This is our story. I would love to read any comments on your strategies for surviving these outages and how they’ve worked out for you in the past.

Startup Social Contract?

I whine on twitter a lot about how vapid much of what’s posted as “tech news” is. Normally I’m bitching about “articles” that are just glorified press releases. There’s always quite a bit of meta junk on “startup life” and being an entrepreneur or just life in the valley as well though and recently the amount of it seems to be on the uptick.

This particular editorial on PandoDaily drove me more nuts than usual though. Below I’ll take down the pieces that bothered me the most.

Only startups “change the world”

Employees need to pick which one they want: A messy mission that could fail or change the world or a predictable, polite place to go from nine to five.

I have limited work experience having only worked at 3 companies over the span of my 5ish years of work and internships, but it has been varied. I’ve worked at a multi-national food company, a large (6000+ employee) software company, and now a start-up (25 people). All of them think they’re changing the world. They all have lofty ways of describing themselves and they’re actually rooted at least somewhat in reality.

More importantly the employees who enjoy their job (which is the majority of them at any decent company, small or large) believe they’re changing the world too. They buy into the large company’s mission at a rate comparable to those working at startups. They don’t believe it’s a “polite place to go from nine to five.” They enjoy their work and are often working long hours. In my own experience I can tell you I worked way more long nights at the large software company than my current startup. I’d be shocked if I was alone in this.

OMG Start-Up Employees Haz Needs!?!

I was recently talking to someone at a young company looking to hire a candidate who wanted competitive compensation from his last job and also more equity than the company wanted to give. I made the point that that should really be an either/or, and if the cash-constrained startup were being asked to pay a huge salary, more equity should be off the table. In other words, allow the candidate to pick one: Money now or upside later. This person actually said, “I don’t think the amount of equity offered should have anything to do with the level of compensation.”

Why is a journalist essentially advocating for the investors / exec staff of companies? If the company doesn’t want to pay up then they have the option to not hire this individual. No one’s forcing them to hire this person. However, if they want this person to work at their company then they should meet his/her demands.

It makes sense for the potential employee to ask for what they think they deserve. The company doesn’t need to agree with it but they should be reminded and are probably quite well aware that there are plenty of larger, more established companies and “large start-ups” that are quite willing to meet or come closer to meeting the potential employee’s expectations and they’re always looking to hire.

If this start-up’s founder or exec team wants to build a great company and great products they may need to shell out the dough for a team capable of doing so. Those employees they’re hiring have their own needs, their own priorities, and their own families. This shouldn’t be surprising, and from what I can tell it’s not given the facts on the ground by which the writer of this editorial is so outraged.

Being professional shouldn’t just be a big company thing

Companies grow quickly and CEOs grow with them. In the best of cases, things break and tempers flare. Unlike being the custodian of a large pre-existing brand, this is very personal.

Again, people at large companies take things personally in their jobs as well so this is just ridiculous. They don’t view it as a custodian-ship. They care deeply and sometimes they act unprofessionally. The difference seems to be that at large companies they’re reprimanded. At start-ups it’s apparently ok. Look, no one’s perfect – I’m not suggesting people be fired for having an outburst but to act like it’s acceptable and not something that needs to be fixed is stupid.


I love living in Silicon Valley and working on software. I love working at a start-up. There are plenty of reasons for this that maybe I’ll get into in another post but the content of this editorial in particular but also in plenty of others has increasingly bothered me.

The constant denigrating of large companies and particularly the good, honest, hard-working people at them is ill informed and insulting. Arguing clearly for the interests of the capital as opposed to the labor is odd although not unexpected for “tech journalism” and it’s myriad investment interests. Lastly, the expectations of teenage angst and accompanying outbursts in start-up employees is something that should be called out as opposed to embraced as long as you’re “not just being a dick.”

I can’t possibly be alone in thinking these things. Am I? Let me know with a comment.

Guess Who’s Going to Warped Tour?

This guy!… plus hundreds of thousands of angsty teenagers. Pretty sweet, no? In all seriousness this is the first time I’ll be able to make it to warped tour in 6 years and I’m absolutely stoked about it. There’s nothing I like better than seeing punk music live and Warped Tour is an opportunity to see a lot of it in one day, which means a lot when you don’t get out that often. Sure, all of the sets are ridiculously short (the headliner of the day played 45 minutes last time I was there, which is completely ridiculous), but the 30 minutes or so I’ll see of each band are sure to be magic.

I’ve been to a few other more traditional festivals – Lollapalooza, Pitchfork Music Festival, and Outside Lands – but there’s one thing I’ve found special about Warped. It’s pretty much exclusively punk music which you pretty much can’t see anywhere else (although I guess I did see Manchester Orchestra at Lolla). I’m a huge punk fan, particularly live, because of the raw energy in the performance and the crowd. I love all types of music but in college I started to get increasingly bored at more “shoe-gazy” shows – even for bands I truly enjoy listening to such as the Shins, Sigur Ros, and Wilco. Particularly if the even is seated it just loses all of its allure for me. At that time I got increasingly more into electronic, dance-driven, downright poppy music and in particular punk and that’s only continued to be the case through the years.

The best news about being able to attend this year’s Warped Tour is the line-up is great. As a huge Early November fan who’s never seen them live I’m really pumped to see both them and Ace Enders doing his thing as I Can Make a Mess Like Nobody’s Business. I haven’t see Motion City Soundtrack since the first Warped Tour I went to in high school so seeing them again should be a trip. Last time I tried to see Chiodos with Craig Owens in the band my plans were thwarted so seeing them reunited is going to be great. Woe, Is Me should be a good time even with all of their drama among the band members and I See Stars I think is going to be a ton of fun live. Definitely looking forward to Like Moths to Flames, Hands Like Houses, VersaEmerge, We Came As Romans and Sleeping with Sirens. Should be fun seeing my old high school-mates blessthefall again.

NeverShoutNever should be a trip. Some of his music is truly great, some of it kind of sucks and lot of it is so sappy and poppy you can’t help but love it. Will be interesting to see the mix he chooses to play and what his stage presence is like. In videos he seems like a giant tool but I do love a lot of his music. I’m really looking forward to Memphis May Fire and Bring Me the Horizon. I’m hoping their performances will be intense and aggressive. I’m sure there are more who unannounced who will be great too.

Overall I have to say this is the most excited I’ve been for a show in years. Can’t wait for July 22nd where I’ll be one of 17 people at the Mountain View Warped Tour old enough to purchase beer. Fuck yeah!

Keep Your iOS App Running in Background Forever

I’ve known many people in my life who have a nasty habit of leaving stuff they’re not using plugged into the wall, just wasting energy. My wife would be a prime example. She’s just not in the habit of unplugging it and changing your habits can be hard. Thus, I set out to create an iOS app to remind her to unplug the charger from the wall every time she unplugs the device from a power source.

One without knowledge of iOS development would imagine this to be a simple task. The logic is certainly simple. Notice the battery state go from “charging” or “full” to “unplugged?” Great, send a simple local notification. Otherwise, just keep monitoring the battery state. So what’s the issue? Apple doesn’t want to keep apps running in background.

The logic behind this is pretty reasonable. Apps running in the background drain battery life faster. Apps draining your battery life gives you a bad perception of the device, not the app, because the average consumer has no idea that it’s an app, yet alone which specific app, that’s making their battery life crappy. Thus, iOS won’t let your app run in the background but it does give you some options, such as push notifications, to provide some data to the user and prompt them to open the app again to get the full details.

This is reasonable, but the problem is that push notifications are driven by data coming from the internet. What if you just want to monitor something on the device (i.e. battery state)? Well, as far as I can tell you’re out of luck without using a hack.

The notable hacks are using location services, playing an inaudible sound, and VOIP. Location services really can drain your battery as GPS is power hungry. Playing an inaudible sound is a good option but your app’s going to die if you actually use your device to play audio/video (and most of us do). So the only option in my mind is to use the VOIP hack.

The logic behind allowing VOIP apps to run in the background forever is that they need to be able to receive calls at all times. Thus, generally iOS tries not to close these apps and as a failsafe if it does (due to running out of memory or a crash or just restarting your device) it’ll automatically relaunch it in the background without making the user do anything.

The VOIP hack takes advantage of this to work with any application that wants to run in the background forever. There’s scattered information online, particularly on stack overflow on how to do this but it’s poorly documented and I couldn’t even get it to work for my application without some changes. Thus, I wanted to put it out here as clearly as possible to help anyone else working on an application like mine that needs to monitor the device in the background.

The configuration:
The first thing you need to do is modify your app’s plist. Add the following settings.

  • Application does not run in background: NO
  • Required background modes: VOIP

This lets iOS know that your app will be running in the background and thus will ensure that if it’s terminated, iOS restarts it automatically for you.

The code:
First, we need to set up the code for when the application launches. In here we set up our background task (bgTask) and the block that’s called when our background task expires after 10 minutes (expirationHandler). The expiration handler block ends our background task and restarts it. Additionally, you’ll notice that we start the background task immediately (even if we’re about to go into active mode) because when the app is terminated and relaunched by iOS it doesn’t tell the app that it’s going into background mode.

Next, we set up the background job. The key piece here is the dispatch_async block. This is the work we’ll be doing in the background. In my case, the work is just a while loop that checks the battery and then sleeps for 1 second. For 10 minutes this while loop executes and then we see the expiration handler called.

What’s interesting is that even though the expiration handler is ending our task, the dispatch_async block keeps running. Thus, we set a flag in the expiration handler saying the job has expired and use that to trigger the while loop’s end. Finally, we have the expiration handler spin while it waits for the old job to do this and exit the dispatch_async block. If you don’t do this you’ll have multiple dispatch_async blocks running at once and your app will be terminated more quickly.

Lastly, remember that we started the background task as soon as the app launched. If you don’t want this to run when the app is active then do something in applicationDidBecomeActive to stop the task from running. In my case this just meant setting another flag to get the while loop to exit.

That’s it! From my limited testing this seems to work well and not have a huge effect on battery life. Let me know if it works for your app.

The one huge drawback of this approach (as well as any I’m aware of to create this sort of application) is that Apple would reject it from the app store. Thus, if you’re looking to create an app for yourself, your wife, your friends, etc. then this’ll get the job done, but if you want to get it out into the world you’re out of luck.

Thus, I’m doing the best I can by putting my app on GitHub. I’d love any feedback or contributions.

The Past Year’s Gadgets Were Boring… Except for the Wii U

This has been harped on by plenty of people particularly around Christmas when so many get into their “I must buy all of the things” mood and look for something, anything, to buy their best friend’s neighbor’s cousin but I wanted to add my voice to the chorus. When it comes to innovation in consumer gadgets this year didn’t have much to offer, at least as far as I could tell. Everyone released their latest version of the phone/tablet/phablet, we were given even more devices that connect to Netflix/Youtube/Pandora, and cars were given one-button interfaces to be more iPad-like (goody?).

The one exception in my mind was the Wii U. I try to get myself one thing a year just for fun and this year it was unquestionably going to be the Wii U. Why? Well, because nothing else looked interesting (oh sweet, my iPhone screen can be 1/2 inch taller and 1/4 inch thinner), and because I’ve always been a sucker for Nintendo games – especially Mario.

So I got the Wii U on day 1 and immediately loved it. The asymmetric gameplay in Nintendo Land is really interesting. It’s fun playing games where one player does something on the tablet screen while four others do something else using the standard old Wiimotes. For instance in Mario Chase one player using the tablet runs away from up to four others using the Wiimote. The player using the tablet uses a map to see where the other players are and try to avoid them while those using the Wiimotes have to look for the player to chase without a map.

It’s also great being able to play on the tablet screen. You can easily switch between the two depending on if someone else wants to use the TV. I especially took advantage of this with New Super Mario Brothers U.

Another interesting feature of the Wii U is the Miiverse. Miiverse is a social network specifically for Wii U. You can post short messages in communities for each game about your progress, what you liked, what you didn’t, ask for help, whatever you want. Plus you can post screen captures for the game and pictures you draw on the tablet. The pictures are black and white but the detail some people are able to put into them is just amazing. Scrolling through these posts can be a lot of fun.

While the games out now are fun they largely don’t take advantage of the technology as much as they could. This isn’t shocking as software always lags behind new hardware. I’m looking forward to the new games coming out this year and to the new innovation it brings.

3 Things I Wish I Knew Before Building a Hybrid iPad App

At the end of November, the company I work at, Mindflash, released an iPad app. We built it primarily in AngularJS with some native hooks where we needed them. I’m sure I’ll expand more on this in the future but here are 3 quick things I (and I’m sure all of my colleagues) wish I knew prior to starting on the project. Perhaps they’ll help your in your own future projects.


    iOS Safari chokes on iframes. We found iframes to be a constant source of pain causing seemingly every problem from using an inordinate amount of memory to making a vertically scrolling div rendered on the page completely outside of the iframe only scrollable by swiping horizontally (WAT!?!).

    Given the huge number of APIs that use iframes, you can find yourself running into this problem a lot. Even worse for us was that the way in which we were originally communicating from the WebView to the native app was by changing the URL of an iframe in our app. In the end we removed all dependencies on iframes aside from embedding YouTube videos and have never been happier.

  2. Serving your local app from file:// uses a lot more memory than from a web server
    I don’t understand why this is, but the difference is substantial. We saw a drop of around 20-25 MBs of memory usage which is huge on iOS and particularly on the iPad 1.

    So how does one serve their app from the iPad’s file system without using file://? By using an iPad-based HTTP server of course! We used the open source CocoaHTTPServer.

  3. The iPad 1 Is the new IE6
    It seems no matter what you do to improve your app’s reliability and performance it will crash on the iPad 1 much more often than any of the later generations. This is due in part to the tighter memory constraints but also due to the fact that many people’s iPad 1’s were hosed on upgrade to iOS 5. Check out this thread for one of many discussions on the issue.

    There are lots of reports of people primarily on the iPad 1 but also on the iPad 2 seeing Safari (and thus WebView) crashes with breathtaking frequency after upgrading to iOS 5. Since the only guaranteed solution is resetting the iPad with iOS 5, you’ll run into plenty of users who haven’t done that and have just lived with it and there’s nothing you can do about it.

I hope this helps anyone building a hybrid app today. We’ve been thrilled with our decision to architect it this way. Again, I’ll follow up with future posts on the development of our iPad app. Perhaps the next one will be on things I wish we knew how to do – like how the hell do YouTube and Netflix serve their streaming video so quickly?

Insights on programming, music, young parenthood, etc.