It’s Official—iOS 8 Is Apple’s Buggiest Release to Date
BY CHRISTINA BONNINGTON 11.18.14 | 6:30 AM | PERMALINK
wired.com
  Ariel Zambelich / WIRED
When Apple first released the iPhone 6, we were struck by the surprisingly persistent and numerous bugs in iOS 8. Almost all review units (from any company, not just Apple) are thoroughly tested, vetted, and hand-selected as being the best representation of that product. You don’t want a reviewer accidentally ending up with a blemished, defective phone. Bad publicity. So using an iPhone that rebooted itself and got hung up on the keyboard was surprising indeed.
We weren’t alone in that sentiment. WIRED saw similar bugs on the iPhone 6 Plus. Other reviewers pronounced it Apple’s buggiest release yet, and Apple pundit John Gruber wrote “it seems like Apple’s software teams can’t keep up with the pace of the hardware teams” before talking more about getting stuck in an endless reboot cycle.
Turns out it wasn’t just in our heads: Data from app performance monitor Crittercism showed iOS 8’s crash rate was 60 percent higher than iOS 7 during their respective first months on handsets.
“I find myself at once impressed by the overall quality of iOS, and surprised by the seemingly obvious problems that have recently made it past QA,” Red Sweater Software founder and ex-Apple software quality engineer Daniel Jalkut told WIRED via email about iOS 8. Jalkut shed some light on how bugs can slip through:
“Some bugs, like the HealthKit problems that botched iOS 8.0’s debut, and the problems with 8.0.1, were clearly issues that Apple had not identified, or they would have found a way to fix them before releasing. Other, less dramatic issues could really come down to a manager at Apple being hard-pressed to meet a deadline and “punting” (that’s a common term used in reviewing bugs at Apple) the problems down to the next release, or even later.”
This is something that obviously happens with every major software release. But with iOS 8 more slipped by than in any other iOS version before it.
The ReportsLooking at past bug reports seemed like a good way to better understand whether iOS 8 was actually “buggier.” Anecdotally, it’s difficult to quantify how many bugs each iOS version has. There are forum threads upon forum threads, posts upon posts outlining all the issues iOS users have experienced through the years. Some affect hundreds or thousands of users, others, less than a dozen (Take, for example, Bendgate: Despite wide publicity, the issue only affected a small number of iPhone 6 Plus owners).
“We expect perfection, and when we don’t get it, we tell the world,” Matt Johnston, chief strategy officer at app testing and analytics firm Applause, told WIRED. Everyone is holding a digital megaphone thanks to social media. “Even the best software companies stumble, and when they do, users will air their dissatisfaction far and wide.”
And knowing, from reports and forum discussions, that the bugs experienced by an iPhone owner can very dramatically from handset to handset, we couldn’t just go off our personal OS issues, either.
So we turned to Apple’s official Support pages on iOS updates for an objective perspective on iOS bug fixes. These outline the additions and bug fixes for each version (like this one for iOS 8.0.2 or this one for iOS 6.0.1). Counting items that were clearly bug fixes or improvements that resolved issues (rather than ones that added a new functionality or feature) as a metric, iOS 8, to start with at least, wasn’t all that different from past launches.
It had the same number of bugs as the launch of iOS 6: 8 issues fixed in the .0.1 update (which was quickly pulled in the case of iOS 8), and one more tacked on in the .0.2 update. Fewer bugs were addressed initially in iOS 7, but it had 8 to 9 bug fixes in its late October 2013 7.0.3 update. Over the course of their whole existence, iOS 6 had approximately 21 total bug fixes, while in iOS 7, that number climbed slightly to 27.
Chronologically, these updates are coming faster and faster. If you look at the timeline for Apple’s iOS updates, some general trends start to emerge: Apple is rolling out updates to iOS a lot quicker after the initial OS launch. For example, the first OS update to iOS 5 was about a month after launch; the first to iOS 6, a month and a half later; while iOS 7 had four updates from its September 18 debut through mid-November. More bugs are being addressed through these updates.
About a month and a half into iOS 8: We’ve had four updates (8.0.1, 8.0.2. 8.1, and 8.1.1), and we’re officially at 23 listed bug fixes through iOS 8.1. But with iOS 8.1.1, which just went live Monday, Apple doesn’t site specific bug fixes, but rather a general “This release includes bug fixes, increased stability and performance improvements for iPad 2 and iPhone 4s.” It’s not a leap to assume that “bug fixes,” plural, refers to at least two to three fixes, and that stability updates to older iOS devices could bring that number to at least five. That officially makes iOS 8, thus far, the buggiest iOS yet.
A Fuller PictureObviously, straight numbers don’t paint a complete picture of what’s going on.
At the same time that bug numbers are rising, iOS has grown into its most complex, advanced iteration yet, and Apple’s vastly expanded its hardware offerings, too. Instead of a singular iPhone on one carrier, Apple now sells four iPhone models and supports six; sells five iPad models and supports eight; and it sells the fifth-generation iPod touch, too. On top of that, the iPhone and iPad are available on hundreds of different carriers worldwide. That in itself is good reason why there could be more bugs Apple has to fix in iOS 8 than in the past.
But there are other factors too. Starting with the fourth generation iPad (the first with Retina display) and iPad mini, Apple switched the iPad launch date to late fall, following its annual iPhone launch. With this change, it makes sense that the past few years would show a quicker rollout of bug fixes towards the beginning of the iOS life cycle, as iOS may have to be updated to support new hardware models.
Apple could also simply be addressing more bugs than it was able to in the past, fixing smaller issues that affect fewer people, rather than only Antennagate-level woes. Apple’s Support pages don’t say how many people are affected by each bug fix.
However, with an increasingly complex OS and hardware, it doesn’t help the bug issue by launching both at the same time each year.
The Price of Fast-Paced Hardware/Software ReleasesThrowing new hardware and software out at the same time, as Apple does with iOS and iPhone launches, is potentially great for users and sales, but adds more variables to the mix for developers and testers who have to make stuff “just work,” Johnston says. Some have posited that if Apple slowed up its yearly production cycle, unbundling the launch of a big new OS with a big new piece of hardware, Apple could do a better job of preventing bugs from sneaking into final builds. But that could impact Apple’s competitiveness in the fast-paced mobile market.
“I see [Apple's] challenge as one in which they have to try to balance striving for utmost quality with the fact that competitors are coming up with new phones and OS updates that threaten to upend Apple’s relatively strong position,” Jalkut said.
Slowing its product cycle could also impact Apple’s already fragile perception as a leader of innovation. With this in mind, and all those investors to please, it seems unlikely Apple would do that.
And so Apple’s entwined itself in a production cycle that includes shipping with some known bugs and addressing them rapid fire in subsequent platform updates. We are, it would seem, seeing more bugs in iOS than we used to—at least at the outset. With our expectations rising, the richness and complexity of iOS growing at an exponential rate, and a completely new form factor on the horizon, it’s going to be a tough trend to reverse. Perhaps in iOS 9, Apple will be able to reel in more bugs before they hit user’s devices. But as for iOS 8, we’ve probably only seen a fraction of the bugs it’s going to experience over the next year. |
