Phone home

October 19, 2011 11:14 AM

Secret iOS business; what you don’t know about your apps
posted by Artw (125 comments total)

(I would absolutely assume that the same would be the case with any Android apps analysed)
posted by Artw at 11:15 AM on October 19, 2011

People with this kind of knowledge are in severely short supply. I've dealt with developer teams that stare blankly at me when I ask them to justify a network load literally 700 times larger than it should be.

Maybe I've just been super unlucky, but I've met literally no one in the industry who cares about counting bytes anymore. It's a disaster.
posted by odinsdream at 11:24 AM on October 19, 2011 [30 favorites]

Had to jump to the end to get this:

What I discovered about is the result of casually observing some of only a few dozen apps I have installed on my iOS devices. There are about half a million more out there and if I were a betting man, my money would be the issues above only being the tip of the iceberg.

You can kind of get how these issues happen; every man and his dog appears to be building mobile apps these days and a low bar to entry is always going to introduce some quality issues. But Facebook? And Qantas? What are their excuses for making security take a back seat?

Developers can get away with more sloppy or sneaky practices in mobile apps as the execution is usually further out of view.

You can smack the user with a massive asynchronous download as their attention is on other content; but it kills their data plan.

You can track their moves across entirely autonomous apps; but it erodes their privacy.

And most importantly to me, you can jeopardise their security without their noticing; but the potential ramifications are severe.

posted by ZeusHumms at 11:27 AM on October 19, 2011 [4 favorites]

(I would absolutely assume that the same would be the case with any Android apps analysed)

Of course it would. One megacorporation is just like another. The real question is whether a truly open phone would be the same.

Also, speaking of bloated nonsense, I've had XML discussions with 3 separate people today.
posted by DU at 11:28 AM on October 19, 2011 [1 favorite]

If you look for people with grey hair, you'll find people who care about bytes, pixels and processor cycles because we learned to code on machines with 16K of RAM and 1MHz processors. And this is why, despite the fact that I'm no longer in the tech industry, I keep "6502 assembly" and "Atari/Commodore home computers" at the end of my tech resume. Because that's code for "gives a shit about resources."
posted by seanmpuckett at 11:28 AM on October 19, 2011 [58 favorites]

...stare blankly at me when I ask them to justify a network load literally 700 times larger than it should be.

I knew I made that XML comment for a reason. I had someone send me a sample XML file a year or so ago that was, no kidding, 3 orders of magnitude larger than the "legacy" (plain ASCII) file it was replacing.
posted by DU at 11:29 AM on October 19, 2011

seanmpuckett; hey now - I'm under 30 and I definitely care about this kind of thing. Hopefully I'm not totally alone.
posted by odinsdream at 11:30 AM on October 19, 2011 [7 favorites]

I suspect part of the problem has been CEOs getting excited about this trendy new App thing, asking for one from a team that has a dozen other non-trendy-but-essential tasks to be getting on with, and the whole thing being dumped on some intern because it's "easy".
posted by Artw at 11:33 AM on October 19, 2011 [3 favorites]

Wow, that's some unbelievably lazy programming — and a very strong set of security/transparency arguments for browsers, and against trivial apps that ought to be web sites.

Shouldn't a basic review like this be part of the App Store qualifying process at this point? If I ran the App Store, anyhow, this guy would be getting an immediate job offer as an app reviewer, and all these developers' products would be gone from the store pending review and revision.
posted by RogerB at 11:33 AM on October 19, 2011 [3 favorites]

Glad to see companies bringing their A Team B Team … Z Team developers to mobile.
posted by zippy at 11:34 AM on October 19, 2011

I knew I made that XML comment for a reason. I had someone send me a sample XML file a year or so ago that was, no kidding, 3 orders of magnitude larger than the "legacy" (plain ASCII) file it was replacing.

I remember TheDailyWTF once had an XML file encoded inside another XML file. It was amazing.

The use of computing resources acts pretty much like gases: they expand to fill their limits. It seems no matter how much RAM or CPU I've got, I'm always using approximately the same proportion for my every day tasks. In theory these tasks are getting more and more complicated, but still.
posted by kmz at 11:34 AM on October 19, 2011 [4 favorites]

seanmpuckett; hey now - I'm under 30 and I definitely care about this kind of thing. Hopefully I'm not totally alone.

Kids today dont even learn how to program assembly language. I am talking MIT or CMU! Please correct me if I am wrong.

*shakes head sadly. JUMP GETOFFLAWN*
posted by shothotbot at 11:34 AM on October 19, 2011

Unlike iOS, Android at least notifies users when an app will be sharing things like location.
posted by exogenous at 11:35 AM on October 19, 2011 [2 favorites]

Ah, here we go: XML'd XML.
posted by kmz at 11:35 AM on October 19, 2011 [4 favorites]

I've met literally no one in the industry who cares about counting bytes anymore.

Welcome to Agile. Minimum Viable Feature. Get it through the iteration and SHIP IT. Then, well, why work on it, it's shipped?

We had that google developer rant earlier, remember. Accessibility. He implicitly states that Security wasn't important (noting that the Playstation Network made plenty of money).

Efficiency doesn't matter to the developer. That's a hardware problem. The only thing important is shipping code. Period. And I'll bet 95% of the problem could be fixed in a trivial amount of time if companies making these apps cared. But they don't. Efficiency isn't nearly as important as money -- after all, it's not their bandwidth that's being used and their personal information that's being leaked.
posted by eriko at 11:36 AM on October 19, 2011 [11 favorites]

Kids today dont even learn how to program assembly language.

This is stuff that would horrify anyone who has built a rudimentary website though - it's hardly esoteric.
posted by Artw at 11:38 AM on October 19, 2011

The use of computing resources acts pretty much like gases: they expand to fill their limits. It seems no matter how much RAM or CPU I've got, I'm always using approximately the same proportion for my every day tasks. In theory these tasks are getting more and more complicated, but still.

This. When the code runs on your current hardware in an "acceptable" time frame, good enough, because you're just going to upgrade your hardware later. Or at least that is the excuse. But that doesn't stop me from becoming very frustrated at bloated over-the-top code.
posted by mbatch at 11:41 AM on October 19, 2011

Honestly, no matter how much I want to stand on my porch and shake my cane with feeble fury, this really happens for the same reason that toxic sludge gets dumped behind factories. Because bandwidth usage is an externality. They simply don't give a shit: there's no motivation for them to spend the corporate effort to resize the images or cache the data. Why should they? Especially when these apps are written by contractors who will absolutely charge extra for bandwidth optimization tuning.

When it's not an externality, e.g. when Google is sending search results off their servers through their pipes, they'll spend hours optimizing away bytes of data making their pages more efficient because it makes a difference to their bottom line.

Anyway, yeah, there are lots of younger folks who love to hack about on limited resource machines, just look at the fun people are having with Arduinos and other PICs. It's not really about programmer laziness -- though that's a factor -- it's about decisions being made (consciously or not) to simply not give a shit about the externalities.
posted by seanmpuckett at 11:41 AM on October 19, 2011 [21 favorites]

Qantas has the winner at the bottom. Password and PIN sent plaintext over standard HTTP. That seems ripe for the picking.
posted by Mister Fabulous at 11:41 AM on October 19, 2011 [2 favorites]

Unlike iOS, Android at least notifies users when an app will be sharing things like location.

Your qualification is inaccurate -- an app has to ask permission in IOS to gain location, it does not gain it automatically.
posted by cavalier at 11:42 AM on October 19, 2011 [9 favorites]

Kids today dont even learn how to program assembly language. I am talking MIT or CMU! Please correct me if I am wrong.

Really? No assembly at all? I find that hard to believe. I'm pretty sure CS310 is still one of the first core courses at UT, and it covers assembly pretty extensively. Looks like the current flavor is LC-3, a teaching assembly language. Way better than back when I took it and got the instructor that used unsimulated 68k on Macs. Which meant lots of crashing Macs in the labs.
posted by kmz at 11:42 AM on October 19, 2011 [1 favorite]

> Unlike iOS, Android at least notifies users when an app will be sharing things like location.

Actually, it also notifies user anytime an app is sharing a location (prompts at first query), and includes a view to see which apps have requested locations in X hours, and also shows you whenever an app is currently querying your location in the status bar.

The mention of location in the article appears to be City specific (and IP Address), which anyone on the webserver side of the communication can probably get from a browser if they wanted. Again, this would probably show up in an android app as it isn't actually making a call to the location services API, so location preferences don't apply.
posted by mrzarquon at 11:43 AM on October 19, 2011 [4 favorites]

Kids today dont even learn how to program assembly language.

Really? I went to UPitt, and we had an assembly language course. Hell, even the tiny liberal arts school I started at had an assembly language course. Why the hell doesn't MIT or CMU?
posted by odinsdream at 11:43 AM on October 19, 2011 [2 favorites]

Shouldn't a basic review [of security/transparency] be part of the App Store qualifying process at this point?

The lack of security, and total transparency, is part of the business model around mobile devices. They probably deny apps that fail to include tracking back to sites like flurry.
posted by Chuckles at 11:43 AM on October 19, 2011 [2 favorites]

The bandwidth thing is kind of a known issue. There's a reason that the switch to tiered, metered data on AT&T and Verizon were immediately precipitated by iPhone rollouts.

As an Android user on Sprint, this concerns me a bit.
posted by kafziel at 11:46 AM on October 19, 2011 [1 favorite]

And HOL-EEE shit, 1.6 meg thumbnails!
posted by cavalier at 11:48 AM on October 19, 2011

Timely. I had a discussion/fight with my son last week about his iPod hammering the hell out of my network. He's a teenaged boy, so he's got that thing loaded up with all sorts of nonsense and crap. I don't even think he knows what half of it does.

One or more of his apps was trying to contact a server, and since the server wasn't responding, it was blasting my network with thousands of attempted connections per second. This flood was an iPod's imitation of a DDOS attack on my outbound firewall and it was really pissing me off.

My son didn't believe me, even after I showed him the logs from my Smoothwall. I offered him a bet - nuke the thing, update to ios5 and I bet the problem goes away.

Turns out, the old man with 16years of IT experience was right - he needs to be more careful about the crap he downloads.
posted by Pogo_Fuzzybutt at 11:50 AM on October 19, 2011 [15 favorites]

To sort of play devil's advocate, I think that re-using the ginormous PNGs on the mobile site is a good thing. It's something where if they decided to change it, then they can just drop it in and it will change everywhere else. It seems to be that there should be a technical solution for the mobile site- maybe on-the-fly picture scaling, where it caches the thumbnails from the original? That way the art department gets ease of updating and the programmer doesn't have to think about digging up each version of a picture or keeping track of all the different types of thumbnails.

Computers are supposed to make our lives (yes, even programmers) easier.
posted by thewumpusisdead at 11:50 AM on October 19, 2011 [4 favorites]

You know what was a sign of things to come? The "load images" checkbox disappearing from easy to get to menus in mainstream browsers. One of the reasons I use Opera is that it has single-keystroke image loading toggling. (I think the latest versions have messed that up a bit, but it's pretty easy to get the old bindings.)

Even today not everybody has broadband. Yet I haven't heard about people size-optimizing images for years (except for cases like LJ icons that are specifically limited).

The worst is people using IMG height and width to make thumbnails. I don't care what size it displays as if my browser still has to download 500k to load it, dumbass!
posted by kmz at 11:52 AM on October 19, 2011 [6 favorites]

I like that he went through the effort of shrinking that thumbnail before including it in his article. Consistency makes me smile.
posted by Shutter at 11:54 AM on October 19, 2011 [3 favorites]

Holy shit, I hadn't even noticed that they're actually doing that in mobile apps when I made my imagesize rant above.

I think that re-using the ginormous PNGs on the mobile site is a good thing.

This is bullshit.

on-the-fly picture scaling

This is correct way to do things, and contradicts your first point.
posted by kmz at 11:56 AM on October 19, 2011

I think that re-using the ginormous PNGs on the mobile site is a good thing

With limited bandwidth plans now common, it's mostly a good thing for the cellphone companies.
posted by bonehead at 11:58 AM on October 19, 2011 [2 favorites]

It seems to be that there should be a technical solution for the mobile site- maybe on-the-fly picture scaling, where it caches the thumbnails from the original?

Yeah, it's called imagemagick and takes 20 seconds to implement.
posted by empath at 11:58 AM on October 19, 2011 [9 favorites]

I shelled out 99¢ for the Atomic Browser iPhone app just so I could disable images while browsing the web, because default browser Safari won't let you do it. Want to use your iPhone's Safari to visit a image-heavy website with no mobile version? Haha, better be connected on wifi, or that's your data plan being sucked away by banner ads!

The crappy little Nokia I used before the iPhone made it easier to cap web data use, as it simply died whenever you tried to load any page larger than 4MB. I learned to love sites like Wikipedia Mobile that let you disable images sitewide by default.
posted by nicebookrack at 11:59 AM on October 19, 2011 [1 favorite]

Computers are supposed to make our lives (yes, even programmers) easier.

They do. This is really just an example of lazy programming. The issue of thumbnail auto-scaling has been solved repeatedly. There are plenty of other ways to achieve the desired result without the giant network load, AND without making developers' lives harder.

It does, however, require a modicum of planning and discussion on the architectural end of the project.
posted by odinsdream at 12:03 PM on October 19, 2011

Or, what empath said.
posted by odinsdream at 12:03 PM on October 19, 2011

The A-team developers are all off making their own apps or counting their stock options. Since everyone and their grandma wants an app nowadays, you have lots of crappy and inefficient apps.

But security is another deal. To Apple's credit, they are phasing out unique device identifiers to prevent exactly this kind of cross-app correlation where giving up your identity (e.g. logging into Facebook) in one app identifies you to all other apps.

iOS actually has relatively few opportunities for information leakage -- unless you explicitly leak your information by say, logging into Facebook or Twitter via the app. The only issue I can think of that hasn't been addressed is an app's ability to snarf your contact list without permission.

Comparatively, apps that use insecure HTTP and cookies are the least of your worries on Android. Users are encouraged to blindly authorize a laundry list of permissions upon download (and it's not fair to wish for less ignorant users) which gives an app permission to say, send unauthorized SMSs to shortcodes which are billed to your phone number. And at least the App Store's review process prevents fake trojan Netflix apps from appearing very often.
posted by RobotVoodooPower at 12:03 PM on October 19, 2011 [1 favorite]

I've only been developing iOS apps for a couple of months, but what I've seen so far convinces me that you need to be a pretty decent programmer to even think about tackling writing an iOS app. It's a very exacting environment to work with, doubly so if you don't already have Objective-C experience. And I'm pretty sure that any developer bright enough to cope with all the memory management, delegation and what-have-you is also clued up enough to be aware of bandwidth issues on mobile devices.

I don't think it's lack of programming skill that's an issue here at all. I think part of it is that a typical iOS developer is building their app largely in XCode and testing it in a simulator. And even when they test it on an iPad or iPhone, they're probably connected wirelessly and have the benefit of office or home broadband. It's easy to forget the real world.

On top of that, optimising bandwidth may be there somewhere in the spec, but it comes way down the list below things like getting the app working, getting it done on time, and getting the client to sign off. The client doesn't care about bandwidth (unless things are really sluggish). The end-user downloading the app doesn't care. The reviewers probably won't mention it (again, unless network performance is really terrible). So the incentive to put in that extra time and effort isn't there. And did I mention the deadline?
posted by le morte de bea arthur at 12:04 PM on October 19, 2011 [1 favorite]

Sorry, my mistake about iOS location sharing
posted by exogenous at 12:10 PM on October 19, 2011 [4 favorites]

I note with interest that iOS 5 (at least) has a "Cellular Network Data" usage tally, which is resettable by the user. If network usage totals were broken down by app (which would be trivial for the OS to do), users would know without a doubt which apps were the big sucks. So I think bandwidth bloat is something that the device vendors could absolutely help address, simply by providing accountability on the device.

Once users can see how much that recipe or radio station or whatever app is costing them, they can make informed decisions about using those apps, and if they're not happy about what they see then complains can be made that are backed up with real-world data.

(I've got my phone's 3G data profile routed to a WAP gateway that resizes images anyway, which is kind of nice especially for load times when I'm on the road.)
posted by seanmpuckett at 12:12 PM on October 19, 2011 [4 favorites]

The thing that drives me batty is (and I don't know if this is Google or the ROM makers) some apps in the Android market are marked "update automatically" when they are installed. I'm on a prepaid data plan, so downloading a few of those apps automatically wastes my money rather dramatically. I finally got into the habit of turning off the data connection when I'm on wifi (or any time I'm not actually using it) and unchecking auto-update for individual apps when I run across them, but that's nothing like an ideal solution. There's currently no setting to turn off auto-update globally, nor is there a setting to allow auto-updates only on wifi. Here's a nice feature request about it, filed over a year ago, with no response from Google. It's crazy.

Oh, that turned into a rant, didn't it?
posted by dirigibleman at 12:17 PM on October 19, 2011 [1 favorite]

Man, that's insane. App updates are still manual in iOS (but you just do it in one place) and all heavy data usage options (iTunes Match, iCloud, Backup) have a check box to enable usage while on cell phone which is disabled by default.

And annoyingly, it appears ATTWIFI is added as an "auto join" wifi access point by default, with no option to forget it (you can disable the autojoin thou).

Guess I should go create my rogue ap now, called attwifi, and start snooping everyone's insecured app data.
posted by mrzarquon at 12:22 PM on October 19, 2011 [1 favorite]

I don't think it's lack of programming skill that's an issue here at all. I think part of it is that a typical iOS developer is building their app largely in XCode and testing it in a simulator. And even when they test it on an iPad or iPhone, they're probably connected wirelessly and have the benefit of office or home broadband. It's easy to forget the real world.

Easy to forget, sure, but XCode provides brain-dead simple tools to simulate crappy network connections when testing your app. If you've never seen it, here is the Network Link Conditioner. Testing on bad cell networks isn't exactly rocket science.

Android is adding very detailed per-app network usage data to Ice Cream Sandwich (Android 4). See the screenshots.
posted by zachlipton at 12:27 PM on October 19, 2011 [5 favorites]

've dealt with developer teams that stare blankly at me when I ask them to justify a network load literally 700 times larger than it should be.

Sturgeon's Law holds for developers just like fore everything else.
posted by DreamerFi at 12:27 PM on October 19, 2011

Nicebookrack - you might want to try Opera Mini for the iPhone. It heavily compresses all page elements, including images, which can be extremely helpful if you are worried about going over your bandwidth cap.
posted by The Lamplighter at 12:28 PM on October 19, 2011 [1 favorite]

> Android is adding very detailed per-app network usage data to Ice Cream Sandwich (Android 4). See the screenshots.

That it something I would love to see Apple add to the preferences panel (which should be it's own subfield by now) for each app. As they start pushing their phones in lower end markets (more and more folks are getting 200mb plans with their free 3gs, etc), I'd imagine they would want to go that route.

Also include the location services, notification options, and backup options (which are currently stored in their own gui elements that make sense as well), so that I can either fiddle with all my notifications at once, or just fiddle with how facebook is working, etc.
posted by mrzarquon at 12:30 PM on October 19, 2011

This might sound prejudiced, but is this possibly less of a problem with US apps? The screenshots from the story make many of these apps look a little amateurish, to be honest.

My experience with US iPhone apps is that they are generally super polished and load reasonably quickly - typically far faster than the web version. If they were downloading 13 megs of data for a couple of thumbnails I think I'd notice.
posted by The Lamplighter at 12:31 PM on October 19, 2011

Unlike iOS, Android at least notifies users when an app will be sharing things like location.

This statement is false.
posted by Blazecock Pileon at 12:33 PM on October 19, 2011 [2 favorites]

I'm 36 and I got my degree in systems programming back in '98 or thereabouts.

I was, literally, the last person to graduate from that college with a degree plan that required assembly programming. To the school's credit, assembly classes were still offered but they were no longer required for a degree in CS.

The nearest 4 year university to where I live does not even offer assembly classes as an optional course.

I think that's a terrible mistake. Not only is it good from a resource management standpoint for programmers to learn assembly, I think it's good from a theory/philosophy/whatever standpoint. It makes people better programmers. It forces you to get close to the machine, to abandon all the fluff and convenience of higher level languages and get down to the bare bones of what is really going on.

When you program assembly it's you and the machine. You know exactly what is happening, you control absolutely everything, and while programming in any language is all about brutal clarity that's doubly true in assembly. You have to actually pay attention to the registers, and the stack, and so everything else that is really happening.

A programmer becomes a better programmer when they learn assembly. It's like learning recursion, it isn't so much that you will be using it daily, or possibly even ever, but the process of learning improves your brain.
posted by sotonohito at 12:34 PM on October 19, 2011 [5 favorites]

They do. This is really just an example of lazy programming. The issue of thumbnail auto-scaling has been solved repeatedly. There are plenty of other ways to achieve the desired result without the giant network load, AND without making developers' lives harder.

It does, however, require a modicum of planning and discussion on the architectural end of the project.

This is even a bigger problem with security-related issues. At least with network activity there are basic ideas everyone can agree on like downloading the least possible amount of data to get things done. With security issues, it's often either not prioritized as a requirement by whoever is managing the project or it's not understood by either the people running the project or the people writing the code. There are way too many people in software development who see Base64-encoded plaintext data as being more or less the same thing as a thoroughly tested encryption scheme like SSL, just because the data is obfuscated in both cases. And unlike something like bandwidth usage, there's no easy way to automatically test software for having a sound security scheme. I don't see any way to make the majority of apps and sites more secure without pretty much everyone becoming a lot smarter about security, and I don't see that happening any time soon.
posted by burnmp3s at 12:34 PM on October 19, 2011

you know, i always had this hunch that, being without a smartphone, i (1) wasn't missing all that much and (2) would just be pissed off by the thing if i had one. it was really just a baseless, vague hunch...until now.
posted by fetamelter at 12:35 PM on October 19, 2011 [1 favorite]

you know, i always had this hunch that, being without a smartphone, i (1) wasn't missing all that much and (2) would just be pissed off by the thing if i had one. it was really just a baseless, vague hunch...until now.

I bet you didn't write that while sitting on the shitter at work like I am right now.
posted by pjaust at 12:38 PM on October 19, 2011 [31 favorites]

(I would absolutely assume that the same would be the case with any Android apps analysed)

Presumably, bandwidth would be a worse problem on an ad-supported platform like Android. You're not just downloading the content you want, as would be the case on iOS, but also all the advertising cruft that subsidizes your usage.
posted by Blazecock Pileon at 12:39 PM on October 19, 2011

I believe that there are a few iOS apps that do track data usage and per App data usage for those worried about limits.
posted by stratastar at 12:40 PM on October 19, 2011

This is not specifically a phone problem. I see similar problems with routine web pages. Check it out for yourself. For example, in Safari, go to the menu item Window>Activity. Look at some of the web pages you've loaded, it shows each element on the page. I routinely see web pages with over a hundred elements, most of them unnecessary cruft. My particular pet peeve is the "Networked Blogs" widget which displays 25 thumbnails of people I don't care about. Then there's Meebo, which ads tons of useless cruft that I cannot turn off, only hide. And a lot of these useless widgets are mere window dressing on blogs that have main content of just some text and maybe one .jpeg.

I remember the days when people accessed the internet via 56k modems, and web developers went to great lengths to reduce the load times of web pages by reducing the amount of data transmitted. For example, many sites used .gif files with more compression, sacrificing image quality to improve load times. But now it's gone the exact opposite way, people are trying to figure out how much crap they can put into a web page. They're taking up too much memory and slowing down my browser, causing crashes and poor performance generally. If everyone had metered broadband access at home (and many do) this would be a major issue on ALL websites, regardless of whether they were accessed by mobile or desktop computer.
posted by charlie don't surf at 12:42 PM on October 19, 2011 [3 favorites]

Presumably, bandwidth would be a worse problem on an ad-supported platform like Android. You're not just downloading the content you want, as would be the case on iOS, but also all the advertising cruft that subsidizes your usage.

The ad-platforms are the only things that definitely are optimised to minimise bandwidth (to keep infrastructure costs down for the ad providers)
posted by atrazine at 12:45 PM on October 19, 2011

This is stuff that would horrify anyone who has built a rudimentary website though - it's hardly esoteric.

Not anyone, Artw. There are tons of shit website editors that hump tons of drek on top of the useful code, just using the basic tools.

This problem is everywhere - anyone ever noticed how HUGE the size of a Word file jumps when you embed a single picture? or Firefox' many memory problems? - but when you pay through the nose for bandwidth, and have limited RAM on a mobile device, it sucks much more.
posted by IAmBroom at 12:48 PM on October 19, 2011 [2 favorites]

This is not specifically a phone problem. I see similar problems with routine web pages.

YES!

OK, sorry about that, but seriously, I have been ruing the slow disappearance of the "text only" option on websites for years now. Granted, I am a curmudgeon who refuses to install Flash & runs NoScript practically everywhere, but still.
posted by aramaic at 12:49 PM on October 19, 2011

I think that's a terrible mistake. Not only is it good from a resource management standpoint for programmers to learn assembly, I think it's good from a theory/philosophy/whatever standpoint. It makes people better programmers. It forces you to get close to the machine, to abandon all the fluff and convenience of higher level languages and get down to the bare bones of what is really going on.

I didn't come into programming from a formal training background and am basically self-taught/on-the-job trained, but I try to be as cognizant as I can about good software design and engineering practices in general, and I think I strongly agree with this sentiment. Any suggestions for good (free) resources for teaching myself assembly programming?
posted by saulgoodman at 1:00 PM on October 19, 2011 [1 favorite]

Some enterprising whipper snapper might set up a consumer protection/information site.

1. Track data volume by application.
2. Translate that into cost per unit-byte per provider-plan.
3. Post on web
4. Profit?? well ok maybe not
posted by Xoebe at 1:02 PM on October 19, 2011 [1 favorite]

Better yet would be a per-application maximum bandwidth setting, either per day or session. So you end up getting warnings "Holy crap, DumbAss Inc's LoserApp is requesting 10 megs of PNGs, continue yes/no?"
posted by slater at 1:09 PM on October 19, 2011

oops--got too impulsive there. didn't mean to derail. if so, memail me.
posted by saulgoodman at 1:10 PM on October 19, 2011

Heh. This makes me happy. Happy to hear that the competition is still sloppy.

I've been working on a secret project for an Android app, and I care about bandwidth optimization. As such not even the free demo version is going to be ad-supported. Even the local graphics are minimal and not pulled from a server, and I am intending to use as much of the default Android libraries as possible to reduce app footprint and system load.

I might be leaving some money on the table, but I know that people who care about these things are early adopters and they will appreciate a de-cluttered, polite app that does just what it says on the box, that isn't technically a trojan horse - and they will in turn will be more likely to actually use the app and recommend the app to others.

This is an easy choice for this project because this particular app isn't network-centric, I'm not in a position to build a server infrastructure and I can't afford to pay my engineer to stuff it full of garbage anyway. And though future versions may pull text data from other apps or APIs, but even then... all I'll be pulling is text strings.

Keeping it simple works.

I'm also watching this thread and taking notes about users who know what resource optimization is for headhunting purposes.

I don't have money for you right now, but with a little luck and a lot of hard work hopefully I'll be able to hire you in the future. I'm looking for folks with arcane and unusual skillsets - particularly tight, fast code - but stuff like the ability to handle physics, machine vision, fast graphics. I'm looking for engineers as well as programmers. Eventually I'll hopefully need hardware folks, too. I also could be in the market for UI/UX designers and graphics folks who actually know what they're doing, but for now that's one of my hats at the moment and I like designing things.

Though if there are any engineers/programmers out there that have too much free time on their hands right here and now and want to get started for shares/spec, I have some killer ideas for unique apps. I could totally put another iron in the fire and get it started, I just can't afford to pay you a deposit yet. I'm a good, flexible boss, I understand the value of navel-gazing and procrastination, as well as the value of the many kinds of deep hack modes.

For now the focus is Android/Java/Dalvik, but Objective C iOS folks will also be needed. I'm also interested in other IDEs like Ansca's Corona/Lua, Processing or more.
posted by loquacious at 1:18 PM on October 19, 2011 [2 favorites]

Any suggestions for good (free) resources for teaching myself assembly programming?

Code: The Hidden Language of Computer Hardware and Software is an excellent book. It starts literally from wires and bulbs and moves gracefully to microprocessors.

Computer Organization and Design is a great textbook. Use this MIPS simulator for the projects.
posted by odinsdream at 1:21 PM on October 19, 2011 [6 favorites]

isn't technically a trojan horse

I'm going to start putting this in the description of all my projects.
posted by odinsdream at 1:25 PM on October 19, 2011 [7 favorites]

saulgoodman: assembly languages vary depending on the hardware that uses it. What would you like to play with? The languages are almost always free but the hardware will cost you something unless you want to program assembly on your phone or desktop machine, in which case that will be frustrating because it's wrapped in so much OS. Maybe instead you could look at tinkering systems like the Arduino or Lego Mindstorms NXT which allow you to hook up lights and motors and sensors and do stuff in an hour or two for a couple hundred bucks in hardware spend. Ultimately I recommend you connect with a local hackerspace, if available, and talk to someone there about getting into embeddable programming. You get a community and you get help and you get your hands dirty right away doing cool things with random electronic shit -- and you program most of it with assembly language or languages that are the next best thing.
posted by seanmpuckett at 1:30 PM on October 19, 2011 [1 favorite]

Firefox' many memory problems

For me at least Firefox's leaks and general hoggishness seem to be greatly improved in versions 6 and 7. It's been a long time coming, though.
posted by We had a deal, Kyle at 1:37 PM on October 19, 2011 [2 favorites]

@saulgoodman 'Fraid not. I learned way back in the old days and did it on a desktop machine using a textbook I've long since forgotten the title of.
posted by sotonohito at 1:38 PM on October 19, 2011

This makes me think of an issue that came up at work. We got a call about a web site that displays payroll reports had stopped working. We took a look and realized that the page was trying to build a simple table of entries....that had grown to over 100MB of HTML.

There's a cut-off on the web server that blocks pages when they reach this size. We contacted payroll and explained the issue offering to have the site package up a CSV file for downloading, and could we talk about setting up some better filters on the report? They refused the change and we were eventually ordered to increase the web server limitation to 200MB. The reason we were given was that the CSV didn't look as nice when it was printed out as the web page does.
posted by Eddie Mars at 1:40 PM on October 19, 2011 [1 favorite]

saulgoodman: assembly languages vary depending on the hardware that uses it.

of course. that makes sense. different hardware, different switches, different sequences of bits to control them... assembly code is dependent on the potential circuit pathways on the chip, right? and the sequences of bits in the underlying machine language basically close/open different circuit pathways to send power, etc., to different electrical components? I think I understand, conceptually, how assembly languages work; just don't have any practical experience with it.
posted by saulgoodman at 1:40 PM on October 19, 2011

I've only been developing iOS apps for a couple of months, but what I've seen so far convinces me that you need to be a pretty decent programmer to even think about tackling writing an iOS app.

If you can program in C, you can learn to write an iOS app pretty quickly. But the issue here is not programming skill, so much as just not understanding how to design a mobile app so that it is smart about network usage.

This is an issue largely independent of any framework or platform, as anyone who has had to deal with web services can attest to. Saying this is an "iOS problem" is myopic.
posted by Blazecock Pileon at 1:43 PM on October 19, 2011 [1 favorite]

And once you've finished Patterson & Hennesy, there's Hennesy & Patterson. The best class I ever took used the 1st edition of that book & we redesigned the datapath of a MIPS-like.

I don't know that there's that much utility in learning assembly these days (for learning economy, I'd prefer doing things in plain C with only standard library), but Arduino isn't really assembly. Better to buy an AVR development board from Sparkfun & see what the guys at AVR Freaks are up to. Maybe use an classic game console emulator on a modern computer instead. 6502 has a nice architecture that has been studied extensively. My vote for emulation is Atari 400/800 because the GTIA improved on some of the bonehead limitations of the TIA in the 2600 console and you also get POKEY and ANTIC. You also get linear graphics memory, unlike Woz's "cleverness" in the Apple ][.
posted by morganw at 1:47 PM on October 19, 2011 [2 favorites]

saulgoodman; assembly languages are still an abstraction above "open switch 403912". See if your library has the textbook I linked to. It's going to answer your questions pretty quickly.
posted by odinsdream at 1:49 PM on October 19, 2011 [1 favorite]

Really? I went to UPitt, and we had an assembly language course. "

I took that same class at Pitt (probably in '96). Was it using the SPIM MIPS simulator when you took it?
posted by octothorpe at 1:50 PM on October 19, 2011

Yep. I loved that class.
posted by odinsdream at 1:51 PM on October 19, 2011

One of the best courses I had as an undergraduate was a Computer Organization course in which we had a single-board Intel 8080-based computer hooked up to a KSR Teletype (this was in the late 1970s), for which we had to write a program in assembly language to do simple command-line editing (backspace, control-U), hand-assemble the code, store it on a paper tape via the TeleType, load it into the machine via the paper-tape reader, and run the code. It taught me more about the real nuts and bolts of software than any other undergraduate course (including the IBM 360 assembly language course).

Also, Computer Systems: A Programmer's Perspective (2nd Ed.) by Bryant and O'Hallaron is an excellent text for learning about computer architecture as it relates to programming. They mainly use the x86 for examples.
posted by Crabby Appleton at 1:52 PM on October 19, 2011

Re: ads on android can easily blocked by an app, which also blocks ads in the browser. I have no fucking idea how people can surf the net without blocking ads. Everytime I sit down at a computer, smartphone what ever, I install something to block ads.
posted by handbanana at 1:56 PM on October 19, 2011

I learned assembly (well, I did exercises in it for three months), but it's not why I know not to download or upload more than is necessary.
posted by ignignokt at 2:00 PM on October 19, 2011

We took a look and realized that the page was trying to build a simple table of entries....that had grown to over 100MB of HTML.

!!!

That's... that's a fucking atrocity. WTF are they trying to display? Gotta be at least hundreds of thousands of rows. I can't even imagine how long it takes a browser to load a page like that.
posted by kmz at 2:00 PM on October 19, 2011 [3 favorites]

Maybe I've just been super unlucky, but I've met literally no one in the industry who cares about counting bytes anymore. It's a disaster.
posted by odinsdream

Even worse when you're buying bandwidth by the kilobyte as its sold in many developing nations...

but seriously, this thread, I don't know whether to jump in and rant about bandwidth hogging websites not thinking about the rest of the world in the wide web and how it may not be on unlimited fat pipe shoved straight into the guts or just sit back and read it and keep nodding my head and yelling at the screen with my fingers off the keyboard
posted by infini at 2:01 PM on October 19, 2011 [2 favorites]

One vaguely related thing I've always wondered about: do location aware smart phone apps generally have to ping a location server constantly/intermittently, or do they just sync to some initial set of location coordinates and then use the accelerometer to track/calculate changes in position on the client side relative to those initial starting coordinates? Or is it some combination of the two, with intermittent resyncing? GPS based apps don't just ping servers constantly, do they?

Anyway, on the article: this is crazy stuff. What's up with all the round trips to different servers? Seems to me apps should always do as much as possible client side.
posted by saulgoodman at 2:02 PM on October 19, 2011

hey! I went to UPitt too! Go Panthers etc
posted by infini at 2:02 PM on October 19, 2011

They refused the change and we were eventually ordered to increase the web server limitation to 200MB.

In the Glorious New Regime such people will be dragged from their offices by jackbooted thugs and unceremoniously sent to spend the rest of their short lives in the arsenic mines.

...yes, that's right, we're not even going to do them the honor of laboring in the Antarctic Copper Smelting Complex.

Nope, straight to the arsenic mines, to die amidst choking fumes in a sunless nightmare.
posted by aramaic at 2:05 PM on October 19, 2011 [1 favorite]

The use of computing resources acts pretty much like gases: they expand to fill their limits. It seems no matter how much RAM or CPU I've got, I'm always using approximately the same proportion for my every day tasks. In theory these tasks are getting more and more complicated, but still.

Working in bioinformatics, I have to write code that is performant, portable and scalable. A major trade-off is a set of assumptions about input. The fewer complications with input, the faster my code can be. In bioinformatics, in particular, this can be a dangerous thing, as data integrity is a real problem; systems biology has increased data collection at an exponential rate, and errors and format tweaks become issues over time. Wrappers and frameworks handle a lot of edge cases that developers need to account for, which in turn eats away at memory and performance improvements in hardware. That's not to say this is the case for all projects, but I suspect that the need to support umpteen different encoding or serialization formats, say, has built up cruft in code libraries over the years. But this remains a different issue entirely from unnecessarily pushing high-resolution bitmaps to low-res mobile screens, or not caching files which do not change frequently, etc.
posted by Blazecock Pileon at 2:06 PM on October 19, 2011 [1 favorite]

If you look for people with grey hair, you'll find people who care about bytes, pixels and processor cycles because we learned to code on machines with 16K of RAM and 1MHz processors. And this is why, despite the fact that I'm no longer in the tech industry, I keep "6502 assembly" and "Atari/Commodore home computers" at the end of my tech resume. Because that's code for "gives a shit about resources."

The problem, though, is bandwidth caps on mobile devices is ridiculous low. Like in the megabytes and in places like Australia or Canada there are even annoying bandwidth caps for home users (they're coming to the U.S. but aren't too onerous too far)

Really? I went to UPitt, and we had an assembly language course. Hell, even the tiny liberal arts school I started at had an assembly language course. Why the hell doesn't MIT or CMU?

I'm pretty sure they do. I had MIPS assembly when I was in college. I had already taught myself x86 when I was in highschool.
posted by delmoi at 2:08 PM on October 19, 2011

> do location aware smart phone apps generally have to ping a location server constantly/intermittently, or do they just sync to some initial set of location coordinates and then use the accelerometer to track/calculate changes in position on the client side relative to those initial starting coordinates? Or is it some combination of the two, with intermittent resyncing? GPS based apps don't just ping servers constantly, do they?

In iOS, the devices location is provided by Location Services, a specific set of code that Apple maintains and you can query with your code as a developer. The services interact with the aGPS/GPS chipset on the phone to determine position. I believe it just provides raw GPS Coordinates if it doesn't have any data signal (so your GPS app works without a data connection), but if you want a street number, it has to query outside servers first to do a lat/long to map lookup.

With location being a central service, it allows for the master "on/off" switch for the service, and also for the graceful background processing that Apple allows apps to be location aware without draining battery by constantly querying the GPS system. For example, the geofencing aspect of reminders is probably based on the fact that apps can request change of location notifications whenever the iPhone hops to a new cell tower (usually meaning you've moved a few blocks). Reminder.app lays dormant until a cell tower change, the iPhone gives it the rough coordinates base on that, and checks to see if it needs to alert you about something near that area, if not, it goes to sleep. If it does, it goes from sleep to idle, and gets a more accurate fix, then issues a notification.
posted by mrzarquon at 2:14 PM on October 19, 2011

One vaguely related thing I've always wondered about: do location aware smart phone apps generally have to ping a location server constantly/intermittently, or do they just sync to some initial set of location coordinates and then use the accelerometer to track/calculate changes in position on the client side relative to those initial starting coordinates? Or is it some combination of the two, with intermittent resyncing? GPS based apps don't just ping servers constantly, do they?

I'd think it's some combination of the two. Although there isn't a location server as such. GPS satellites broadcast messages with info that GPS chipsets, in phones or other devices, translate into useful data. And there are supplemental data sources for location, as mrzarquon notes.
posted by ZeusHumms at 2:17 PM on October 19, 2011

It's not anything that's going to be relevant to app bandwidth use, at any rate.
posted by Artw at 2:23 PM on October 19, 2011

:( This post needs an 'Everything Is Terrible' tag.
posted by danny the boy at 2:41 PM on October 19, 2011

and then use the accelerometer to track/calculate changes in position

It won't be this. I know this because I'm working with trying to design something that uses inertial navigation.

Inertial navigation is really complicated, processor-heavy and the accel/gyro chip is a massive battery hog, especially when running in high precision modes. The amount of data an accel/gyro chip can output is enormous and unwieldy, and it needs a lot of massaging through various DSP or DSP-like algorithms to be smooth and useable.

I have yet to see a smartphone application that does real time, long term and long distance inertial navigation. You basically need to turn the accel/gyro chip up to it's highest resolution/accuracy mode to get any kind of coherent accuracy that's more than a few dozen feet of movement, and then the code and math behind inertial navigation will make your average non-engineer programmer/coder gibber with terror and phear.

Most phones actually get most of their "GPS" data from cell towers themselves through enhanced GPS or whatever they call it. The GPS chip gets a local fix once in a while, then the phone just triangulates to within a few hundred feet based on the locations of the cell towers.
posted by loquacious at 2:45 PM on October 19, 2011 [2 favorites]

What bugs me is that I can't download an app or podcast (through iTunes) larger than 20MB over a 3G connection. But if an app wants to download 30MB over that same connection without telling me, that's fine.
posted by Gary at 2:46 PM on October 19, 2011 [6 favorites]

loquacious: "and then use the accelerometer to track/calculate changes in position

It won't be this. I know this because I'm working with trying to design something that uses inertial navigation.

Inertial navigation is really complicated, processor-heavy and the accel/gyro chip is a massive battery hog, especially when running in high precision modes. The amount of data an accel/gyro chip can output is enormous and unwieldy, and it needs a lot of massaging through various DSP or DSP-like algorithms to be smooth and useable.

I have yet to see a smartphone application that does real time, long term and long distance inertial navigation. You basically need to turn the accel/gyro chip up to it's highest resolution/accuracy mode to get any kind of coherent accuracy that's more than a few dozen feet of movement, and then the code and math behind inertial navigation will make your average non-engineer programmer/coder gibber with terror and phear.

Most phones actually get most of their "GPS" data from cell towers themselves through enhanced GPS or whatever they call it. The GPS chip gets a local fix once in a while, then the phone just triangulates to within a few hundred feet based on the locations of the cell towers."

Not to mention the fact that the more you hit the accel/gyro chip and the GPS and the data, the faster your battery dies.
posted by Samizdata at 2:48 PM on October 19, 2011

Not to mention the fact that the more you hit the accel/gyro chip and the GPS and the data, the faster your battery dies.

That's one of the first things I mentioned. It's currently in my top 3 worries about the project I'm working on, because it requires the high precision modes of the accel sensor calls in the android API as well as full screen brightness. Newer phones with OLED or AMOLED screens handle this a lot better, and the newer gyro chips aren't such huge battery hogs, but... still.

Using high precision mode for the accel/gyro chip is like throwing gasoline on a brush fire.
posted by loquacious at 3:00 PM on October 19, 2011

The Lamplighter: I actually used Opera Mini on my previous crappy little Nokia and loved it, specifically for the image disabling/compression. I started out on iPhone switching between Opera & Atomic Browser Lite, and settled on Atomic for...some reason? I think the ability to print, I forget. I should go poke at Opera again to see what's new.

But in general, yeah, yay Opera Mini for being gentle to crappier phones/connections! And if there are other apps that let you disable images / compress downloaded content, I definitely want those.
posted by nicebookrack at 3:04 PM on October 19, 2011

What bugs me is that I can't download an app or podcast (through iTunes) larger than 20MB over a 3G connection.

Wait, really? You can't even do that manually? Like, "yes, I know I'm on a 3G connection, I still want to download that 50MB podcast!"

I actually used Opera Mini on my previous crappy little Nokia and loved it, specifically for the image disabling/compression. I started out on iPhone switching between Opera & Atomic Browser Lite, and settled on Atomic for...some reason? I think the ability to print, I forget. I should go poke at Opera again to see what's new.

Do keep in mind that Opera Mini uses a similar system to the upcoming Amazon Silk. All traffic goes through Opera's proxies. I don't mind myself (I'm not going to any sites with important passwords), but others might.
posted by kmz at 3:07 PM on October 19, 2011

Yeah you need to jailbreak to break that restriction. I'm at home on wifi and somehow iOS has managed to download 200mbs over 3g. What the fuck.
posted by stratastar at 3:10 PM on October 19, 2011

Wait, really? You can't even do that manually? Like, "yes, I know I'm on a 3G connection, I still want to download that 50MB podcast!"

Not through the iTunes App. Other apps like SimpleCasts will happily download it without even telling you how big the file is. However, they store it internally and can't transfer it to be with the rest of your Music.
posted by Gary at 3:24 PM on October 19, 2011

It'll also happily download 5x10mb files. :-(
posted by Artw at 3:40 PM on October 19, 2011

I am very curious to see how the battle plays out, between networks that cap your bandwidth (driving users to use workarounds to block images, typically ads and videos) and advertisers who really, really, really want you to see those ads.
posted by davejay at 3:54 PM on October 19, 2011

Oh, and considering how much effort we put in at my company to ensure that individual images for our web application are condensed into a small number of sprites and optimizes to a shockingly low number -- and this is for an internal app, it isn't even public-facing -- the waste demonstrated so clearly in that article is painful to look at. It actually hurts me. Deep inside. In places you're not allowed to touch.
posted by davejay at 4:00 PM on October 19, 2011 [4 favorites]

I know... I know i am fussy about these things, doing the same kind of optimization you are, but this... it's several orders of magnitude beyond where I would start getting upset at people.
posted by Artw at 4:05 PM on October 19, 2011

Not being an iOS developer I can't be sure about this, but the 'location' the apps are sending is very close to the language setting - I suspect it is actually the timezone.

People who care enough about ads and bandwidth usage to do something about it seem to be pretty rare. I have an ad supported application on Android, with approximately 1000 active users. I provide an option where for a one time fee of approximately $2, the ads are permanently removed. So far, less than 10 people have opted for that option. The interesting thing though is that the income from those users is higher than the income from the ads.
posted by netd at 4:15 PM on October 19, 2011 [2 favorites]

Echoing what davejay just said.

My personal silver lining to this article is that I now feel like an infinitely better developer than I did before reading it.
posted by Magnakai at 4:16 PM on October 19, 2011

we have a major app in use (public transport) that sends tonnes of uncompressed SOAP to unsuspecting punters. We have a compressed json version as well - but -
government + itil + risk aversion + ignorance = we can't do nothing to change it.
posted by the noob at 4:19 PM on October 19, 2011 [2 favorites]

And at least the App Store's review process prevents fake trojan Netflix apps from appearing very often.

And yet "Teenage Mutant Ninja Turtles" lasted for six weeks before they removed it. Apple appears to be lax about certain kinds of content (copyright infringement) while amazing uptight about others (making political points about Apple's editorial policy).
posted by JHarris at 5:06 PM on October 19, 2011 [2 favorites]

But in general, yeah, yay Opera Mini for being gentle to crappier phones/connections! And if there are other apps that let you disable images / compress downloaded content, I definitely want those.

iCab mobile has a 'Display Images' switch, FWIW.
posted by ZeusHumms at 5:39 PM on October 19, 2011

Thanks for the info. Duh--of course there's a dedicated GPS chip with a service layer. (I just totally spaced on the existence of GPS chips for some reason and pictured all these servers communicating with GPS satellites and spitting out coordinates all day. Maybe it's because the very limited extent of my professional work with GPS data has been in the context of GIS mapping databases.)

This thread should offer lots of helpful guidance on what not to do when I get ready to actually delve into developing for mobile devices. (Suppose I'll have to get a proper mobile device first, though. Can't imagine developing on an iPod touch.)
posted by saulgoodman at 5:48 PM on October 19, 2011

Other apps like SimpleCasts will happily download it without even telling you how big the file is. However, they store it internally and can't transfer it to be with the rest of your Music.

And there's an issue with this that Marco Arment, creator of Instapaper, highlighted a few days ago in a post on iOS 5 clearing Caches and tmp directories. App developers are being directed to save content that doesn't need to be backed up but should always be available in Caches and tmp directories inside the app. These are presumed to be redownloadable, and can include pictures, podcasts, audio recordings, and downloaded articles/ebooks/comic books.

The problem is, in iOS 5, when the device is low on space, these directories are fair game for being cleared out automatically. The user is never told this, it just happens. One download of anything - a new app, a newly downloaded article, a song or audio file, and woosh, everything goes bye. Stockpiled articles, podcasts, etc... deleted.

A common scenario: an Instapaper customer is stocking up an iPad for a long flight. She syncs a bunch of movies and podcasts, downloads some magazines, and buys a few new games, leaving very little free space. Right before boarding, she remembers to download the newest issue of The Economist. (I think highly of my customers.) This causes free space to fall below the threshold that triggers the cleaner, which — in the background, unbeknownst to her — deletes everything that was saved in Instapaper. Later in the flight, with no internet connectivity, she goes to launch Instapaper and finds it completely empty.

The user is put out, and blames the developer. There are lot of scenarios where redownloading isn't going to happen at all, or would be expensive (3G roaming, low transfer networks, slow broadband).
posted by ZeusHumms at 5:51 PM on October 19, 2011 [4 favorites]

That "cleaning" process is well intentioned technically but dumb as hell for the end user.

Way near the top of UX rules is "don't delete anything the user doesn't tell you to delete if you can't recreate it effortlessly." If my books or podcasts get "cleaned up" while I'm away from WiFi, then that rule has been broken. "Where is my content? I have to sync to get it? Why? IT was just there? Where did it go?!"

If they get "cleaned up" while I'm on a pay-per-byte 3G plan, then adhering to that rule costs my users money. "Where is my podcast? I need to download it again? But I'm on a bus and I want to listen to it now, that is why I subscribed to it and why I made sure I a dozen episodes before we left town! Downloading them again will set me back $20 in bandwidth!"

This is a big stumble for user confidence (one of the most important intangible benefits of Apple devices, and part of the It Just Works scheme) and it needs to be fixed ASAP.
posted by seanmpuckett at 6:55 PM on October 19, 2011 [8 favorites]

The directions for using Cache and Tmp are also to manage stuff on the Apple side of things: anything in the App directory gets backed up to the 5gb cloud storage each user gets free with their iOS device (not counting music/ebooks/applications themselves/other content that can just be redownloaded from apple store).

So until iOS 5, developers have been told that cache and tmp are almost equal to app in terms of storage. But if you want data to be persistent to the user, regardless of their internet connection, then app is the only directory they should store their data in.

Of course, the problem is three fold: 1 apple probably built their icloud service offerings on what is in the app directory (ie, assuming what the majority of people would want to have backed up) while 2. at the same time telling developers that cache / tmp is for stuff that doesn't need to be backed up (under iOS 4 terms, where that meant stored on the users local iTunes machine), 3. then introduces in iOS 5 the ability to auto purge cache/tmps, assuming (wrongfully) that all app developers were putting things that they didn't want autopurged in the App directory, because the function of Auto-Purge caches didn't exist until iOS5 and iCloud were introduced.

Now Apple is stuck in a problem situation: continue what it is doing (deleting data it really wasn't clear to developers that it planned to put on it's "to purge" list), ask developers to store data in the app directory that shouldn't be purged (asking developers to change code, maybe increasing the baseline "what people expect to be backed up by icloud" storage amount), or stop auto purging data (and running into the problem of devices being full and users having to manually delete data, not what they wanted to accomplish).

I'd hope Apple would realize that the second option would be the cheapest, long run (backup storage is cheap), and maybe credit the few developers who bought the extra 10gb of space when they decide to offer 10-16gb for free.
posted by mrzarquon at 7:41 PM on October 19, 2011

shothotbot: Kids today dont even learn how to program assembly language. I am talking MIT or CMU! Please correct me if I am wrong.

All CMU CS and ECE majors need to take 15-213, which does cover some assembly IIRC. At the very least, the second and third assignments require you to run a program in a debugger and modify its behaviour (including constructing a stack-based buffer overflow), which would require some knowledge of assembly.

I believe there are other courses which might not teach x86 assembly, but perhaps MIPS or ARM, or some toy assembly language. The point being that you learn enough to know how to code in any assembly language given the right reference materials.
posted by destrius at 8:14 PM on October 19, 2011

If you look for people with grey hair, you'll find people who care about bytes, pixels and processor cycles because we learned to code on machines with 16K of RAM and 1MHz processors.

A-fucking-men. This is the reason why jQuery is absolutely forbidden from any web code my team develops, and is similarly the reason why HTML5 is doomed to fucking failure.

I would love to rant more about this because it is a rant-worthy topic, but quite honestly I'm currently working at 3 in the morning on a company software project and I simply cannot be assed.
posted by Civil_Disobedient at 12:06 AM on October 20, 2011 [2 favorites]

If you've got time later, I'd love to see that rant about HTML5 and resources.
posted by harriet vane at 6:19 AM on October 20, 2011 [2 favorites]

I learned a couple of toy assembly languages in college (Java bytecodes, and MIPS).

Honestly what people are missing is compilers knowledge. There are some optimizations that the compiler can make very easily, and others that are obvious to a human but the compiler has no clue about. I recently made a function run 8 times faster by converting it from a loop to an easily-pipelined unrolled loop; there's no way I would have done that without having some knowledge about how the compiler worked.

Meanwhile I see code all the time that does ugly, incomprehensible things that don't actually increase performance because the compiler would have made the same optimization. Many C++ people still return values by reference, or use macros instead of inline functions, for "performance" reasons. (It's not limited to C++ -- the best example of this I saw recently was in LISP.)

Does any of this have to do with bandwidth hogs? I think not.
posted by miyabo at 7:44 AM on October 20, 2011 [1 favorite]

DU: "I knew I made that XML comment for a reason. I had someone send me a sample XML file a year or so ago that was, no kidding, 3 orders of magnitude larger than the "legacy" (plain ASCII) file it was replacing."

Yeah, but how did it compress? My recollection was that XML compresses extremely well, given the large amount of redundant data in it. If you're not gzipping significant network traffic, I'm not sure you can complain about file formats.

The format failed to live up to its expectations and design goals, no doubt, but I'm not quite sure that this is the best point of comparison.

That said, JSON's totally the way forward for web services. I just started working on a project utilizing CouchDB, and am simply flabbergasted that we haven't been doing things this way all along. (And, after working on another project that literally had an 800-page DB Schema, the document model starts looking awfully good)
posted by schmod at 7:48 AM on October 20, 2011 [2 favorites]

A-fucking-men. This is the reason why jQuery is absolutely forbidden from any web code my team develops, and is similarly the reason why HTML5 is doomed to fucking failure.

I do hope you're kidding :) By that logic, the web is doomed to failure, because all those processor cycles spent rendering web pages could be used far more efficiently if every page was hand-coded in X86 assembly. As far as HTML5, do you not think that a browser's built-in video player can't be better optimized for its environment than the frickin' Flash plugin?

jQuery comes in at about 32KB when compressed and minified. Since it's used all over the web, you can serve it from Google's CDN and your users will most likely have it in their browser caches already. Or use one of its competitors, many of which are larger. And once the library is downloaded, it can be reused on every page of your site with no network overhead.

Unless you're doing web development for highly resource constrained embedded systems (at which point you're arguably doing it wrong to begin with), developer time is a far more precious resource than computing cycles (in the general sense, there's plenty of room for targeted optimization where it really counts). Kick-ass libraries like jQuery save enormous amounts of time for developers, which is why so many people love and use them. Frankly, that's worth the slight performance and resource penalty, because so many apps and features wouldn't be built at all without such powerful tools.
posted by zachlipton at 12:32 PM on October 20, 2011 [5 favorites]

Boy, y'all are a stingy bunch. Here's my counterpoint:

Image rescaling frustrates a lot of optimization efforts. If you have one image at 100k and resize it so it's only 50k, great that you saved the space but now you're serving a dynamic file from your application server instead of a static file from your CDN. If the user then goes to a page requiring the 100k version (obvious example: thumbnail -> full size), you've actually cost yourself an extra 50k for that user in addition to slower loading and using extra computational resources and requiring extra development time and adding a layer of complexity that needs to be tested.

Hating on jQuery? Sure, it's technically less efficient than a perfectly developed solution that's customized for your needs. But the custom thing is never perfectly developed, and your needs will change. Two years from now when your application needs to select elements within a particular form instead of the whole page, it's nice that the one-line change to jQuery code still leads to a pretty-efficient implementation, instead of hoping you'll have the time to figure out the glut of custom code and do new changes as perfectly as the original implementation.

Furthermore, it's easy to think of development time as a separate thing, where taking the time to optimize may cost money and time but leads to higher-quality applications. But that's not necessarily true; having two 80% efficient apps in the same market means you get to compete on features, and you may find out from seeing your competitors' successes that the code you optimized actually reduced efficiency compared to the code you didn't think to write. If your competitor couldn't afford to optimize 100% and you did, you'd only have your own app as a basis for comparison and never realize it at all.

Some of the examples in the article are obviously at the other extreme, and certainly the security concerns are universal. But I think many of you are overlooking the benefits we've enjoyed because "lazy programming" meant that applications actually got to be released.
posted by Riki tiki at 1:44 PM on October 20, 2011 [2 favorites]

Image rescaling frustrates a lot of optimization efforts. If you have one image at 100k and resize it so it's only 50k, great that you saved the space but now you're serving a dynamic file from your application server instead of a static file from your CDN. If the user then goes to a page requiring the 100k version (obvious example: thumbnail -> full size), you've actually cost yourself an extra 50k for that user in addition to slower loading and using extra computational resources and requiring extra development time and adding a layer of complexity that needs to be tested.

This example is very, very, very far removed from what's being discussed in the FPP.

The thumbnail being used was at least 480 times larger than required, not 2 times larger than required.
posted by odinsdream at 1:56 PM on October 20, 2011 [1 favorite]

This is the reason why jQuery is absolutely forbidden from any web code my team develops

That seems needlessly cruel to your developers - Making developers deal with the vagaries of the DOM and it's various cross browser inconsistencies will cause more problems and inefficiencies than using a solid library will, and though I've seen jQuery overused in horrible ways there is o reason it can't be used in an efficient and responsible manner. I;d actually say that using jQuery has freed me up to become a better JS developer.

Your team is probably cursing you, and with good reason.
posted by Artw at 2:06 PM on October 20, 2011 [2 favorites]

Until it is explained further, I will simply assume that he was working at 3am fixing DOM problems that would have been easily solved by jQuery. That's totally unfair, but the only way I can reconcile calling out jQuery of all things when the examples given in the original link are hundreds of times worse than any of jQuery's sins.

But maybe he has great reasons, and I'm really just egging him on for a follow-up.
posted by Gary at 2:49 PM on October 20, 2011 [1 favorite]

What you really don't want to do is let your designers become aware of jQuery UI or any jQuery plug-ins, as they will be begging you to add all kinds of random crap all over the place.
posted by Artw at 2:51 PM on October 20, 2011 [1 favorite]

loquacious: "Not to mention the fact that the more you hit the accel/gyro chip and the GPS and the data, the faster your battery dies.

That's one of the first things I mentioned. It's currently in my top 3 worries about the project I'm working on, because it requires the high precision modes of the accel sensor calls in the android API as well as full screen brightness. Newer phones with OLED or AMOLED screens handle this a lot better, and the newer gyro chips aren't such huge battery hogs, but... still.

Using high precision mode for the accel/gyro chip is like throwing gasoline on a brush fire."

Yeah, well, problem being is that my chipset was not in high precision mode. Sorry 'bout that.
posted by Samizdata at 7:25 PM on October 20, 2011

Instead of banning libraries outright, just make everyone use a profiler.
posted by ignignokt at 7:30 PM on October 20, 2011 [2 favorites]

You probably want a library to handle your AJAX too - another area where you are going to be better off going with what jQuery gives you rather than rolling your own and hitting every problem they've already solved (and maybe missing a few).
posted by Artw at 8:31 PM on October 20, 2011

Striking It Rich In The App Store: For Developers, It's More Casino Than Gold Mine
posted by Artw at 10:20 AM on November 2, 2011