On The Importance of Overdraft Protection

July 15, 2009 10:37 AM

Overcharged for cigarettes.A NH man was charged $23,148,855,308,184,500 for a pack of cigarettes-- a "little" over a quadrillion dollars per cigarette. Jokes about tobacco taxes aside, it wasn't an isolated incident. Numerous Visa customers discovered the same charge.

Curiously, the charge is both enormous and yet not a clear overflow value such as 2^n - 1. The number itself only shows up in Google in reference to related stories. Drop the last two zeroes and you find a few references to a bug in an obsolete database package.

posted by justkevin (97 comments total)

Also: Unruly Teen Charges $23 Quadrillion At Drugstore
posted by Kadin2048 at 10:40 AM on July 15 [1 favorite]

I think Michael Bolton might not have gotten the coding exactly right on the Superman 3-inspired program he had been bragging about.
posted by Astro Zombie at 10:42 AM on July 15 [7 favorites]

"For that amount of money, I could actually own Wolfgang Puck himself," Seale said.

Oh, you do have your sights aimed high, Mr. Seale.
posted by boo_radley at 10:42 AM on July 15 [1 favorite]

IT'S HAPPENING JUST LIKE JOE THE PLUMBER SAID IT WOULD
posted by brain_drain at 10:47 AM on July 15 [22 favorites]

That is $23 quadrillion (£14 quadrillion) - many times the US national debt.

Oh, for Heaven's sake! Give Obama a chance, people! He's only been in office for seven months! The Dems just got their filibuster-proof majority last week! $23 quadrillion isn't out of reach! Keep the audacity of hope alive!
posted by It's Raining Florence Henderson at 10:48 AM on July 15 [11 favorites]

You know, Visa could do this intentionally on the chance that some high maintenance woman's husband doesn't notice, and pays it.
posted by weapons-grade pandemonium at 10:49 AM on July 15 [1 favorite]

Personally, I feel worse for the suckers who got billed $23 quadrillion and just paid the balance without questioning it.
posted by briank at 10:51 AM on July 15 [18 favorites]

I was going to say that since it went past their credit-limit-detector, presumably it wasn't on the backend transaction server that this happened. Some account-browsing front-end, right? Except the overdraft charge indicates the servers really did think he was deep in the red.

This is the part where the Free Market Disciples come in and explain that, because CC companies have so much at stake, their software is top-notch and rigorously tested and we should have nothing to fear wrt data losses and the like.
posted by DU at 10:52 AM on July 15 [5 favorites]

Got to pay for that health care plan somehow...
posted by Pantengliopoli at 10:54 AM on July 15

If I were him, I'd apply for a 0% balance transfer card under a fake name.
posted by blue_beetle at 10:56 AM on July 15 [6 favorites]

With $23,148,855,308,184,500, which is 381 times global GDP, you could buy the entire world, Wolfgang Puck included.
posted by jedicus at 10:57 AM on July 15 [4 favorites]

It's a good thing he bought the cigs in NH instead of NYC, or the taxes on that would really hurt.
posted by exogenous at 10:58 AM on July 15 [14 favorites]

This is why I don't use automatic bill pay.
posted by bottlebrushtree at 10:58 AM on July 15 [6 favorites]

"For that amount of money, I could actually own Wolfgang Puck himself," Seale said.

Are you looking for a better dining experience? Do you have a "little" over a quadrillion dollars? While some companies may try to sell you a single Wolfgang Puck for that price, we at ChoiceChefs, Intl. realize that the desires of the insanely wealthy go beyond the limited scope of "fusion" cuisines. Looking for authentic French food from an authentic French chef? A meager $4 million USD! Or maybe you want a Hawaiian burger? We can provide you with a Hawaiian chef AND his kitchen for $5 million USD! How about a taste of home? We have a variety of authentic mothers from a range of ethnic backgrounds, all ranked "best home-made meal" by our panels of expert critics, and a steal at $2.2 million USD!

Remember, you may be insanely wealthy, but you don't need to spend it all on your dining pleasures and whims. Let ChoiceChefs, Intl. fulfill your chefs, and save a little for your star destroyer!
posted by filthy light thief at 10:58 AM on July 15 [3 favorites]

I could buy and sell Wolfgang Puck at least three times for that much money!
posted by Astro Zombie at 11:00 AM on July 15

I suspect that the overdraft charge is applied by a database trigger which fires when the balance is updated. Because the charge did pass the credit limit logic (presumably in a Business Logic Layer somewhere), the bug had to be in the database layer. My geek curiosity is killing me...
posted by JeffK at 11:00 AM on July 15

And I thought $10.50 was a lot for a pack of smokes!
posted by futureisunwritten at 11:01 AM on July 15 [1 favorite]

For $23 quadrillion, I'll sell you Wolfgang Mozart. Alive.

(after an intensive, well-funded time travel research program)
posted by DU at 11:05 AM on July 15 [14 favorites]

Curiously, the charge is both enormous and yet not a clear overflow value such as 2^n - 1. The number itself only shows up in Google in reference to related stories. Drop the last two zeroes and you find a few references to a bug in an obsolete database package.

I wouldn't be surprised if it's something like this that's the cause. A common business-expense-filing software package I've used in the past had a similar bug one day; typically the software package was linked to each user's corporate credit card account, so it could download the charges off the credit card account into a database, and then I came along and updated the data about each charge (what it was, what business project it applied to, etc.) and then submitted each expense electronically.

One day, however, the software had a weird bug: it downloaded each charge off each user's credit card account, but then added five million dollars to each figure and THEN posted it to each user's database. So, for example, Tom Businessman's $19.50 car to the airport was showing up on my screen as a $5,000,019.50 car to the airport. The poor expense-account-processing department spent an hour being flooded with calls from secretaries all asking "what the hell?" and they called tech support, who were all able to fix the problem pretty quickly. Near as anyone could figure out, the software just had the hiccups.
posted by EmpressCallipygos at 11:05 AM on July 15

This is horrible journalism, because they don't even go into whether or not the guy wound up paying it anyway.
posted by jeremy b at 11:06 AM on July 15

This is the part where the Free Market Disciples come in and explain that, because CC companies have so much at stake, their software is top-notch and rigorously tested and we should have nothing to fear wrt data losses and the like.

I think a Free Market Disciple would not argue that the software is top-notch or rigorously tested, but rather that rational free market actors optimize according to the following criterion:

if (cost of eliminating error > loss from error * probability of error) then do nothing

In other words, on average, in the long run, Visa bets that it will make more money with shoddy software, despite its screwups, than by paying for perfect software.

That's what a hypothetical FMD would argue, anyway. An easy counterargument is that Visa may not actually be the best entity to make that judgment. Another would be that the fallout from some errors is so potentially devastating that, despite their expected rarity, it's still worth forcing Visa to guard against them.
posted by jedicus at 11:07 AM on July 15 [2 favorites]

But does he get the cashback rewards??? Because 1% of a quadrillion dollars would be pretty awesome.
posted by GuyZero at 11:08 AM on July 15 [14 favorites]

jeremy b: "This is horrible journalism, because they don't even go into whether or not the guy wound up paying it anyway."

He got a payday loan from Check into Cash to cover it.
posted by boo_radley at 11:08 AM on July 15

Does he get to keep the 231,488,553,081,845 rewards points? He may be able to get Puck anyway.
posted by asusu at 11:08 AM on July 15 [5 favorites]

This is the good thing, actually, about digital as opposed to human errors. In the nature of things they're more likely to be so wildly wrong as to be immediately apparent as errors.
posted by yoink at 11:10 AM on July 15 [2 favorites]

You want Wolfgang Puck? I can get you Wolfgang Puck. Believe me, there are ways, Dude. You don't want to know about it, believe me. Hell, I can get you Wolfgang Puck by 3 o'clock this afternoon, with nail polish.
posted by dances_with_sneetches at 11:15 AM on July 15 [16 favorites]

This is why I don't use automatic bill pay.

I don't either, but checks are no guarantee either. Last year US Bank double charged/entered our mortgage payment which was sent in by check, right before we left on vacation, counting on that "extra" money. turns out they did the same thing to hundreds of people that day, and ended up on the line for a lot of overdraft charges.
posted by edgeways at 11:18 AM on July 15

"I thought somebody had bought Europe with my credit card," said Josh Muszynski, from New Hampshire.

How much would Europe cost? There was an NYTimes article a short while ago about how ordinary people are able to estimate very large numbers using basic assumptions. I wonder if he's anywhere close...
posted by Sova at 11:27 AM on July 15 [1 favorite]

JeffK: I suspect that the overdraft charge is applied by a database trigger which fires when the balance is updated. Because the charge did pass the credit limit logic (presumably in a Business Logic Layer somewhere), the bug had to be in the database layer. My geek curiosity is killing me...

But surely the relevant column(s) in the DB don't support numbers that large.
posted by tippiedog at 11:30 AM on July 15

>... they called tech support, who were all able to fix the problem pretty quickly. Near as anyone could figure out, the software just had the hiccups.

Which suggest the problem wasn't fixed at all. Let's hope it doesn't start adding a less obvious number, say, five dollars to each transaction.
posted by ChurchHatesTucker at 11:42 AM on July 15 [1 favorite]

Many database platforms support arbitrary precision math, at least in theory.
posted by Skorgu at 11:45 AM on July 15 [1 favorite]

Wikipedia has the GDP of the EU at $16523780000000000 (I think). 5*GDP seems like a reasonable amount of money to me... looks like he could buy about 1/3rd of Europe.
posted by Leon at 11:47 AM on July 15

At least they waived the overdraft fee. I wouldn't have bet on it. 'Cause I'm a cynical consumer.
posted by Thorzdad at 11:48 AM on July 15 [2 favorites]

That is $23 quadrillion (£14 quadrillion)

That's fourteen thousand, seven hundred billion guineas!
posted by Sys Rq at 11:49 AM on July 15

If one converts 23,148,855,308,184,500 into mills (i.e. 1/1000 dollar, which I believe is often used for actual calculations in financial systems) and represent that as binary, we get:

0b10100000101000001010000010100000101000001010000001011000000000000

Note the repeating 8-bit pattern. It's suggestive of... I'm not sure what exactly, but something.
posted by buxtonbluecat at 11:51 AM on July 15 [11 favorites]

dances_with_sneeches Wolfgang Puck wears nail polish?

buxtonbluecat - Suggestive of... end of the world? Maybe you just found the missing link?!?
posted by gloege at 11:56 AM on July 15

I just can't believe it took nearly 15 minutes and eight posts for someone to turn this into an opportunity to impugn Free Market Economics. If this were a post about, I don't know, hamsters, 15 minutes would probably be acceptable. But this post is actually related to money; it should have taken 3 minutes, tops. This is Metafilter, people, c'mon, pull it together.
posted by blenderfish at 12:15 PM on July 15 [6 favorites]

From the BBC article:

But his overdraft had pushed him into the red - by an amount equivalent to many times the entire US national debt.
"It is a lot of money in the negative," he said. "Something I could never, ever, afford to pay back.

Is it just me, or do journalists these days go out of their way to pick the dumbest-sounding quotes from the people they interview?

Of course, he then goes on with "My children could not afford it, grandchildren, nothing like that."

They'll be able to afford it if the economic crisis triggers hyperinflation!
posted by spitefulcrow at 12:18 PM on July 15

If one converts 23,148,855,308,184,500 into mills (i.e. 1/1000 dollar, which I believe is often used for actual calculations in financial systems) and represent that as binary, we get:

0b10100000101000001010000010100000101000001010000001011000000000000

Note the repeating 8-bit pattern. It's suggestive of... I'm not sure what exactly, but something.

This is going to turn into a tie-in for another bad Nicolas Cage movie, isn't it?
posted by justkevin at 12:18 PM on July 15 [2 favorites]

>Note the repeating 8-bit pattern. It's suggestive of... I'm not sure what exactly, but something.

Ones and zeroes?
posted by ChurchHatesTucker at 12:23 PM on July 15 [1 favorite]

I thought this was just the new tax on cigs to fund health care.
posted by jamstigator at 12:33 PM on July 15

buxtonbluecat- Brilliant! Decoded, that 8 bit string is ASCII for a non-breaking space. Ascii #160.

That's right people. Watch that spacebar carefully. One extra tap could cost you quadrillions!
posted by Saydur at 12:33 PM on July 15 [5 favorites]

I think Frank may be on to something:

In base 16, 2314885530818450000 (with the extra 00 on the end meaning cents) is 2020202020201250. That looks a lot to me as if it’s really meant to be 1250 (for example, meaning $12.50), but somehow with a pile of spaces stuck on the front (since 20 is a common computer code for space).

posted by Bort at 12:33 PM on July 15 [10 favorites]

Metafilter: One extra tap could cost you quadrillions!
posted by stevil at 12:37 PM on July 15

I think a Free Market Disciple would not argue that the software is top-notch or rigorously tested, but rather that rational free market actors optimize according to the following criterion:

if (cost of eliminating error > loss from error * probability of error) then do nothing

In my experience, Free Market Disciples never look at the world with this kind of depth and clarity. Their argument begins and ends with "Competition forces businesses to provide the best, most efficient service possible". They never seem to argue that corporations maximize profits first, and provide only the lowest cost service needed to maximize a profit.
posted by heathkit at 12:39 PM on July 15

Many database platforms support arbitrary precision math, at least in theory.

Precision vs. accuracy
posted by lukemeister at 12:45 PM on July 15

Remember when credit card companies used to call you when you had an anomalous charge on your card? (it wasn't that long ago) I can just imagine the call.......
posted by The Light Fantastic at 12:46 PM on July 15

you guys are seriously overvaluing wolfgang puck. only ditzy moms from glendale go to spago these days.

that said I would kill to find out how exactly this happened and how I could take advantage of it. especially someone like wolfgang puck.

or jamie oliver, the biggest twatwaffle on this planet.
posted by krautland at 12:47 PM on July 15

Saydur: "One extra tap could cost you quadrillions!"

Just ask Eliot Spitzer.
posted by KevinSkomsvold at 12:51 PM on July 15 [9 favorites]

Remember when credit card companies used to call you when you had an anomalous charge on your card? (it wasn't that long ago) I can just imagine the call.......

The bug may have been introduce after the 'fraud detectors'. That is, the code that looks for 'anomalous' charges sees $12.50, and passes it on to the next bit of code, which actually has the error.
posted by delmoi at 12:57 PM on July 15 [1 favorite]

you guys are seriously overvaluing wolfgang puck.

For that kind of money your could probably get him to cook and serve himself.
posted by blue_beetle at 1:12 PM on July 15 [1 favorite]

This bug is fascinating. I love it when controls fail... or are completely missing. It makes me want to pick up a career as an info systems auditor.
posted by Enki at 1:28 PM on July 15

The really not-so-funny part is, in all stories I've read about this, and all of them specifically state that Visa hasn't waved the $20 overdraft fee yet.

Fuckers.
posted by DreamerFi at 1:32 PM on July 15 [1 favorite]

In base 16, 2314885530818450000 (with the extra 00 on the end meaning cents) is 2020202020201250. That looks a lot to me as if it’s really meant to be 1250 (for example, meaning $12.50), but somehow with a pile of spaces stuck on the front (since 20 is a common computer code for space)

The space (0x20) thing seems quite probable, but 0x1250 = 46.88$ decimal.
I guess 0x1250 could be binary coded decimal (as the blog quoted above is implying, perhaps without even realizing it,) but then he would have been charged $202020202020...

More likely is, unlike the cutesy 'bought a pack of cigarettes' claim, he probably actually bought 46.88$ worth of groceries, with tax.
posted by blenderfish at 1:43 PM on July 15 [1 favorite]

Did the vendors get the same amount credited to their accounts, or was it simply on the cardholder's side?
posted by clorox at 1:43 PM on July 15

It seems like every time I go to a gas station, I notice how much the cost of a pack has gone up. But quadrillions?

Man I'm glad I quit back when you were only paying billions. And that was for a carton!
posted by quin at 1:53 PM on July 15

The really not-so-funny part is, in all stories I've read about this, and all of them specifically state that Visa hasn't waved the $20 overdraft fee yet.

I'd write them a check for the $23,148,855,308,184,500 and make them take me to court when it bounced.
posted by Smedleyman at 1:58 PM on July 15 [4 favorites]

It seems quite plausible to me that either the space or the (ISO8859-1/win1250) nbsp has been inserted for padding purposes. This implies, however, that the data would have been extracted from a certainly numeric field in a database table, then converted to string, padded, and written back into the database at some later point in time. It's a weird thing to do.
posted by dhoe at 2:15 PM on July 15

You seem to be under the misapprehension that this is an error.
posted by The same thing we do every night, Pinky at 2:41 PM on July 15

This implies, however, that the data would have been extracted from a certainly numeric field in a database table, then converted to string, padded, and written back into the database at some later point in time. It's a weird thing to do.

Naw, it probably wasn't anything nearly that deliberate. 0x2020202020201250 is 64 bits, so it's likely unintentional pointer aliasing between a 64-bit integer and a padded text buffer. It could be bad memory allocation (perhaps a free() too early?) or failure to clear the buffer after allocating it. One likely scenario, assuming this is C code, is that there was a fixed-size array of chars being used as a string buffer and a 64-bit int sitting next to each other on the stack in some function, and some code wrote spaces past the end string buffer and over top the int. Maybe someone wrote a quick and dirty parser to import price data that handles some kind of text format, and the file for this particular company had a lot of spaces on a line (or maybe in a product description or something) for some reason, breaking the parser. Anyway, there are tons of ways I can imagine this happening.
posted by blenderfish at 2:47 PM on July 15 [4 favorites]

A human-caused overcharge accident at a hot wings place about ten years ago cured me of using debit cards. Our $19 and change lunch got keyed in as $19K. Unfortunately for us, we had the down payment for the house we were about to buy sitting in our savings account. We were able to get the bank to fix it and close the house on schedule, but I don't use my debit card for anything other than the ATM now.

I hear stories about debit card miskeys happening on a small scale all the time, though. Within the last six weeks, my husband had a miskey of $9 at lunch as $90, and another friend had another, similar story for another reasonably small amount. I'm not surprised that Visa has similar problems, either miskeys or just plain glitches.
posted by immlass at 2:56 PM on July 15

23 quadrillion bucks, same as in town.
posted by darkstar at 3:09 PM on July 15 [1 favorite]

This is what happens when you don't read the fine print on your cardholder agreement. Especially the "updated" agreement which they send you every two or three months and you implicitly accept by continuing to use the card.
posted by DevilsAdvocate at 3:26 PM on July 15

It just goes to show, you can't be too careful.
posted by Bonzai at 4:12 PM on July 15 [1 favorite]

All the credit card machines in the world run the Pentium 90 chip.

I am really showing my age here
posted by mr_crash_davis mark II: Jazz Odyssey at 4:12 PM on July 15 [3 favorites]

immlass, do your debit cards not display the total you're about to be charged and force you to OK it? Here in Canuckistan, our nationwide debit system (Interac) has a confirmation step between the keying and the authorization.
posted by Fraxas at 5:33 PM on July 15

I was amused at how Muszynski was also initially charged a $15 overdraft fee.
posted by A dead Quaker at 5:49 PM on July 15

More likely is, unlike the cutesy 'bought a pack of cigarettes' claim, he probably actually bought 46.88$ worth of groceries, with tax.

New Hampshire has no sales tax at all, let alone on groceries. Federal taxes excepted, of course, if applicable.
posted by jock@law at 6:40 PM on July 15

blender, that sounds plausible at first blush -- except that these are all ostensibly Intel machines and the spaces are in the high end of the number. The buffer overflow would have to have written the 0x1250 too. While 0x50 isn't too hard to imagine, 0x12 is pretty damn unlikely to have been in a text field.
posted by jock@law at 6:44 PM on July 15

New Hampshire has no sales tax at all, let alone on groceries. Federal taxes excepted, of course, if applicable.

Interesting. So that lends credibility to $12.50. However, based on the hex representation, I still bet the total bill was $46.88, though, and not $12.50. But who knows. Very minor point.
posted by blenderfish at 6:48 PM on July 15

blender, that sounds plausible at first blush -- except that these are all ostensibly Intel machines and the spaces are in the high end of the number. The buffer overflow would have to have written the 0x1250 too. While 0x50 isn't too hard to imagine, 0x12 is pretty damn unlikely to have been in a text field.

That's an excellent point. The valid bytes being at the beginning would point away from the typical/simple blowing past a buffer case. I guess I was thinking that the bytes would get whacked to 0x20 first, then used in some kind of math op, like added to or something. (Of course, 0x2020202020201250 < 0x2020202020202020, so that's tricky.) I'll think about this, and it could be fun to come up with plausible code with the bug later tonight if I have the time. But, I still would bet it is memory corruption via pointers or overflow of some kind and not a deliberate database op or anything like that.
Of course, I'm operating on nearly no information, so this is all just amusing speculation.
posted by blenderfish at 7:06 PM on July 15

On second thought... since it does really look otherwise like a buffer overflow of a space-padded field... if all the credit card machines run Intel chips... I dunno. It would be interesting to know if Visa's backend runs on big-endian metal. The fact that it happened to numerous Visa customers (no MasterCard customers?) would point that direction too.
posted by jock@law at 7:08 PM on July 15

$46.88? I think $4.688 is more likely.
posted by sfenders at 7:22 PM on July 15

The fact that it happened to numerous Visa customers (no MasterCard customers?) would point that direction too.

I'm guessing it's in-house proprietary software.

$46.88? I think $4.688 is more likely.

Yes, and not saying that's impossible, but then where does the .8 cents come from? My credit card receipts don't charge me fractions of cents ever. Also, it makes it into a more-than-64-bit int, which is less computer-friendly than a 64-bit int, and changes the 0x20, which is ASCII space, into something more esoteric. Also $4.69 seems slightly too cheap for cigarettes with all the crazy taxes, unless you're in Virginia or something (probably something someone could look up on the Internet.)
posted by blenderfish at 7:28 PM on July 15

Maybe $4.688 would be $4.40 plus a 1% fee for the credit card processor. Fractions of cents are very likely used at some point internally, without showing up on the invoice.
posted by sfenders at 7:33 PM on July 15

So weird, I was thinking an overflow error but... it doesn't quite match up for a primitive. I would have expected to see a negative number there, as well.

I imagine we're looking at several bugs, not just one.
posted by Talanvor at 7:36 PM on July 15

Clearly Visa denominates all their internal transactions in Zimbabwe Dollars (ZWD), they just forgot to convert back when printing the bill.
posted by madajb at 9:01 PM on July 15

2314885530818450000 in Hex is 2020202020201200, convert that to ascii characters and you have " 1200", lazy (or stupid) programming.

Warren Buffett carries that much in cash btw.
posted by m@ at 9:19 PM on July 15

Should be six spaces before that 1200, Mefi is censoring me!
posted by m@ at 9:20 PM on July 15

void buyStuff( User *user, Item *item )
{
char strBuf[64];
int totalCost;

totalCost = item->cost; // maybe do more transforms here...

if ( user->owed + totalCost > user->creditLimit )
{
sprintf(strBuf, "Overdraft buying %s", item->name );
DebitUserAccount( user, OVERDRAFT_FEE, strBuf );
totalCost += OVERDRAFT_FEE;
}

sprintf(strBuf, "Purchased %s", item->name );
DebitUserAccount( user, totalCost, strBuf );
}
posted by blenderfish at 12:29 AM on July 16

Err.. you wouldnt charge the overdraft twice, but you get the general idea.
If item->name has too many characters, for example trailing spaces, (lets say some kind of careless import process, or a kitty on the keyboard,) you're hosed.

Anyway, we'll probably never know.
posted by blenderfish at 12:31 AM on July 16

Um... It wouldn't go from 46.88 to 4.688. You're switching the decimal point in the wrong base.
posted by vernondalhart at 12:50 AM on July 16

Alternately, the buffer was filled with 0x20 (plain old ASCII space) and then $35.36 was subtracted from it, with it being a 64-bit integer count of cents, leaving the buffer containing 0x2020202020201250. Still doesn't explain why multiple people are getting exactly the same weird charge, though; why $35.36 (or $46.88)? It looks like those are both plausible prices for a carton of cigarettes in or near NH, but presumably not everyone would be buying an item of the exact same price.

Maybe it's a viral for the next season of Lost.
posted by hattifattener at 1:37 AM on July 16 [1 favorite]

You're switching the decimal point in the wrong base.

I saw no decimal point in the hex, thought where it should go was just a guess based on assuming the unit is cents. More likely they'd use integer tenths of a cent; either way, the decimal point gets added only after you convert to decimal base.

1.69% plus $0.20 appears to be about what they commonly charge. That shouldn't get charged to the customer, but it could be part of the amount that somehow over-wrote what was meant to be billed.
posted by sfenders at 5:00 AM on July 16

They may also have been reading text from an input field buffer, using it to dynamically assemble a SQL INSERT statement, and letting the database handle the string-to-numeric casting. I sit in database code reviews ALL THE GODDAMN TIME with (ostensibly) professional DB developers that don't have a handle on the fact that

string SQL = 'insert into cc_transactions (id, dollar_amount) values (' + $ID + ', \'' + $TOTAL_AMOUNT + '\')';

is a wholly different creature than

string SQL = 'insert into cc_transactions (id, dollar_amount) values (' + $ID + ', ' + $TOTAL_AMOUNT + ')';

"Well, the package doesn't have any errors, and the three tests we made of it with the numeric input '5' inserted the right values into the table, so we must be able to use the two interchangeably." That's fucking FANTASTIC, guys. It compiles, ship it!

All it would take is a $TOTAL_AMOUNT variable that was read from a fixed-width string input that front-padded with spaces to make right-alignment work in the UI, and a really stupid backend string-to-int casting implementation, and bang, 23 quadrillion dollar transaction.
posted by Mayor West at 6:00 AM on July 16

looks like Stack Overflow comes through with the win for why this happened
posted by Mach5 at 8:08 AM on July 16 [1 favorite]

fraxas, the server handled the card input for our $19K accident. That's how a lot of restaurants in the US still operate; I'm always surprised when I get to swipe my own card at a restaurant. I'm sure she had to approve the amount, but she did it without thinking because it's part of her routine. I'm also sure that if it hadn't been my house down payment, the look on her face as she realized what she'd done would have been hilarious. Oops!
posted by immlass at 8:49 AM on July 16

With that much money I could clone Wolfgang Puck and cook him as a
PuckPuckPuckTurDuckEn.
posted by lothar at 9:48 AM on July 16

As I (essentially) said above, the Stack Overflow answer only works if the string buffer overflow happened on a big-endian architecture. Since the vast majority of computers in use today are little-endian, I think there's a slightly higher burden of proof before just saying "yeah, that's it."

Besides, multiple Visa customers are getting the exact same charge, which means that the 0x1250 must be caused by the same bug -- and 0x1250 is emphatically not the kind of thing you find in a text field.

I think Talanvor was right in that this might be multiple bugs, OR it's a text buffer overflow on a platform that uses 0x1250 as a string terminator, OR there is a pointer error.

The idea that this is the first we've heard of it, and at approximately the same time this happened to "less than 13,000" people -- combined with the improbability of it being a text buffer overflow -- leads me to believe that this is a pointer error somehow implicating the date or time.
posted by jock@law at 11:01 AM on July 16

jock@law: No, it works regardless of endianity, if you assume that the whole buffer was overwritten and then followed up by some numeric operation. And really, there are plenty of big-endian machines in the world; not everyone has succumbed to the Intel architecture.

The only mystery to my mind is what that secondary numeric operation might have been.

One thought is that a 64-bit int was overwritten by 0x20, and then the number was cast to a floating point format that truncated the last few bits of the matissa. IEEE-754 double precision doesn't quite work (it'd have produced a magic value of $23148855308184535.04). IEEE-754 decimal floating point would come awfully close to producing the right wrong answer, though. Maybe if there's a hidden rounding bit somewhere?

In which case the scenario could be:

1. 64-bit integer buffer containing cents is overwritten with ASCII spaces, producing $23148855308184535.36. (bug 1)
   
2. Number is stored or transmitted in IEEE-754-decimal format (bug 2; never use floating point for financial transactions!) and truncated.

Yeah, decimal floating point is a pretty obscure format, but big financial applications are the reason it exists, as far as I know.
posted by hattifattener at 11:46 AM on July 16 [1 favorite]

At least they waived the overdraft fee. I wouldn't have bet on it. 'Cause I'm a cynical consumer.

Yeah, I've been hit by the "Oh! You have an overdraft fee! We'll just take take a little money out for that... Oh! You have another overdraft fee because we just took out some money to cover your first overdraft! That's another overdraft! We'll just take a little..." process.

Here, I'll kick it old school for ya'll:
150 IF USERACCT <= 0 THEN USERACCT = USERACCT - 35 : GOTO 150
posted by Avelwood at 5:21 PM on July 16 [3 favorites]

Good thinking, hatti.

Looking at the FPP, the picture doesn't actually _show_ the cents (they're cropped.) They could have rounded it off in the text. So it could be "...500.69" or something.
posted by blenderfish at 5:54 PM on July 16

the software just had the hiccups.

That is a particularly splendid piece of bullshit. I must remember to use it.
posted by A Terrible Llama at 1:05 AM on July 17

Looking at the FPP, the picture doesn't actually _show_ the cents (they're cropped.) They could have rounded it off in the text. So it could be "...500.69" or something.

Actually, the MSNBC article DOES show the cents as 00.

Good discussion, CS-folk!
posted by disillusioned at 3:29 AM on July 17

I have a former co-worker who works for a company that does huge amounts of data processing for customers like Visa. I should ask him if it's all his fault.
posted by mkb at 6:55 AM on July 17

BCD isn't an uncommon format for storing numeric data in financial transaction messages. It's used in the ISO8583 standard for a few payment processors. The thing is, if the field was stored as BCD, it would be padded with 0s 0x00, not spaces 0x20. So I agree with that frank dude. The thing is, you would expect to see crap all the time in that case. (Why would the software pad with spaces sometimes, and not others.)
posted by chunking express at 7:43 AM on July 17

Look cobber, the High-Optional, Logical, Multi-Evaluating Supervisor, Mark IV, Mod. L. can get fouled up if warden's goons let flies into it.
(Free Luna)
posted by Smedleyman at 10:52 AM on July 17