The sky above the port was the color of a NAND IC... turned out with dead channels.
February 16, 2010 10:28 PM Subscribe
Ghost shift ghost chips. A tale about a Chumby hardware developer with a keen investigative eye noticing some oddities about microSD FLASH cards from supposedly reputable suppliers.
Reminds me of a lecture I attended last summer; an archaeologist spoke at great length about the various types of clays and firing methods practiced by different regional potteries in 19th Century China. He was analyzing the shards unearthed from the site of an early Chinese fishing village in California to date its founding, expansion, and extinction. I see the same amazing attention to detail regarding the mundane and ephemeral, and how much is revealed by this attention.
posted by squalor at 10:55 PM on February 16, 2010 [5 favorites]
posted by squalor at 10:55 PM on February 16, 2010 [5 favorites]
Checking out some of his other other blog posts I came across This video of industrial robots making other industrial robots which is both awesome and kind of freaky.
posted by delmoi at 10:55 PM on February 16, 2010 [2 favorites]
posted by delmoi at 10:55 PM on February 16, 2010 [2 favorites]
Fascinating read, even to this non-techy person.
posted by Alvy Ampersand at 10:59 PM on February 16, 2010
posted by Alvy Ampersand at 10:59 PM on February 16, 2010
Oh god this is amazing. This kind of detail oriented street vs corporate distribution culture/manufacturing is why I read William Gibson books. Tonight loquacious, my brain thanks you.
posted by saturnine at 11:03 PM on February 16, 2010 [3 favorites]
posted by saturnine at 11:03 PM on February 16, 2010 [3 favorites]
Could somebody please summarize this?
posted by Afroblanco at 11:08 PM on February 16, 2010
posted by Afroblanco at 11:08 PM on February 16, 2010
"Significantly, Kingston is revealed as simply a vendor that re-marks other people’s chips in its own packaging."
When I saw this elsewhere earlier, my first thought was "yeah, so what's new?". Kingston doesn't own a fab, never has, so they've always done that - and their early reputation for consistently high quality hasn't been particularly true for several years. I suspect they've been squeezed into taking lower-priced (& lower/variable quality) parts for a while.
posted by Pinback at 11:11 PM on February 16, 2010 [2 favorites]
When I saw this elsewhere earlier, my first thought was "yeah, so what's new?". Kingston doesn't own a fab, never has, so they've always done that - and their early reputation for consistently high quality hasn't been particularly true for several years. I suspect they've been squeezed into taking lower-priced (& lower/variable quality) parts for a while.
posted by Pinback at 11:11 PM on February 16, 2010 [2 favorites]
As a result, Kingston plays a significant and important role in stabilizing microSD card prices and improving fab margins, but at some risk to their own brand image.
I always thought it was well known that Kingston repackaged memory from other vendors. Still, this was a very interesting read. Cheers.
posted by Blazecock Pileon at 11:16 PM on February 16, 2010
I always thought it was well known that Kingston repackaged memory from other vendors. Still, this was a very interesting read. Cheers.
posted by Blazecock Pileon at 11:16 PM on February 16, 2010
Could somebody please summarize this?
-- Afroblanco
Some guy's company had some defective SD flash cards that turned out to probably be counterfeit. So he went out to the street and bought some clearly counterfeit cards and opened them up to take pictures of the counterfeit chips inside.
His conclusion: buy the cards from the original manufacturer.
posted by eye of newt at 11:17 PM on February 16, 2010 [1 favorite]
-- Afroblanco
Some guy's company had some defective SD flash cards that turned out to probably be counterfeit. So he went out to the street and bought some clearly counterfeit cards and opened them up to take pictures of the counterfeit chips inside.
His conclusion: buy the cards from the original manufacturer.
posted by eye of newt at 11:17 PM on February 16, 2010 [1 favorite]
actually, eye of newt, Kingston themselves said the questionable MicroSD cards, bought from an authorized distributor, were authentic. So the conclusion is that Kingston is happy to put their name on junk SD cards.
posted by zsazsa at 11:45 PM on February 16, 2010 [3 favorites]
posted by zsazsa at 11:45 PM on February 16, 2010 [3 favorites]
Not only that, but they were happy with selling one of their customers junk and telling them to go fuck themselves until he went the extra mile to prove that they were selling him junk.
posted by empath at 11:51 PM on February 16, 2010 [11 favorites]
posted by empath at 11:51 PM on February 16, 2010 [11 favorites]
This is an awesome blog. I loved the Gibsonian feeling of the grey market stall with the chain smoking texting girl selling fake cards and the mom and pop business with the fake holograms and packaging (Did any cyberpunk author get it right with respect to China/Japan?), the second robot video (robots cutting steel plates thicker than a door like it was warm butter?), the very interesting tidbit on the little metal squares added to a chip's design so that the grinders can grind evenly, the economic analysis on why consumer SD cards include a memory controller for free (cheaper than the depreciation of a chip tester testing each memory chip). I just won 2 hours of my life reading this site.
I just checked four SD cards I own. The three expensive ones, bought in reputable San Francisco shops, all have the 00/2000 date that the fake/defective cards in the article have. The super cheap one I got shipped from China with my Nintendo DS hacking cartridge is the real deal.
I am re-reading Virtual Light, and tonight when I ride my bike under the Bay Bridge I will be thinking of this article.
posted by dirty lies at 12:22 AM on February 17, 2010 [3 favorites]
I just checked four SD cards I own. The three expensive ones, bought in reputable San Francisco shops, all have the 00/2000 date that the fake/defective cards in the article have. The super cheap one I got shipped from China with my Nintendo DS hacking cartridge is the real deal.
I am re-reading Virtual Light, and tonight when I ride my bike under the Bay Bridge I will be thinking of this article.
posted by dirty lies at 12:22 AM on February 17, 2010 [3 favorites]
"Ghost shifts" are a serious problem for most companies outsourcing manufacturing to China. You order x widgets, they make y, deliver you the ordered amount and sell the y-x difference on the grey market. It may seem pretty shameless, but it's a problem that the outsourcers themselves have brought upon themselves, by putting so much pressure on their suppliers that they can't turn a profit (low Chinese wages and undervalued yuan notwithstanding) without cheating. And God, there's an awful lot of cheating going on there...
What's unusual is that the ghost shifting Chinese manufacturer ends up selling those extra numbers to the outsourcer himself, as appears to have been the case here. It suggests serious economic hardship on both sides.
Many "manufacturers" these days are little more than glorified quality inspectors. They put their brand on stuff made in faceless Chinese factories and are supposed to check that the stuff meets their quality requirements. Nowadays, they don't even bother to pretend that they give any significant R&D input. What this guy has discovered about Kingston suggests that they've even lost control on their supply chain. Coupled with what he says about the economics of memory cards, it's a sign that Kingston is circling the drain.
posted by Skeptic at 1:11 AM on February 17, 2010 [15 favorites]
What's unusual is that the ghost shifting Chinese manufacturer ends up selling those extra numbers to the outsourcer himself, as appears to have been the case here. It suggests serious economic hardship on both sides.
Many "manufacturers" these days are little more than glorified quality inspectors. They put their brand on stuff made in faceless Chinese factories and are supposed to check that the stuff meets their quality requirements. Nowadays, they don't even bother to pretend that they give any significant R&D input. What this guy has discovered about Kingston suggests that they've even lost control on their supply chain. Coupled with what he says about the economics of memory cards, it's a sign that Kingston is circling the drain.
posted by Skeptic at 1:11 AM on February 17, 2010 [15 favorites]
His conclusion: buy the cards from the original manufacturer.
No, his conclusion is that the original manufacturer had effectively done what he did, and walked around town to buy up chips from other manufacturers. Some of these chips are of lower quality than we might expect from a popular retailer such as this.
posted by Sutekh at 1:23 AM on February 17, 2010
No, his conclusion is that the original manufacturer had effectively done what he did, and walked around town to buy up chips from other manufacturers. Some of these chips are of lower quality than we might expect from a popular retailer such as this.
posted by Sutekh at 1:23 AM on February 17, 2010
Kingston have always seemed fishy to me: in the nineties they permanently booked-out the back cover of several UK tech magazines to tell you how special their RAM was compared to lesser brands. If you went to Macworld UK you would be waylaid at the door by reps spouting the same message. That kind of strategic differentiation in the consumer marketplace doesn't come cheap, and it's no surprise to see that its hollowed them out over time.
Love the article. Marx meets McGuyver.
posted by GeorgeBickham at 1:37 AM on February 17, 2010
Love the article. Marx meets McGuyver.
posted by GeorgeBickham at 1:37 AM on February 17, 2010
This is fascinating and it also great advertising. It puts a genuine human face on the manufacturing of the Chumby and shows that they care. Brilliant engagement with potential customers.
posted by srboisvert at 1:37 AM on February 17, 2010 [2 favorites]
posted by srboisvert at 1:37 AM on February 17, 2010 [2 favorites]
Bunnie writes some great stuff.
If you are interested in his blog post, I can recommend his book, "Hacking the Xbox". It's the story of a tenacious underdog wrangling electronic archaeology.
posted by Sutekh at 1:39 AM on February 17, 2010 [1 favorite]
If you are interested in his blog post, I can recommend his book, "Hacking the Xbox". It's the story of a tenacious underdog wrangling electronic archaeology.
posted by Sutekh at 1:39 AM on February 17, 2010 [1 favorite]
I know the pain of counterfeit components.
Let me tell you a story :
For a recent sculpture I commissioned an electronics company to manufacture around 500 circuit boards for me. Two weeks before I was due to install the piece they called me, frantic, saying they’d got the boards back from the assembly company but every single board had problems, and in four days of 24 hour-a-day panic they’d only managed to get five boards working properly. After a lot of investigation we ended up X-raying the LED driver chips on the boards and found that they were complete rubbish - *really* bad. We had a look at a sample of about 100 chips - many of them had broken connections (bond wires) inside. Some of the chips didn’t actually have any bond wires at all. Several of them actually had the die inside MOUNTED UPSIDE DOWN.
The theoretical manufacturer of the chips says it’s impossible for their products to be so bad, and that they have no record of the serial numbers on the chips etc. so they must be counterfeit. They look exactly like the real thing though, and even have the manufacturers logos on the dies inside. Even if you X-rayed a couple of chips from the batch it’s impossible to tell they’re not real - apart from the fact that a large proportion of them don’t work. The best theory that anyone has come up with is that these must be reject chips smuggled out of the factory by an employee and passed on to another company that has packaged them with fake markings and seals.
The result is that I’m completely fucked. There are about 50 of these chips on each board, which makes it more or less impossible to repair the boards, meaning they will have to be rebuilt from scratch. This is going to cost about £120,000. No-one in the supply chain will accept responsibility - the broker who sold the chips is a tiny company who will probably go into liquidation rather than pay the cost of rebuilding my boards. Their supplier is a distributor in China, and it seems very unlikely that we’ll ever get any compensation from them.
If anyone on Metafilter has any experience at all of dealing with these kinds of problems please get in touch with me…. I’m desperate for any good legal or practical advice on what to do.
posted by silence at 1:47 AM on February 17, 2010 [28 favorites]
Let me tell you a story :
For a recent sculpture I commissioned an electronics company to manufacture around 500 circuit boards for me. Two weeks before I was due to install the piece they called me, frantic, saying they’d got the boards back from the assembly company but every single board had problems, and in four days of 24 hour-a-day panic they’d only managed to get five boards working properly. After a lot of investigation we ended up X-raying the LED driver chips on the boards and found that they were complete rubbish - *really* bad. We had a look at a sample of about 100 chips - many of them had broken connections (bond wires) inside. Some of the chips didn’t actually have any bond wires at all. Several of them actually had the die inside MOUNTED UPSIDE DOWN.
The theoretical manufacturer of the chips says it’s impossible for their products to be so bad, and that they have no record of the serial numbers on the chips etc. so they must be counterfeit. They look exactly like the real thing though, and even have the manufacturers logos on the dies inside. Even if you X-rayed a couple of chips from the batch it’s impossible to tell they’re not real - apart from the fact that a large proportion of them don’t work. The best theory that anyone has come up with is that these must be reject chips smuggled out of the factory by an employee and passed on to another company that has packaged them with fake markings and seals.
The result is that I’m completely fucked. There are about 50 of these chips on each board, which makes it more or less impossible to repair the boards, meaning they will have to be rebuilt from scratch. This is going to cost about £120,000. No-one in the supply chain will accept responsibility - the broker who sold the chips is a tiny company who will probably go into liquidation rather than pay the cost of rebuilding my boards. Their supplier is a distributor in China, and it seems very unlikely that we’ll ever get any compensation from them.
If anyone on Metafilter has any experience at all of dealing with these kinds of problems please get in touch with me…. I’m desperate for any good legal or practical advice on what to do.
posted by silence at 1:47 AM on February 17, 2010 [28 favorites]
Ouch silence. I wish I knew about your specific problem. This is definitely worth putting on Ask Metafilter. Is the electronics company you commissioned to manufacture the boards not responsible for testing the components or at least a prototype board before assembly? Does the broker not carry liability insurance?
Maybe an ask mefi with the design/function of your boards and what usable components you have and try and get ideas on reducing costs? (I'm sure you've been through this already, but I thought I'd mention it)
posted by BrotherCaine at 3:37 AM on February 17, 2010
Maybe an ask mefi with the design/function of your boards and what usable components you have and try and get ideas on reducing costs? (I'm sure you've been through this already, but I thought I'd mention it)
posted by BrotherCaine at 3:37 AM on February 17, 2010
As a note, this isn't *a* Chumby developer. This is *the* Chumby developer. This is Bunnie Huang, the same guy who broke the original X-Box. He's quite the legend, and it's great to see him keeping up the insanity.
posted by effugas at 3:41 AM on February 17, 2010 [6 favorites]
posted by effugas at 3:41 AM on February 17, 2010 [6 favorites]
The gray market-
SEG Plaza
and linked off the main post.
Ridiculous
posted by pantsrobot at 3:52 AM on February 17, 2010 [2 favorites]
SEG Plaza
and linked off the main post.
Ridiculous
posted by pantsrobot at 3:52 AM on February 17, 2010 [2 favorites]
I love this stuff! Thanks for posting!
posted by zerobyproxy at 4:07 AM on February 17, 2010
posted by zerobyproxy at 4:07 AM on February 17, 2010
@BrotherCaine - thanks for the tip. I’ll try to get an organised post together for AskMefi, but I’m travelling at the moment and things are hectic.
I avoided posting too many details in my story, because I don’t want to cause any more problems than I already have by mentioning manufacturers etc. by name. In answer to your questions -
• yes, the company I commissioned did test prototypes before building them, but when they made the prototype they only needed a small quantity of parts, so they ordered them directly from Farnell or some other big retail distributor like that. The prototype worked fine. When they did the actual manufacturing they sent a bill of materials out to tender by different suppliers. They specified the same components on the BOM as in the prototype, and, reasonably enough, expected that that was what they’d get.
• The counterfeits look *really* genuine - no one could have spotted them by just looking.Even if you had x-rayed or de-encapsulated a couple of them you still wouldn’t have been able to tell they weren’t legitimate. They look very much like genuine parts, just with absolutely no quality control - that’s what leads me to think that these were probably made on the same assembly line as the genuine parts, but were either rejects that got recycled or else “ghost shift” parts.
• No, the broker claims that this isn’t covered by their insurance - but they seem weirdly reticent to push their insurance company.
• Short of a re-design of the whole thing I’m not sure there’s much that can be done to reduce costs. Each board is basically a module of a large LED display system - every board has around 250 LEDs on it, along with drivers, microcontroller and networking. The vast majority of the cost is the LEDs and drivers. The dodgy components are the drivers - there are quite a few of these per board, and because of the nature of the faults diagnosing which are bad is difficult. Besides which, looking at the x-rays it seems probable that even chips that are functional now will be quite likely to fail in the near future (many have very dubious looking bond wires etc). Desoldering all the chips of the board and replacing them with new chips would be incredibly labour intensive (wouldn’t be much cheaper than re-building the whole thing), and would almost certainly compromise the reliability of everything else on the board.
thanks for the empathy !
posted by silence at 4:45 AM on February 17, 2010
I avoided posting too many details in my story, because I don’t want to cause any more problems than I already have by mentioning manufacturers etc. by name. In answer to your questions -
• yes, the company I commissioned did test prototypes before building them, but when they made the prototype they only needed a small quantity of parts, so they ordered them directly from Farnell or some other big retail distributor like that. The prototype worked fine. When they did the actual manufacturing they sent a bill of materials out to tender by different suppliers. They specified the same components on the BOM as in the prototype, and, reasonably enough, expected that that was what they’d get.
• The counterfeits look *really* genuine - no one could have spotted them by just looking.Even if you had x-rayed or de-encapsulated a couple of them you still wouldn’t have been able to tell they weren’t legitimate. They look very much like genuine parts, just with absolutely no quality control - that’s what leads me to think that these were probably made on the same assembly line as the genuine parts, but were either rejects that got recycled or else “ghost shift” parts.
• No, the broker claims that this isn’t covered by their insurance - but they seem weirdly reticent to push their insurance company.
• Short of a re-design of the whole thing I’m not sure there’s much that can be done to reduce costs. Each board is basically a module of a large LED display system - every board has around 250 LEDs on it, along with drivers, microcontroller and networking. The vast majority of the cost is the LEDs and drivers. The dodgy components are the drivers - there are quite a few of these per board, and because of the nature of the faults diagnosing which are bad is difficult. Besides which, looking at the x-rays it seems probable that even chips that are functional now will be quite likely to fail in the near future (many have very dubious looking bond wires etc). Desoldering all the chips of the board and replacing them with new chips would be incredibly labour intensive (wouldn’t be much cheaper than re-building the whole thing), and would almost certainly compromise the reliability of everything else on the board.
thanks for the empathy !
posted by silence at 4:45 AM on February 17, 2010
silence Where is the assembly of your boards physically located? If the chips were imported into the European Union by the broker, this would be at the very least trademark infringement (since the purported manufacturer's logo is in there), and he could be not just commercially liable, but even criminally so, if he had an inkling that he was selling counterfeit goods.
If the boards were already assembled in Asia, then the situation is definitely murkier. I advice you get in touch with a good intellectual property lawyer, and also with the purported chip manufacturer: after all, they should have the most interest in cracking down on counterfeiters (especially if, as it seems here, they have leaks in their supply chain).
posted by Skeptic at 5:09 AM on February 17, 2010
If the boards were already assembled in Asia, then the situation is definitely murkier. I advice you get in touch with a good intellectual property lawyer, and also with the purported chip manufacturer: after all, they should have the most interest in cracking down on counterfeiters (especially if, as it seems here, they have leaks in their supply chain).
posted by Skeptic at 5:09 AM on February 17, 2010
@Skeptic - they were assembled in the UK and the components were imported to the UK. I think we’ve reported the distributor to the Trading Standards authority now, but it’s difficult to prove that they would have known they were counterfeit.
The broker claims they aren’t liable for consequential damage, and only liable to replace the faulty parts. I’m no lawyer, but I’m pretty sure this must be bullshit if the parts aren’t just faulty, but aren’t actually what you ordered (...though perhaps that’s a grey area if they actually are rejects from the same factory???)
If we did get some kind of legal ruling that they were liable I’m fairly sure they would simply go out of business and we would still be in the same situation, except with large legal bills on top.
We’ve reported the issue to the chip manufacturer and they seem to have a counterfeit department that are looking into it, but I’m pretty sure they aren’t going to admit to any liability themselves (though it seems it might be a security breach on their production line that allowed these things into the wild). I guess it’s possible that they might be capable of going after the chinese distributor directly. Suing a company in China for IP breaches doesn’t sound like much fun, or very likely to succeed, but it’s pretty much the only hope we’ve got as far as I can tell.
posted by silence at 5:43 AM on February 17, 2010
The broker claims they aren’t liable for consequential damage, and only liable to replace the faulty parts. I’m no lawyer, but I’m pretty sure this must be bullshit if the parts aren’t just faulty, but aren’t actually what you ordered (...though perhaps that’s a grey area if they actually are rejects from the same factory???)
If we did get some kind of legal ruling that they were liable I’m fairly sure they would simply go out of business and we would still be in the same situation, except with large legal bills on top.
We’ve reported the issue to the chip manufacturer and they seem to have a counterfeit department that are looking into it, but I’m pretty sure they aren’t going to admit to any liability themselves (though it seems it might be a security breach on their production line that allowed these things into the wild). I guess it’s possible that they might be capable of going after the chinese distributor directly. Suing a company in China for IP breaches doesn’t sound like much fun, or very likely to succeed, but it’s pretty much the only hope we’ve got as far as I can tell.
posted by silence at 5:43 AM on February 17, 2010
I'd never have expected this subject to be so compelling. Thanks for a great read!
posted by Mister_A at 5:46 AM on February 17, 2010
posted by Mister_A at 5:46 AM on February 17, 2010
silence Whether the chips were from the same factory or not is immaterial if they were never sold with the consent of the trademark owner. IANAL, but I guess you could put a lot more pressure on the broker if he's made aware that he may not just lose some money, but also look forward to potential criminal prosecution unless he at least takes the chips back and destroys them.
The reason for contacting the trademark owner isn't to get compensation from them (good luck with that), but to get together to crush the Chinese distributor's balls. Enforcing IP in China is indeed no picnic, but the Chinese authorities are starting to get serious, and they certainly also have an interest in ensuring that IP owners (Chinese and foreign) can trust their supply chains in China.
posted by Skeptic at 6:03 AM on February 17, 2010
The reason for contacting the trademark owner isn't to get compensation from them (good luck with that), but to get together to crush the Chinese distributor's balls. Enforcing IP in China is indeed no picnic, but the Chinese authorities are starting to get serious, and they certainly also have an interest in ensuring that IP owners (Chinese and foreign) can trust their supply chains in China.
posted by Skeptic at 6:03 AM on February 17, 2010
Silence: I'd email the guy that wrote the blog entry, it sounds like something he might be interested in.
posted by empath at 6:04 AM on February 17, 2010
posted by empath at 6:04 AM on February 17, 2010
Silence: This is where you talk to a lawyer. You've got what could be a pretty good contract claim here, but you need someone who will 1) represent YOUR interests, and not just cover their ass 2)determine what, if any, actual claims you have.
posted by leotrotsky at 6:08 AM on February 17, 2010
posted by leotrotsky at 6:08 AM on February 17, 2010
Thanks Skeptic - yes, I’ve been thinking that getting the manufacturer to help us put pressure on the chinese distributors was our best hope, it’s good to have someone else’s opinion back that up - I’ve been pretty lonely with this problem, because I really don’t know anything about the law involved etc.
Good idea empath - I’ll give it a go.
posted by silence at 6:10 AM on February 17, 2010
Good idea empath - I’ll give it a go.
posted by silence at 6:10 AM on February 17, 2010
Hey, Silence, I'm sure you've thought this through pretty carefully but it seems that since you hired a mfr to make and test boards, that if the boards don't work then the mfr is on the hook for that, not you. Did you buy the parts, or did you give them a BOM and tell them to buy the parts? I have no idea how contracts in this realm actually read, but from my naive consumer point of view if one buys an X, and the X doesn't work, then that's not your problem aside from consequential damage.
So, yeah. Lawyer up. Especially if a hundred thousand pounds is on the line.
posted by seanmpuckett at 6:15 AM on February 17, 2010
So, yeah. Lawyer up. Especially if a hundred thousand pounds is on the line.
posted by seanmpuckett at 6:15 AM on February 17, 2010
What a great article. Thanks for posting it, I would have never seen it otherwise.
posted by gemmy at 6:43 AM on February 17, 2010
posted by gemmy at 6:43 AM on February 17, 2010
Did someone ever think Kingston made memory? Because as far as I know they've been nothing but a rebadging and marketing shop for 20 years. On top of that because it's a recognizable name and their branding largely consists of applying some cheap silkscreens to existing products of course there are going to be Chinese knockoffs on the grey market all over the place.
When you go to DealExtreme or whatever, and you pay a third of the retail price for the "Chinese version" of a product, did you really think you were certain to get the genuine article?
When you buy a Kingston product, did you think little elves with a K on their hat were somehow conjuring the products from the ether?
I appreciate that guy here decided to do some really awesomely detailed analysis on the matter, but the other way to prove the assertion that the Kingston name doesn't mean a goddamn thing would be to, I dunno, not be totally frickin blind?
posted by majick at 6:45 AM on February 17, 2010
When you go to DealExtreme or whatever, and you pay a third of the retail price for the "Chinese version" of a product, did you really think you were certain to get the genuine article?
When you buy a Kingston product, did you think little elves with a K on their hat were somehow conjuring the products from the ether?
I appreciate that guy here decided to do some really awesomely detailed analysis on the matter, but the other way to prove the assertion that the Kingston name doesn't mean a goddamn thing would be to, I dunno, not be totally frickin blind?
posted by majick at 6:45 AM on February 17, 2010
When you buy a Kingston product, did you think little elves with a K on their hat were somehow conjuring the products from the ether?
Could you be any more condescending?
How about this:
When you buy a Sony product, did you think little elves with a S on their hat were somehow conjuring the products from the ether?
When you buy an Apple product, did you think little elves with a A on their hat were somehow conjuring the products from the ether?
When you buy a Nike product, did you think little elves with a N on their hat were somehow conjuring the products from the ether?
Do you think everyone on the planet is born with a little detector in their head that tells them whether products are manufactured by the supplier or are rebadged? Cuz that sure as hell didn't come standard with my model, and I never actually wondered whether Kingston manufactured their own components or not.
But see, I was never a blithering idiot that somehow thinks that everything came from magical little elves. No, see, at some point the components are manufactured and assembled. No, you don't need to assume I'm stupid enough to think that elves are doing everything. But hey, what exactly is it that tells me the memory labeled "Kingston" is not actually made by Kingston, but the memory made labeled "Sandisk" is made by Sandisk? It sure as heck isn't the absence of magical elves.
But hey, you knew about it, so OBVIOUSLY everyone who didn't must be totally blind and dumb as a stump besides, right?
Jesus fucking christ.
posted by splice at 7:18 AM on February 17, 2010 [22 favorites]
Could you be any more condescending?
How about this:
When you buy a Sony product, did you think little elves with a S on their hat were somehow conjuring the products from the ether?
When you buy an Apple product, did you think little elves with a A on their hat were somehow conjuring the products from the ether?
When you buy a Nike product, did you think little elves with a N on their hat were somehow conjuring the products from the ether?
Do you think everyone on the planet is born with a little detector in their head that tells them whether products are manufactured by the supplier or are rebadged? Cuz that sure as hell didn't come standard with my model, and I never actually wondered whether Kingston manufactured their own components or not.
But see, I was never a blithering idiot that somehow thinks that everything came from magical little elves. No, see, at some point the components are manufactured and assembled. No, you don't need to assume I'm stupid enough to think that elves are doing everything. But hey, what exactly is it that tells me the memory labeled "Kingston" is not actually made by Kingston, but the memory made labeled "Sandisk" is made by Sandisk? It sure as heck isn't the absence of magical elves.
But hey, you knew about it, so OBVIOUSLY everyone who didn't must be totally blind and dumb as a stump besides, right?
Jesus fucking christ.
posted by splice at 7:18 AM on February 17, 2010 [22 favorites]
Agreed - very interesting article, and a bit ominous. Like the "rot" in Wall St as evidenced by bad consumer debt sold as AAA paper, this confirms that the legit margins are disappearing from outsourced manufacturing, and rejects, ghost-shift parts and counterfeits are being allowed into what are supposed to be reputable name-brand products.
You don't have to be an industry analyst to notice that most of our products are made offshore, and that most branded articles (eg consumer electronics, power tools) have been dropping in price... and quality.
The banks hate to go public in cases of charge-card or bank fraud, they usually prefer to quietly reimburse the victim, to preserve the bank's reputation . Likewise, I expect the big brands (eg Kingston) would prefer to exchange any defective parts with as little fuss as possible, without publicly admitting that substandard product had carried their brand.
So, Silence, I expect that you (with a little help from a lawyer) simply need to promise to the jobber and to the IC maker that you are capable of making an extremely loud public noise about this, and you may find that your boards are remanufactured or replaced at no cost to you.
posted by Artful Codger at 7:23 AM on February 17, 2010 [2 favorites]
You don't have to be an industry analyst to notice that most of our products are made offshore, and that most branded articles (eg consumer electronics, power tools) have been dropping in price... and quality.
The banks hate to go public in cases of charge-card or bank fraud, they usually prefer to quietly reimburse the victim, to preserve the bank's reputation . Likewise, I expect the big brands (eg Kingston) would prefer to exchange any defective parts with as little fuss as possible, without publicly admitting that substandard product had carried their brand.
So, Silence, I expect that you (with a little help from a lawyer) simply need to promise to the jobber and to the IC maker that you are capable of making an extremely loud public noise about this, and you may find that your boards are remanufactured or replaced at no cost to you.
posted by Artful Codger at 7:23 AM on February 17, 2010 [2 favorites]
When you go to DealExtreme or whatever, and you pay a third of the retail price for the "Chinese version" of a product, did you really think you were certain to get the genuine article?
Conversely, can you be sure that the heavily marked-up version of the product found in high-street shops is going to be to a higher standard than a version you buy on eBay from Hong Kong for 1/3 of the price?
posted by acb at 7:55 AM on February 17, 2010 [1 favorite]
Conversely, can you be sure that the heavily marked-up version of the product found in high-street shops is going to be to a higher standard than a version you buy on eBay from Hong Kong for 1/3 of the price?
posted by acb at 7:55 AM on February 17, 2010 [1 favorite]
Here's the thing about Bunnie. There's a lot of us in the software world that are capable of doing minor hacks, reverse engineering, breaking a little copy protection. Thousands of engineers out there who've worn a grey hat on occasion. But if we hit the big impenetrable wall of hardware, we're stuck. Magic chip locking you out of running your own code? Special firmware you can't dump? Duh, I'm stuck.
Not Bunnie. He just rolled up his sleeves, started pulling the chips apart, and showed us how it works. Got a funky Flash chip? No problem, just dissolve it in nitric acid and figure out what's going on! When the Xbox came out it was one of the first consumer products that used sophisticated hardware to lock hackers out. And Bunnie just stripped it apart and showed us how he did it. Thanks!
posted by Nelson at 7:58 AM on February 17, 2010 [4 favorites]
Not Bunnie. He just rolled up his sleeves, started pulling the chips apart, and showed us how it works. Got a funky Flash chip? No problem, just dissolve it in nitric acid and figure out what's going on! When the Xbox came out it was one of the first consumer products that used sophisticated hardware to lock hackers out. And Bunnie just stripped it apart and showed us how he did it. Thanks!
posted by Nelson at 7:58 AM on February 17, 2010 [4 favorites]
When you buy a Kingston product, did you think little elves with a K on their hat were somehow conjuring the products from the ether?
I think you're mocking bunnie. You should not mock bunnie. You know that ultra-nerdy saying about turning you into a small shell script? Bunnie can actually do that!
posted by dirigibleman at 8:54 AM on February 17, 2010 [2 favorites]
I think you're mocking bunnie. You should not mock bunnie. You know that ultra-nerdy saying about turning you into a small shell script? Bunnie can actually do that!
posted by dirigibleman at 8:54 AM on February 17, 2010 [2 favorites]
Nelson - exactly. I'm surprised there's not a decent write-up of him on the Blue. Until then, Linux Dev Center has a good write-up.
In summary: Andrew Huang, aka Bunnie, is now around 35 years old. He studied at MIT, focusing mostly on computer architecture, include embedded systems and computer security issues. He went on to design Chumby, a little clock-radio sized "ambient consumer electronics product".
posted by filthy light thief at 8:57 AM on February 17, 2010
In summary: Andrew Huang, aka Bunnie, is now around 35 years old. He studied at MIT, focusing mostly on computer architecture, include embedded systems and computer security issues. He went on to design Chumby, a little clock-radio sized "ambient consumer electronics product".
posted by filthy light thief at 8:57 AM on February 17, 2010
You know that ultra-nerdy saying about turning you into a small shell script? Bunnie can actually do that!
It's true. It's not really so bad. Being grepped is actually kind of nice, but please don't open me in Emacs.
posted by loquacious at 9:31 AM on February 17, 2010 [3 favorites]
It's true. It's not really so bad. Being grepped is actually kind of nice, but please don't open me in Emacs.
posted by loquacious at 9:31 AM on February 17, 2010 [3 favorites]
I loved his series on the manufacturing process behind the original chumby - it gives some great insight into all the detailed work that goes on in the factories in China. My favorite piece showed the mad skills used to sew the bags together, but if you follow that category on his blog you'll see plenty more interesting stuff.
posted by babar at 9:35 AM on February 17, 2010 [2 favorites]
posted by babar at 9:35 AM on February 17, 2010 [2 favorites]
I wasn't previously familiar with this "ghost shift" issue. (Though it reminds me of "lunchbox special" guns that factory workers would occasionally make off the books, using spare or rejected parts, and smuggle out. But that was one or two guns at a time!)
So I Googled around a bit and found this amazing Fortune article from 2006, describing how athletic shoe maker New Balance had a Chinese contractor go rogue on them. Went way, way beyond making a few extra pairs of shoes than they were supposed to. They basically metastasized and consumed New Balance's Chinese market whole.
posted by Naberius at 10:13 AM on February 17, 2010 [1 favorite]
So I Googled around a bit and found this amazing Fortune article from 2006, describing how athletic shoe maker New Balance had a Chinese contractor go rogue on them. Went way, way beyond making a few extra pairs of shoes than they were supposed to. They basically metastasized and consumed New Balance's Chinese market whole.
posted by Naberius at 10:13 AM on February 17, 2010 [1 favorite]
I never actually wondered whether Kingston manufactured their own components or not.
I didn't start wondering about this issue generally until about a dozen years back when I was buying my first large-ish tv and went through a co-worker whose wife worked at a electronics store. At that time, I was informed that the manufacturer of a particular decent brand of tv would no longer be as of the next year, or rather, they would no longer manufacture the product, though their name would still be on it. Since then it's come to mind, but knowledge of which company re-brands what (maybe everything, maybe a line, maybe a single model) and knowledge of this as a general issue are pretty different beasts. So thanks for that rebuttal.
posted by Durn Bronzefist at 10:14 AM on February 17, 2010
I didn't start wondering about this issue generally until about a dozen years back when I was buying my first large-ish tv and went through a co-worker whose wife worked at a electronics store. At that time, I was informed that the manufacturer of a particular decent brand of tv would no longer be as of the next year, or rather, they would no longer manufacture the product, though their name would still be on it. Since then it's come to mind, but knowledge of which company re-brands what (maybe everything, maybe a line, maybe a single model) and knowledge of this as a general issue are pretty different beasts. So thanks for that rebuttal.
posted by Durn Bronzefist at 10:14 AM on February 17, 2010
"Ghost shifts" are a serious problem for most companies outsourcing manufacturing to China. You order x widgets, they make y, deliver you the ordered amount and sell the y-x difference on the grey market. It may seem pretty shameless, but it's a problem that the outsourcers themselves have brought upon themselves, by putting so much pressure on their suppliers that they can't turn a profit (low Chinese wages and undervalued yuan notwithstanding) without cheating.
But how is this a bad thing for the company doing the outsourcing? They get the amount of units that they think they can sell in their target markets, and they get those units at a cheaper price because the suppliers can produce them for a loss and still make money off of the gray market sales. If the company doing the outsourcing is selling their products to affluent consumers in developed countries, they aren't going to lose any money from shady sales in developing countries because the people buying those products aren't their target customers.
posted by burnmp3s at 11:38 AM on February 17, 2010
But how is this a bad thing for the company doing the outsourcing? They get the amount of units that they think they can sell in their target markets, and they get those units at a cheaper price because the suppliers can produce them for a loss and still make money off of the gray market sales. If the company doing the outsourcing is selling their products to affluent consumers in developed countries, they aren't going to lose any money from shady sales in developing countries because the people buying those products aren't their target customers.
posted by burnmp3s at 11:38 AM on February 17, 2010
I just "Ghost shifts" are run by actual ghosts. Because, you know, spooooky. *wiggles fingers*
He went on to design Chumby, a little clock-radio sized "ambient consumer electronics product".
I have one of these, a broken one was given to me as a bonus for a photography gig I did. It needed a new firmware which I could fix pretty easily, and the person wasn't really using it.
Sadly, I haven't been able to get them to de-register it, so all I can do is play tic-tac-toe and look at the world population (the default installed products).
It is a pretty cool piece of tech though.
posted by quin at 11:56 AM on February 17, 2010
He went on to design Chumby, a little clock-radio sized "ambient consumer electronics product".
I have one of these, a broken one was given to me as a bonus for a photography gig I did. It needed a new firmware which I could fix pretty easily, and the person wasn't really using it.
Sadly, I haven't been able to get them to de-register it, so all I can do is play tic-tac-toe and look at the world population (the default installed products).
It is a pretty cool piece of tech though.
posted by quin at 11:56 AM on February 17, 2010
so this is where I think the 5 bucks REALLY pay off. Thanks loquacious.
posted by Laotic at 12:01 PM on February 17, 2010
posted by Laotic at 12:01 PM on February 17, 2010
"Isn’t it fun to connect the dots, all the way from silicon die markings to the linux kernel to end users, and all the businesses in between?"
Abso-freakin-lutely!
There is something poetic about this. I totally appreciate the desire to disassemble the mcguffin and really inspect it to discover its secrets. Good, bad and indifferent.
More please!
posted by Ruthless Bunny at 12:29 PM on February 17, 2010
Abso-freakin-lutely!
There is something poetic about this. I totally appreciate the desire to disassemble the mcguffin and really inspect it to discover its secrets. Good, bad and indifferent.
More please!
posted by Ruthless Bunny at 12:29 PM on February 17, 2010
Yes, this is a lot more than just a guy finding Kingston doing something ethically grey, though wholly mundane.
This is about the whole chip industry, outsourced products in general, and ultimately about the safety and security of products used in consumer, communications, medical, public safety and military applications.
It's an issue much bigger than Alton Sinclair's The Jungle. And much, much harder to control.
posted by Xoebe at 1:45 PM on February 17, 2010
This is about the whole chip industry, outsourced products in general, and ultimately about the safety and security of products used in consumer, communications, medical, public safety and military applications.
It's an issue much bigger than Alton Sinclair's The Jungle. And much, much harder to control.
posted by Xoebe at 1:45 PM on February 17, 2010
I like the idea of an Alton Sinclair - he documents the filthy gut-strewn abattoir of the meatpacking plant, then prepares the Perfect Prime Rib.
posted by benzenedream at 2:27 PM on February 17, 2010 [3 favorites]
posted by benzenedream at 2:27 PM on February 17, 2010 [3 favorites]
How did this dude get the CID, datecode, vendor ID etc. of the SD cards themselves? I wanted to check out my cards, but all I can get to from /sys entries is the card reader itself, not the actual card. I have a built-in USB reader on my netbook, maybe that won't work for some reason?
posted by DecemberBoy at 2:30 PM on February 17, 2010
posted by DecemberBoy at 2:30 PM on February 17, 2010
Also, re: the Chumby - isn't this a very, very 1999 idea? It just reminds me of the 3Com Audrey and all those other failed personal Internet devices that failed because people started being less scared of computers.
posted by DecemberBoy at 2:32 PM on February 17, 2010
posted by DecemberBoy at 2:32 PM on February 17, 2010
DecemberBoy: He answers that question a ways down in the comment thread: The procedure iss a little bit tricky and machine-specific. You can’t just have the card mounted as a USB drive via an SD – to – USB converter, since the USB converter hides all of the details of the SD card from the host; you need to have a native SD card interface, like those found on many mobile phones or, as it happens to be, on the chumby One. (and then detailed instructions)
posted by zhwj at 5:49 PM on February 17, 2010
posted by zhwj at 5:49 PM on February 17, 2010
"I wasn't previously familiar with this 'ghost shift' issue. (Though it reminds me of 'lunchbox special' guns that factory workers would occasionally make off the books, using spare or rejected parts, and smuggle out. But that was one or two guns at a time!)"
It's been crazy in woodworking tools space the last 10-15 years. Tools will come out that are obvious ghost shift products. And I'm not refering to chisels and saw blades but table saws and jointers. Often the only difference is in cosmetic machining and finishing of cast parts. Parts are all 100% interchangeable. In a perversion of being punished for doing the right thing the ghost shift products often beat the primary manufactures product to market because the primary spends more time seasoning their castings before final machining.
"But how is this a bad thing for the company doing the outsourcing? They get the amount of units that they think they can sell in their target markets, and they get those units at a cheaper price because the suppliers can produce them for a loss and still make money off of the gray market sales. If the company doing the outsourcing is selling their products to affluent consumers in developed countries, they aren't going to lose any money from shady sales in developing countries because the people buying those products aren't their target customers."
Well things tend to leak out from the developing countries via the grey market. Plus it harms the chances of the primary being able to expand into the developing market.
posted by Mitheral at 6:20 PM on February 17, 2010
It's been crazy in woodworking tools space the last 10-15 years. Tools will come out that are obvious ghost shift products. And I'm not refering to chisels and saw blades but table saws and jointers. Often the only difference is in cosmetic machining and finishing of cast parts. Parts are all 100% interchangeable. In a perversion of being punished for doing the right thing the ghost shift products often beat the primary manufactures product to market because the primary spends more time seasoning their castings before final machining.
"But how is this a bad thing for the company doing the outsourcing? They get the amount of units that they think they can sell in their target markets, and they get those units at a cheaper price because the suppliers can produce them for a loss and still make money off of the gray market sales. If the company doing the outsourcing is selling their products to affluent consumers in developed countries, they aren't going to lose any money from shady sales in developing countries because the people buying those products aren't their target customers."
Well things tend to leak out from the developing countries via the grey market. Plus it harms the chances of the primary being able to expand into the developing market.
posted by Mitheral at 6:20 PM on February 17, 2010
December Boy: I read the data using a half completed project with an ATMega microcontroller. I am sorry I don't have the source code anymore, but the standard is easy to find online. I based teh code on a project by some Ronald or Roland or something like that guy.
posted by dirty lies at 7:02 PM on February 17, 2010
posted by dirty lies at 7:02 PM on February 17, 2010
December Boy: I read the data using a half completed project with an ATMega microcontroller.
Oh, cool, I could probably do that with an Arduino pretty quick if I got bored, or even just a regular AVR. Needing to get closer to the hardware makes sense. Bitbanging SD cards is pretty easy, people throw in SD cards even if they don't really need storage in homebrew projects just because it's so easy to do.
posted by DecemberBoy at 7:08 PM on February 17, 2010
You could even build a bogus card detector that could read a card's datecode and light an LED if it's uninitialized (0-2000). That would be kind of cool, and most people could build it out of parts just lying around. You could use an SD-to-microSD adaptor instead of buying an actual card slot assembly. Test all your friends' cards and see how widespread this "ghost shift" problem is and across what manufacturers.
posted by DecemberBoy at 7:12 PM on February 17, 2010
posted by DecemberBoy at 7:12 PM on February 17, 2010
Ha, not "most people", "most people" that would know how to program microcontrollers, though.
posted by DecemberBoy at 7:13 PM on February 17, 2010
posted by DecemberBoy at 7:13 PM on February 17, 2010
You are this ---> <>$100 high voltage parallel programmer I noticed that there is not a minimum clock frequency when reseting the fuses and bitbanged them back to life with some AA batteries, an 8 position dip switch for the parallel data and a toggle switch for the clock.
I'd love {\} to try that with an SD card.
posted by dirty lies at 7:25 PM on February 17, 2010 [1 favorite]
I'd love {\} to try that with an SD card.
posted by dirty lies at 7:25 PM on February 17, 2010 [1 favorite]
Woah, metafilter ate half my comment. It makes absolutely no sense now.
Basically I was saying that bitbanging a memory card would be fun, like the time I bitbanged an AVR micro that I had fuse locked into high voltage parallel programming mode only. Instead of buying the programmer, I did it by hand. Literally.
posted by dirty lies at 8:25 PM on February 17, 2010
Basically I was saying that bitbanging a memory card would be fun, like the time I bitbanged an AVR micro that I had fuse locked into high voltage parallel programming mode only. Instead of buying the programmer, I did it by hand. Literally.
posted by dirty lies at 8:25 PM on February 17, 2010
Was "rouge worker" a mistype? Should've been "rogue"?
posted by Trochanter at 8:43 PM on February 17, 2010
posted by Trochanter at 8:43 PM on February 17, 2010
Was "rouge worker" a mistype? Should've been "rogue"?
No.
posted by pantsrobot at 2:31 AM on February 18, 2010 [1 favorite]
No.
posted by pantsrobot at 2:31 AM on February 18, 2010 [1 favorite]
The Gibsonian vibe made me wonder if there's any connection between Bunnie Huang and a minor character in Virtual Light, Chevette's boss at the bike messenger service, a dude also nicknamed Bunnie. Homage? Nothing about this in Wikipedia, which was the extent of my search.
Interesting if depressing find, loquacious.
posted by Quietgal at 12:10 PM on February 18, 2010
Interesting if depressing find, loquacious.
posted by Quietgal at 12:10 PM on February 18, 2010
« Older Gamers more scary than bikers | Spy and Pyro Newer »
This thread has been archived and is closed to new comments
posted by orthogonality at 10:55 PM on February 16, 2010 [2 favorites]