April 13, 2001

12:51 AM

It has to stop!(via rc3) Someone puts up a website, people like it and come back for more, then they tell their friends - and so on. The problem is, the site becomes popular and prohibitively expensive and a valuable resource either gets put behind a pay per view gate, disappears, or the site owner has to bite the bullet and pay a huge hosting fee. (more inside)
posted by owillis (36 comments total)

My brain says the solution is some sort of peer to peer based architecture, but I can't say for sure. I've created a discussion list open to anyone to discuss these issues and a solution more in depth. I don't want these venues to die, and I think the solution has to come from the web community.

More commercial interests have a vested stake in keeping the existing system in place, as this is the cash that lines their pockets...
posted by owillis at 12:54 AM on April 13, 2001

It's now a given that if you happen to create something that really catches on, you run the risk of being punished with high server fees. Nosepilot.com is a recent example. Al created something amazing, art for the sake of art, and got stuck with a ridiculous $16,000 bill for bandwidth. Unfortunately, if it were two years ago, he probably would have gotten a fat contract from Macromedia to produce shorts for Shockwave.com, but in the current climate he has to get a lawyer and fight to reduce the fees.

What if Mahir (the "I kiss you!" guy) was on his own $20/month domain account? Could you imagine the bill he'd get slapped with after his URL was emailed around millions of times?

People have theorized a distributed caching system, possibly being some sort of P2P variant, could fill a need to spread free bandwidth to personal publishers, but I think the lack of control and lack of recognition when your work gets spread out among many, many servers will prevent that from ever becoming a reality.

This site, for example, would cost me at the very least $100-200 a month for the bandwidth alone, not to mention basic server rackspace rental, if I were hosting it at a colocation facility.

I don't think donations are the answer, because they are very seldom enough. It seems free bandwidth of some sort (either from your employer or from a friend) is about the only way to maintain a high traffic, high bandwidth site.

Or, as I told someone last week "the secret is to toil in obscurity, create things that no one will ever see."
posted by mathowie at 1:08 AM on April 13, 2001

Matt, I'm thinking about a distributed caching system with some sort of "ownership" scheme. I'm thinking about it, whether I have the know-how to implement it is another ball of wax.
posted by owillis at 1:10 AM on April 13, 2001

The webtechniques article about the first eight days of amihotornot.com was a real eye-opener, in a "holy shit, better rethink that plan for world domination or make friends with someone who knows about scaling issues" sort of way.

It seems to me that the Net itself has simply got to get smarter, if the egalitarian ideal is to remain within reach. Some sort of caching should be built into the heart of things so that oft-requested data gets buffered closer and closer to the most likely requests. Software can only be a stopgap solution, though it will be on the order of a generation (which is basically a singularity in Internet time) before new protocols can be put in place.
posted by sudama at 1:26 AM on April 13, 2001

On a side (if not irrelevant) note, can someone please tell me what the heck waxy.org is?
posted by kchristidis at 3:36 AM on April 13, 2001

As I said to someone last week regarding the new web climate, 'the goal of publishing is not to be read.'

It seems we've come to a new digital divide between those who can afford to be published and those who cannot. The free bandwidth solution that Matt suggests does seem to be the only current solution that work, unless you want to be cluttered with geocities ads from a "free" host.

The problem with putting a box at an employer is that it could be shut down without notice unless you have a very understanding employer. In that sense, it's no different from any of the hosting solutions currently available.

There are some ISP's (inch.com's personal pages come to mind) that limit bandwidth by measurement and when it exceeds, it's automatcially shut down to prevent overusage. With those sites you have to get up pretty early in the morning to catch the content.

It would be a shame to see the web move to a "time of day" content system.
posted by gdavis at 4:26 AM on April 13, 2001

I dunno. I kinda like "toiling in obscurity." but then I'm a neurotic fruitcake.
posted by ZachsMind at 4:54 AM on April 13, 2001

don't we already have a caching solution with caching proxy servers? the problem with this solution lies, i believe, with regularly changing content. the only way to get the most up to date version of the site, is to connect directly to it. certainly static content can be mirrored or cached, but it's not generally static sites that gain this sort of popularity.

why not just run banners for porn and gambling sites, they are after all the big money spinners of the net :)
posted by titboy at 5:50 AM on April 13, 2001

11G a week is a lot of traffic, but I'm guessing he could probably find a host that would serve it for $250/month. That means he only needs to make $8 a day on click-throughs and affiliate fees to be self-sustaining.

The dilemma here is for sites like Nosepilot (or MetaFilter) that don't want to go commercial with garish ads. If those publishers can't find a generous ISP, they should look for opportunities like IBiblio at UNC, which hosts non-commercial sites for free if they see value in the content.
posted by rcade at 5:53 AM on April 13, 2001

The PHP-Nuke (and variants) community has a couple of typical examples of free sites running free software which have suddenly realized that there ain't no such thing as free bandwith. But in a simple analysis I recently did of two sites that were suffering, I found that more than half of the front page download size was graphics.

For non-professional, not-for-profit sites, the first logical answer to this is: reduce the size of each page. We're still designing too large. On any given day's browsing, I'm encountering sites with more than 100K downloads on the front page. And reduce the size of the graphics themselves. I've got a RAM disk holding my browser cache that's just stuffed with too-large banner heads and useless logos.

The second answer is: give the graphic hosting task to someone else. On one of my sites, I use Apple's iDisk to host my images. They've got good uptime and decently fast servers and a long-term history that suggests they'll be around for a while. My visitors don't know the difference; my bandwith needs are a fraction of what they might be.

The third answer is: mirrors and a round-robin Javascripts or CGI. There are still many, many free services out there. Use one as a front page to distribute the script and mirror links, the others to host the content. (Greenspun, home of other free services, might consider this one, too: free round-robin CGI hosting).
posted by Mo Nickels at 6:40 AM on April 13, 2001

rcade, $8 a day ($72 a week) on clickthroughs is a lot to ask. The ad network I have on one of the pages I'm on pays a whopping 12 cents per click. At 11,000 page views a week he'd need a clickthrough rate of 5.4%, or 600 clickthroughs @ $.12.

Given that the going clickthrough rate now is less than 1%, that's a tall order.
posted by werty at 7:06 AM on April 13, 2001

kchristidis, waxy.org is the home of our very own waxpancake.
posted by Aaaugh! at 7:08 AM on April 13, 2001

Would everyone consider selling shirts through CafePress evil? I would bet the MetaFilterians would buy up shirts with Matt's snappy MeFi taglines on them. I appreciate the concerns of keeping certain parts of the Web commercial free, but I don't think anyone should expect the producers of cool sites to lose money every month. It could be like PBS. Donate $25 and get a sweatshirt.
posted by quirked at 7:39 AM on April 13, 2001

Metafilter shirts and mugs are already avilable through Cafe Press.
posted by waxpancake at 9:22 AM on April 13, 2001

On a more related note (and a more useful comment), I just joined the mailing list. I think there's a big future in distributed hosting. (Think Akamai meets Napster.)

I'd love to help other interested people tackle this massive problem.
posted by waxpancake at 9:37 AM on April 13, 2001

That thought about putting the graphics someplace like your ifiles site is brilliant. Gotta be careful though, some of the "free" sites are configured to not allow graphics unless they are being "referred" from a page on the site.

Another thing to do is to forget javascript and do as much scripting you can on the server, where it doesn't take bandwidth. I've seen javascript be two or more times bigger than actual content, often to accomplish really minor things that could have been done from the server.
posted by gdavis at 9:38 AM on April 13, 2001

I've been thinking about this, too. So has the founder of Freenet; he's seeking to leverage the lower-level infrastructure of that venture into a commercial product that would help in situations like this.

I think Grant's ultimately right, though. A lot of sites are just not optimizing what they're serving for the purpose they exist for. I wonder about that 11Gb that the baseball site is spitting out: how much is graphics? how much is data? would an XML interface be more efficient? what about deltas? In a nutshell, does every page view count for the most?

sudaman, thanks for the hotornot article. Interesting that soon after hotornot decided that they could allow their users to host their files at geocities, geocities decides they can't be used to serve up graphics anymore.
posted by dhartung at 9:41 AM on April 13, 2001

<rant issue_type="side">
Freenet's are freely available information and email services, usually run in cooperation with public libraries and the like, no matter what Ian Clarke thinks.
</rant>
posted by baylink at 10:07 AM on April 13, 2001

This is an argument for slim design and also, with an AOL account I think you get lots of free space. Maybe a way to harness that "free" bandwidth with good uptime, and no popups. I know I have an aol account for testing, and you can have like what, 10 screennames or something each with its own 5MB of space? That adds up, and might be a means for a DIY akamai. As I think of it, perhaps nosepilot could have done something similar. Host a php file which dynamically chose a different AOL screenname's version, set a cookie, and gave people the flash from there. Hate AOL? Well, most ISP's also provide free space as well. Clever harnessing something you paid for anyway might be good.
posted by artlung at 10:30 AM on April 13, 2001

Metafilter shirts and mugs are already available through Cafe Press.

So they are. They are just being sold at cost. I guess my question should have been more general. Does the "keep it free" crowd think it is ok if a site commercializes (nice word) just enough to break even?
posted by quirked at 10:33 AM on April 13, 2001

DIY Akamai. I like it.
posted by artlung at 10:34 AM on April 13, 2001

"Does the "keep it free" crowd think it is ok if a site commercializes (nice word) just enough to break even?"

Well, I can only speak for myself, but I think it's fine. There's a difference between charging money for content, and asking for money if you like it, especially if you get something in return. Like you said, it's like PBS.

I've tossed a few dollars out to people with personal sites, and even bought a book off Noah's Amazon wishlist as a thank-you for his awesome, free Greymatter blog software.

I've thought about doing the same kind of thing myself, but I need to start updating my site more (read: "providing content") before I ask for anything in return.
posted by CrayDrygu at 12:02 PM on April 13, 2001

"Toil in obscurity" -- I (and I suspect most of us) are experts in the field...

The odd thing is that you never really know what's going to "play in middle America". YOu could create what you think is pure genius, and you get three hits. You create some weird animated singing parrot and suddenly you've got Metafilter, Slashdot and half of Japan breathing down your Alteon.

The network effect is so unpredictable...
posted by fooljay at 12:04 PM on April 13, 2001

I've been running into this problem a lot lately. I run NWTekno.org, a site for electronic music enthusiasts (eg, ravers) in the Northwest (mainly Seattle and Portland areas). It's a really strange site, in that there are thousands of active users, and for the most part, they actually know each other in real life (as opposed to mefi - i don't know any of you). There are kids that use the site as the whole basis for their social lives. It's a weird thing to be responsible for.

Anyway, the site's been growing, and we're hitting about 2GB/day of traffic these days. It was about a third of that when I talked a friend at Dreamhost to host it for free, so I'm not sure how long they'll want to do that. I've certainly learned a bit about scalability issues, especially as it pertains to UBB (which I hate, actually, and plan to switch to vB asap).

We've tried raising money, but in this case, it's pretty difficult. Most of the users are 15-19. Many of them don't have credit cards for PayPal, and the base price at CafePress is too high to get them to buy stuff from there even if they did (although, people have said they'd buy hoodies - if only CP made them). We raffled some prizes, but it netted us only about $200.

Looking into hosting services that I can afford, it seems that ProHosting comes in well, at $100/mo for unlimited everything. If it ever comes to that, I'll probably just have to split the fees with someone, and make anyone that promotes their events on the site agree to put me on the guest list. Maybe then I'd break even.
posted by endquote at 12:35 PM on April 13, 2001

Of course "unlimited" tends to mean "a lot, but not really unlimited"...
posted by owillis at 2:01 PM on April 13, 2001

Werty: I don't think any of the sites we're talking about here could survive on click-throughs alone. But with the right affiliates, the Baseball-Reference webmaster ought to be able to make more than $8/day on the 92,000 pages he serves every day (not 11,000 a week).

With all the books about baseball, an Amazon.Com affiliation alone should generate $10,000 to $15,000 a year for him if he finds a way to serve up affiliate links in a compelling way.
posted by rcade at 8:55 AM on April 14, 2001

Mo's on the right track. The way to cut down on your bandwidth consumption is to push less data. Designing lighter-weight pages is a start -- using CSS positioning rather than complex HTML table hacks allows you to do this without resorting to 1995 design minimalism.

The next step is to take advantage of HTTP 1.1 compression. In short, the server zips up the requested data before it is sent; the browser then unzips it before it is rendered. Most browsers and servers support this now -- all you have to do is turn it on.
posted by jjg at 10:12 PM on April 14, 2001

Caching solutions? Distributed hosting? How about this?

DON'T choose a web host that charges by the amount of files transferred. That's a bad deal.

DO choose a host that charges a flat rate regardless of how much traffic you get and how many files are transferred per day. That's a good deal.

I'm sure there's a local company in your area that would set you up on a commercial server for anywhere from US $100 to $250 a month. Maybe less. On such a server, you can handle any amount of traffic, and you're never penalized for having a good day.

I also don't understand the assumption that independent web publishing is supposed to be a profit center, or at least a self-sustaining enterprise. Sure, that would be nice. But with 10,000,000 personal publishers out there, most of us reaching a very narrow self-selected audience, where do you expect the revenue to come from? Is it really worth fucking up your site with ad banners to cover that $8 a day?

Let me put it another way. If you belong to a gym, do you expect to "break even" on that? If you see a doctor or a shrink, do you expect to earn some bucks back?

I think it's a gift to be able to publish. Generation after generation, people have given up their lives for freedom of speech and the right to be heard. Now we have it, and we want to make a buck off it. I just can't see it.

I don't expect to be paid for dinner conversation. I don't expect to be paid for reading books. I don't expect to be paid for expressing myself on a personal website. I do other things to make money. Those things cover my $8 a day.

One last thing. Low-bandwidth design has always been good and always will be. Not because it saves you money on an insane, no-win hosting bill where you're charged by the file. But because it's good for your visitors, who may have slow dial-up connections.

I'm sorry for being out of step with everything that's been discussed here. Call me old-fashioned. I design corporate sites for money. I self-publish because I'm driven to do so. I don't expect anyone to subsidize my obsessions. If a few people like what I do, that's the payment.
posted by Zeldman at 12:38 AM on April 15, 2001

Ouch.

Few of us expect to make a living on our personal strange web projects, but neither do we expect to become viral sensations, then get screwed by tremendous bills.

Who is there to counsel small publishers on what constitutes a good hosting deal? Hosting decisions are driven by features, then price. Folks figure if they get their Cold Fusion+SQL Server or PHP+MySQL or JSP+Oracle config right and the uptime is good.

As a small publisher serving a niche, what resources are there to call on to try and compare terms on bandwidth for hosting agreements?

I think as leaders in the web world, we should be working to counsel developers, authors, designers, on what kind of questions to *ask* of a potential hosting company. Hosting companies have a tough job, and most of them provide only mediocre service, and some are downright unethical.

And another note: "unlimited bandwidth" means "you get to share a box with a bandwidth hogging porn site."

The constellation of variables in choosing hosting is difficult to navigate in an intelligent way. Bandwidth, costs for add ons, support, backups, service level agreements, issues with intellectual property, issues

BTW, fuckedhosting.com is available, last I checked.

We need a BBB and/or a Consumer Reports for hosting companies, and we need to educate web authors that when they start researching hosting, they're entering a den of theives and need to act accordingly.
posted by artlung at 10:58 AM on April 15, 2001

[i hate when i mean to hit "preview" and instead hit "post"; not that this is any more articulate, of course. ]

Ouch.

Few of us expect to make a living on our personal strange web projects, but neither do we expect to become viral sensations, then get screwed by tremendous bills.

Who is there to counsel small publishers on what constitutes a good hosting deal? Hosting decisions are driven by features, then price. Folks figure if they get their Cold Fusion+SQL Server or PHP+MySQL or JSP+Oracle config right and the uptime is good that they got a good deal.

As a small publisher serving a niche, what resources are there to call on to try and compare terms on bandwidth for hosting agreements? In my experience, not much.

I think as leaders in the web world, we should be working to counsel developers, authors, designers, on what kind of questions to *ask* of a potential hosting company. Hosting companies have a tough job, and most of them provide only mediocre service. Some are downright unethical.

And another note: "unlimited bandwidth" too often means "you get to share a box with a bandwidth hogging porn site."

The constellation of variables in choosing hosting is difficult to navigate in an intelligent way. Bandwidth, costs for add ons, support, backups, service level agreements, issues with intellectual property, and more.

BTW, fuckedhosting.com is available, last I checked.

I think the web needs a BBB and/or a Consumer Reports for hosting companies, and we need to educate web authors that when they start researching hosting, they're entering a den of thieves and need to act accordingly.
posted by artlung at 11:05 AM on April 15, 2001

I use SoftCom for most of my Web hosting. They permit unlimited bandwdith and have never asked me for any extra money, even for a site I own that maxes out a T1 whenever the guy I bought it for releases a new version of his software. Plans start at $10 a month and include ASP; for $15 a month you can host a database with them too. And they don't allow porn, MP3, or warez sites. Speed has been acceptable; although it's not the world's fastest service, it's generally not slow enough to piss off users unless you're serving up really, really huge files. I imagine it could bog down pretty quick if you're doing a lot of back-end database stuff, but if bandwidth is the main bottleneck, you might want to check them out. I'll probably regret telling everyone on MetaFilter about them, because you're all going to start hosting sites there and bog down my sites, but what the hell.
posted by kindall at 11:56 AM on April 15, 2001

zeldman: I'm sure there's a local company in your area that would set you up on a commercial server for anywhere from US $100 to $250 a month.

absolutely, for those who are doing that much traffic to begin with. my hosting service (pair) charges me *much* less than that a month, and it would be a waste of my money to pay more; I never go over the bandwith limits they have set.

the problem would occur if I somehow got a lot of attention and suddenly was going *way* over the limit; and once that happened, I'd have to find a place such as the one zeldman suggests.

best case, I'd pay a heap of money one month and then find an all-you-can-eat plan from then on. but for one month, I'd be screwed.

rcb
posted by rebeccablood at 2:08 PM on April 15, 2001

Kindall, I use SoftCom, too - but I'm not so sure I'd recommend them to others, for two reasons. First, their uptime rate is not terribly good; I think my page was down for the better part of a day earlier this month. Secondly, they don't support Perl scripts, which may or may not be a problem for people. (And they say that all SSI pages have to be in the cgi-bin, too. What's with that?)

BTW, does anyone know anything about Synapse Global hosting?
posted by Aaaugh! at 6:39 PM on April 15, 2001

For the price, I think their uptime's pretty good. I didn't actually notice the outage you referred to, but then, right now most of my sites aren't really doing much. I had more problems with outages at DreamHost when I tried them; they had my e-mail bouncing for most of a weekend.

You can sorta run Perl at SoftCom, if you compile it to an EXE first using a PC tool they suggest. I don't do Perl so I've never tried it, but I suppose it's possible that this tool is completely lame. I think the SSI restrictions are some kind of security precaution, which makes a certain amount of sense given they're running IIS...

This summer I'm planning to be doing lots of development for sites of my own, and I'll probably use SoftCom for 'em.
posted by kindall at 9:19 PM on April 15, 2001

Not to air dirty laundry (I emailed Kindall privately before I realized the discussion turned this way) but I'd recommend not using Softcom in particular.

Last fall, their mail servers blew out my inbox twice while I was checking mail from remote, via their own mail2web service. I lost three weeks of email. Softcom doesn't keep backups of its mail files, so my email couldn't be restored; customer service refused to compensate me in any way or even apologize for the server errors.

With my current host (Pair), I pay more per month -- $27 versus $17 for three URLs -- but I have better uptime, diligent customer service, and no worries.

With site hosting, as anything, you get what you pay for.
posted by werty at 7:13 AM on April 16, 2001

Inch.com has hosted my site since May 1995. They never charge for bandwidth and they've only gone down once in nearly six years — and that one time was not even their fault. They've handled insane amounts of traffic at times (Slashdot) and the performance is the same during super high traffic as it is on slow days. They're also nice people, and responsive. I'd recommend them for anyone who's looking for a new host.
posted by Zeldman at 6:10 PM on April 16, 2001