GlennLog

Turning technology from mumbo-jumbo into rich tasty gumbo

< Dotcom Web Services Commitment | Main | BBC News on Download Fiasco >

May 3, 2003

isbn.nu High on Bandwidth

Many of you all know that I run a book price comparison site that checks on prices at a bunch of online bookstores based on the title(s) you select. It's a good moneymaker, and a great way for me to try out ideas and learn about programming.

The traffic to the site had gotten high enough that our office 768 Kbps DSL line couldn't support it: we weren't saturating our bandwidth, but we were filling it quite close most of the time. This line serves our regular needs as well as a bunch of locally hosted sites and isbn.nu.

I decided to follow the precept that if you're bandwidth limited and you get money based on traffic -- in my case through sales that result from clickthroughs on book prices -- that removing the bandwidth cap should increase one's revenue.

I signed a contract to use digital.forest, a nearly 10-year-old hosting and colocation firm. For about $300 per month, they give me up to 10 Mbps. I pay $1 a gigabyte when I exceed a preset amount each month.

One of my decisions was whether to stick with Linux, or move to FreeBSD or Mac OS X Server. After discussing it with the digital.forest folks, I decided to spent a little extra money and get a dual-1.33 GHz Apple Xserve. It has a lot of nice features, including a pretty way to expand storage. I put an extra gig of RAM in for a total of 1.5 Gb.

Since moving services over early this last week, I've watched bandwidth use grow from a few tens of Kbps up to an average during the day of over 1500 Kbps, sometimes peaking almost twice as high. So far, I'm not seeing the financial returns: several affiliate programs let me see the results of sales immediately or within a day.

But I can't be feeding out that much more bandwidth and speeding up the system and not seeing results. Because I have to retrieve XML and similar documents from each bookstore for each search, being on a 10 Mbps Internet feed allows the searches to happen that much faster. Latency is lower. And the Xserve is somewhat more zippy than the previous isbn.nu box.

This also allows me to have the horsepower in my office to build database more frequently. I'm currently nearly two days into a database build of all the subject, author, publication date, and related databases used for isbn.nu searches. It'll take about five or six hours to transfer these databases over from my site to digital.forest, so I'll copy them in the middle of the night.

When I get clever, I'll wind up automating the database build and copy so that each week, when I have a new feed from my source, it'll run the build, and, when finished, initiate a mirror operation.

My only problem will be that once I get on that cycle, I'll need to make sure that I put in another hard drive on the Xserve just for these mirrored databases -- otherwise I'll forget about it for a few weeks and the machine will start screaming about out of disk storage!

Posted by Glennf at May 3, 2003 11:24 AM

Trackback Pings

TrackBack URL for this entry:
https://db.isbn.nu/mt3/mt-tb.pl/2034

Comments