Copyright ©1997-<$mt:date format="%Y"$> Glenn Fleishman except as noted otherwise. All rights reserved. For permission to reprint, contact Glenn Fleishman at glenn at glennf.com. Photo © 2008 Laurence Chen; used with permission.
Copyright ©1997-<$mt:date format="%Y"$> Glenn Fleishman except as noted otherwise. All rights reserved. For permission to reprint, contact Glenn Fleishman at glenn at glennf.com. Photo © 2008 Laurence Chen; used with permission.
Turning technology from mumbo-jumbo into rich tasty gumbo
� Better and Better Ben | Main | Speak Slowly, Now �On Nov. 13, I posted a graph showing the fast growth in the requested bytes in RSS and similar feeds from my Wi-Fi Networking News and a few (much smaller) other sites. The bandwidth usage showed a growth from the mid-200 MB per day range up to about 350 MB per average per day. During that same time, I wasn't seeing an increase in visitors of that scale--maybe 10 to 20 percent, not 75 percent.
After analyzing logs, I discovered that a small percentage of aggregation sites and aggregation servers were requesting as much as 20 to 30 percent of the bandwidth unnecessarily through aggressive downloads that didn't check the If-Modified-Since headers or other tools to prevent a retrieval of a page that hadn't changed.
I built a simple program running via Apache that throttles RSS downloads: a given IP and user agent combination can only request a given RSS feed file if it's changed since they last retrieved it. Pretty simple. But the effects are profound, as this graph shows.
As you can see, I threw the switch on Nov. 20, just before Thanksgiving, but I haven't seen a real decline in readership at my Wi-Fi site or the other sites--just a decline in bandwidth. The average (with lots of posts over the last week or so, meaning more RSS retrievals because of the update) is back to about 200 MB.
This reveals a lot about the sloppiness of some of the aggregators out there. Right now, my top aggregator is Mozilla (Firefox, primarily), which makes perfect sense: there are a lot of people using the RSS button in Firefox to subscribe to my feed, and if it's the top engine that's because of many unique users.
Since I pay by the gigabyte for overages above my minimum (which I've hit), this change will save me a reasonable pittance: probably $10 or $12 per month. Sounds like someone needs to build a master site for testing aggregation competence so that aggregator software developers can test this, and users and Web site operators can report on it back to developers.
Posted by Glennf at December 7, 2004 5:19 PM
TrackBack URL for this entry:
https://db.isbn.nu/mt3/mt-tb.pl/2815
Listed below are links to weblogs that reference Throttling RSS Seems to Work:
� Throttling RSS Seems to Work from Grubbykid.com :: Links
Throttling RSS Seems to Work... [Read More]
Tracked on December 8, 2004 7:02 AM
� Throttling RSS Seems to Work from The RSS Blog
Randy: Glenn is finding out that well behaved aggregators mean that RSS scales. [Read More]
Tracked on December 8, 2004 8:34 AM
� Bandwidth-gobbling RSS aggregators: foiled! from you've been HAACKED
[Read More]
Tracked on December 8, 2004 3:05 PM
� RSS demands from JD on MX
RSS demands: Glenn Fleishman compares requests for the XML version of his blog to that for his HTML version. His server now refuses too-frequent XML requests, and he hasn't noticed any falloff in visitors who get the full HTML version.... [Read More]
Tracked on December 8, 2004 3:10 PM
� Bandwidth-gobbling RSS aggregators: foiled! from you've been HAACKED
[Read More]
Tracked on December 8, 2004 4:34 PM
� Glenn’s RSS Throttling Scheme: Flawed. from Dann Sheridan's Weblog
I was reading Glenn’s post about his RSS throttling scheme . [Read More]
Tracked on December 9, 2004 7:26 PM
� My Posts on Throttling from Regular Sucking Schedule
I posted three items over the last few weeks about RSS bandwidth use and my attempts to throttle it back. The first one, on Nov. 13, shows a chart of usage and how rapidly its grown. My second, on Nov. 20, shows my attempts to throttle usage through a... [Read More]
Tracked on December 11, 2004 2:13 PM
� Glenn’s RSS Throttling Scheme: Flawed. from Dann Sheridan's Weblog
I was reading Glenn’s post about his RSS throttling scheme . [Read More]
Tracked on December 13, 2004 1:36 PM
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 |
| 8 | 9 | 10 | 11 | 12 | 13 | 14 |
| 15 | 16 | 17 | 18 | 19 | 20 | 21 |
| 22 | 23 | 24 | 25 | 26 | 27 | 28 |
| 29 | 30 | 31 |
Recent Entries
Archives
February 2009 | January 2009 | December 2008 | November 2008 | October 2008 | September 2008 | August 2008 | July 2008 | June 2008 | May 2008 | April 2008 | March 2008 | February 2008 | January 2008 | December 2007 | November 2007 | October 2007 | September 2007 | August 2007 | July 2007 | June 2007 | May 2007 | April 2007 | March 2007 | February 2007 | January 2007 | December 2006 | November 2006 | October 2006 | September 2006 | August 2006 | July 2006 | June 2006 | May 2006 | April 2006 | March 2006 | February 2006 | January 2006 | December 2005 | November 2005 | October 2005 | September 2005 | August 2005 | July 2005 | June 2005 | May 2005 | April 2005 | March 2005 | February 2005 | January 2005 | December 2004 | November 2004 | October 2004 | September 2004 | August 2004 | July 2004 | June 2004 | May 2004 | April 2004 | March 2004 | February 2004 | January 2004 | December 2003 | November 2003 | October 2003 | September 2003 | August 2003 | July 2003 | June 2003 | May 2003 | April 2003 | March 2003 | February 2003 | January 2003 | December 2002 | November 2002 | October 2002 | September 2002 | August 2002 | July 2002 | June 2002 | May 2002 | April 2002 | March 2002 | February 2002 | January 2002 | December 2001 | November 2001 | October 2001 |