Copyright ©1997-2011 Glenn Fleishman except as noted otherwise. All rights reserved. For permission to reprint, contact Glenn Fleishman at glenn at glennf.com. Photo © 2008 Laurence Chen; used with permission.
Turning technology from mumbo-jumbo into rich tasty gumbo
If you want to see a funny juxtaposition of me and an earlier post’s subject, read Leander Kahney’s Web Services story at Wired News. Web services was a huge topic at Emerging Technology: it’s essentially a backchannel method of querying a Web site’s data store. So I can retrieve pricing and other information from Amazon.com through a lightweight XML transaction instead of a heavy, ugly Web page. Not only do I get more information, I get more explicit permission to use it.
But my fear has been that Web services is a one-way street: you sign a license agreement to use the site’s data a certain way or not use it in others, but you don’t get a commitment (in these dotcom cases) in which the site promises availability, persistent, or other service-based agreements. It’s lopsided because the sites, like Google, have the information, and we’re supplicants.
The amusing part is that I had the conversation with Leander standing next to Jeff Bezos, who might have been pointedly ignoring me or just oblivious of my existence (I worked at Amazon.com several years ago and was directly hired by Jeff). Odd to be juxtaposed in the story with Jeff while not having any human contact with him.
The big messages at EtCon in order of importance (most to least).
1. Swarms/social modeling. Groups follow rules. Failure to accept the fact that groups follow rules guarantees failure because regardless of your personal view of reality, groups follow rules. Anticipating group behavior allows you to create rules that can mitigate, not eliminate emergent patterns.
2. ____ ______ (his absence, his politics, his force-of-nature personality, his influence). I suggested putting an empty chair next to Chris DeBona at the Journalism BOF on Wednesday night because his presence was so palpable.
3. Technorati rocks, but bloggers rule.
4. Those who forget the past are condemned to rewrite it.
5. Google! Google! And a triple!
6. Web services has reached overhype, but its influence and deployment grows, just on the downside of the hype curve as its utility increases.
My best personal soundbite about my success in achieving Net.fame from WiFiNetNews.com: “I’m following the pheromone of my own ego.”
More photos from Emerging Technology including exciting shots from Emerging Man, a camping social experiment a few miles away from the Santa Clara Convention Center. Me, I’m burned-out man.
I’m not a particularly interesting person in my own right. I’m very good at collecting stories and facts and turning them into compelling non-fiction stories. This leads people to believe that I might, in fact, be interesting.
At the Emerging Technology conference, I’ve been watching my own interactions and those of people I meet. I’m relatively well known in this crowd and I have a speaker badge. At the risk of being one of those people who says, “There are two kinds of people: people who categorize people, and those who don’t,” I’m seeing a lot of emergent Whuffie-based behavior.
(What’s Whuffie? It’s a catchall term for your relative reputation based on many factors that Cory Doctorow used in his recent novel, and which derives from an even older source he was involved with.)
The more well-known the person you’re near, the more likely you are to be deemed interesting. I now know a number of relatively to extremely well-known people. I’m not very opportunistic. I like people who engage on ideas, and many of the folks here who are well known are the kind of folks who talk and are interactive.
Well-known people tend to avoid interacting beyond a superficial level. I’ve met many people I’ve wanted to meet for a while, and had great conversations with some of them. Others, even though we’ve corresponded or I’m on their radar, they had a sort of learned avoidance behavior in crowds that I was either sucked into, or I gave out signals that triggered that behavior. I may be too effusive, complementary, or cognizant of someone’s written or spoken works.
The last conversation that ends is the one conducted by someone who has a constitutional inability to stop talking. They always win unless you’re rude or Tim O’Reilly. This might offend some people I’ve talked to, but I’ve had a number of excellent, long conversations, some of them over an hour, and I’ll leave my colleagues guessing which ones of them I’m referring to.
Let me explain my Tim O’Reilly crack: it’s not a crack, it’s rather a compliment. Tim has a natural, quiet ability to extricate himself from a conversation when that conversation is done. He’s not offensive about it; it’s very subtle. But it allows him to maximize conversation efficiency. He’s well known enough that the degree of deference paid to him combined with this natural ability allows him to leverage this conversational/social advantage.
Man, am I tired.
A couple of hours ago, I spotted Jeff Bezos here at Emerging Technology talking to J.C. Herz (currently sitting in the row in front of me at a session) and Kevin Kelly (one of Wired’s early gurus among many other projects). I approached to say hello, as I hadn’t seen him in a couple of years, and then the laughter began. The three of them, triggered by Jeff’s remarkable, contagious bray, went into a kind of laugh escalation.
I was standing nearby, looking for a quick chance to say hi, and started talking to Leander Kahney. He’s working on a short piece about Web services, which are a big topic of conversation here at Emerging Technology, and whether applications will wind up increasingy relying on Web services.
My take is that Web services fed by companies that don’t offer two-way contracts — contracts that promise continiuity, uptime, consistency, and persistence — will make application developers nervous about relying on them. My isbn.nu site relies on lots of Web services, but I can’t rely that any of them will exist in the future, that I won’t be cut off from them, etc.
The irony, of course, is that I’m standing next to Amazon.com’s founder and CEO talking to Leander about the fact that Amazon.com isn’t offering any commitment to Web services, it’s a large experiment, but that that’s within their rights. Meanwhile, the laugh jag continued.
I never did get a chance to say hello, either.
Craig Silverstein from Google, employee number one and director of technology, is talking here at Emerging Technology about how Google accomplishes innovates, rather than how they technically carry out their tasks.
Craig says that the goal is to do things that matter. He used to believe that “for a company to be successful, it really had to be evil.” But Google has proven that you can do things that make a difference to people and succeed. “It’s one of the reasons we’re still around after the bubble.”
Google resisted pressure to run popup ads in the early days, as well as banner ads, because they wanted to create lightweight unobtrusive text ads. “It’s successfully bringing in money for us now.”
At Google Labs, the company puts out ideas that they aren’t quite nailed down on yet. They don’t know the precise feature set or how to accomplish a certain goal. The feedback allows them to “do something that’s really useful to people” when and if they turn it into a product.
The cost of switching search engines for the user is zero. “We have to have it that users would rather go to Google than other search engines just to survive.”’
Craig emphasizes hiring: “it’s the key to what makes our process successful.” His slide reads, “brilliant people have good ideas.” Google trusts that its people will do the right process because they’ve hired the right people. “They realize the value of trying to do the right thing.” A creative environment creates creativity.
Design is a first and primary component of their development process. “Our site looks simple, and looks artless almost, but it takes a lot of work to get there.” He shows the first beta search interface for Google, which he says an article described as “anorexic.” Hard to get things wrong with a single form, and fast to download. They counted the bytes on the home page, and they still do.
Google keeps “the top 100” list: the top projects that they’d like to do. It’s not 100, might be 130. They’re not working on all of them, but by keeping this list, and ordering them vaguely by priority, they can keep their mission statement in mind — they don’t wind up working on something unrelated.
He shows a page of the list. The first item is, naturally, Build a Search Engine. Number two: Crawl the Web. Number three; Google News. “Just last month we got the idea of building a search engine, so we’ll be working on that.” (laughter)
There’s an iterative process of figuring out what works. Shows a chart of wireless (cell phone) traffic. It meandered for a while, and then jumped when they signed a partner last Christmas. It was way down on the top 100 list, and now it’s maybe number 12.
They meet weekly and review priority of the “living document.” “It’s very much essential for this to be a success.”
Their approach is to use small teams: “small teams are fast and agile,” the slide says. The whole group works on a project, including design, testing, launch, and continuing engineering. The Web crawler team is Kingson and Jill. They crawl 3 1/2 billion pages per month, and “these two people are the ones responsible for making it happen.” Other teams help, but they own it.
Hierarchical communication is antithetical to their small group approach, so they have to constantly ensure that communications continues across the entire organization. When a project is ready to start, the entire company can be involved, but a few people are responsible for dealing with comments. “Make sure the groups are sharing experiences, sharing technology, sharing ideas.”
They have an interteam group, which is really 10 groups of three, not one group of 30 people. One group helps make sure that new people can understand how to get into the code base. These positions are “volunteer”: people take time out of other projects.
They have a weekly tech talk in which people talk about what’s in progress or finished, and then they put the talks up on the Web site as videos. It helps new employees understand what happened with the project.
Google creates tools to help them foster communications. They have an internal search engine — “you can guess which one” — over their internal Web sites. They have a status page that’s created from emails sent to it and that maintains state for a week.
When Google acquired Pyra Labs (Blogger’s creators), none of the press speculated that they might use blogs internally, but that’s the first thing the group suggested when they came on board.
In the old days, they put stuff up and waited for comments. But now they do user studies. Their first stab post the initial home page was, “I’m waiting for the rest of it.” “Is this some guy’s home page?” “How many people work there?”
Google Labs allows them to test ideas that aren’t ready for prime time. “My particular favor is Google sets.” You enter the first part of a set, like the names of some of the Seven Dwarfs, and it helps complete them.
Google.com/jobs: “I encourage anyone who is thinking about switching jobs or starting a new one to apply to Google.”
They hired a compiler expert, which they didn’t exactly need, but the first thing he did was massively improve the speed of part of the back-end.
They have one hiring committee, just a few people, and this committee hires everyone. “It manages to keep us consistent” and it separates hiring from head count. Someone might say we need 10 engineers to make the ad system up to date, but that person isn’t making the hiring decision: the committee doesn’t have those pressures. “It’s much much more important to use that we hire people who fit this criteria we have than to miss people who are great.”
“Bad employees are the time sink that keep companies from being as successful as they could be.” It allows them to run all the projects they do at the same time without worrying about these hiring issues. Only a few people at Google haven’t worked out.
A Google News home page can’t be lightweight: sort of the opposite of news. They wanted something that was “googly.” He walked through an iteration and how they decided on several elements, including top stories.
During the bubble, people said, “The Web changes everything.” That true? Not exactly. But before the Web how would they have created what they do? Google wouldn’t have existed, of course, but before the Web, simple search engines with unsophisticated search engines worked fine. Wouldn’t be able to have small groups because you couldn’t talk broadly enough efficiently enough. Sharing information on Web pages that archive information “that allows us to have the kind of communication we need to have these small teams.”
The company runs the way you think you might run a company if you weren’t actually running one. Especially unlimited food.
(I visited Google yesterday, and their biggest problem in scaling has nothing to do with their search engine, but rather with their dining room. The Grateful Dead’s former road chef runs the free restaurant, serving lunch and dinner, and they’ve got so many employees at Google now, that even with an overflow outdoor tent, they can’t quickly handle the full-on lunch load. And, man, is that food good.)
Q: With new office in New York, how do you maintain culture? A: Groups there working through same processes.
Q: [inscrutable question on groups] and then why not more than 3.5 billion pages. A: I didn’t understand your first question, so I’m just going to answer the second. Some finite resources, but will be solved over time.
Q: Applied Semantics acquisition? A: Can’t comment directly, but name implies what it does.
Q: How do you rein people in? A: Happy to have unrelated ideas, although they might not get selected to discuss. Gatekeepers are technologists who stand at different parts of the process to see what’s considered.
Q: RIAA sued students who built search engine for college intranet. Google is better at searching MP3s…if you type “MP3s and…” Craig: Stop it! Right there! [laughter] A: Why don’t we have a music search? Exactly this reason: intellectual property. Not much legal music and video on the Web, and “we don’t want to make it easy to find this.” Can only imagine illegitimate uses for it, then won’t do it. But if it has a good goal, they don’t try to secondguess. Only nix pages that the content owner asks for or that are stolen.
I wrote an article that appears in today’s New York Times about my own bandwidth blowout — avoided — and other folks’ similar experiences.
Andrew Orlowski wrote this hilarious, non-fact-based account of how the Emerging Technology conference’s agenda was set. Apparently, Clay Shirky is the god we all worship.
I was one of the track chairs for the event, focusing on wireless, and the process of soliciting and encouraging proposals, and reading and commenting on them was entirely consensus driven with about 7 to 10 people involved in conference calls and online collaboration.
It was a great way to build a conference. The only problem was that there were too many excellent proposals. And a few proposals in which excellent emerging technology was wrapped in marketing instead of clearer talk. I had two proposals rejected myself, and several folks I’d suggested also had their talks rejected. But a few made it through into a very tight schedule that’s bursting with interesting ideas.
The danger of soliciting proposals is that when proposals are rejected, people have sour grapes because their particular hobbyhorse doesn’t wind up fitting into the overall vision for an event. I’m sure academics deal with this all the time.
(Update: I just met Clay Shirky and another track chairperson Geoff Cohen. We discussed the article, which Clay had heard about but not read yet, and we’re all quite amazed at the Register publishing this piece. It reads like an account of the Protocols of the Learned Elders of Wi-Fi. We were there. We know what happened.)
Even later: Tim O’Reilly wrote why the Register article was a hack job. Meanwhile, Clay approach a group of his friends and colleagues, and they prostrated themselves.
Robert Scoble takes a job at Microsoft and before he even moves up here, the Seattle Times is writing about him in their Web business gossip column, I guess you’d call it.
I’m off to the O’Reilly Emerging Technology Conference in a few hours. Running Tuesday (tutorial day) to Friday, this event promises to open my eyes to lots of tech that I’ve heard vaguely about or not at all. The event is based on a Call For Proposal (CFP) approach in which random people submit proposals, and I (chair of the “untethered” track) and others chairs and O’Reilly folk ask folks to write up proposals on topics we think they should present on.
The result is an eclectic, sometimes focused set of subjects, many of which relate to several O’Reilly interests, like privacy, wireless, swarm, biological modeling, emergent behavior, nanotech, grass roots, etc.
I’ll be reporting from the field; I’m part of a tutorial on Tuesday morning and then free to write up what I hear.
I renovated the server room in my office today. Previously, I had a table with stuff crammed above and below it, and was unable to really reach everything without accidentally unplugging equipment or knocking stuff over. A non-ideal situation. (If the headline doesn’t make sense, I’m referring to the female magic of removing a bra without taking off a shirt or blouse.)
I bought an aluminum server rack from a pal who is clearing out old gear as part of a move for his computer consulting business to a new space, and he included some shelves and wire guides.
Because the two main servers are on an uninterruptible power supply (UPS), and because of my fancy footwork, I was able to get everything off the table, power down two non-critical servers and remove them, disconnect everything, and keep the two Unix boxes running the whole time. I had to extract the table through a tight doorway.
Then I drilled some holes and screwed the rack into the floor and wall, mounted the shelves, and pivoted the servers into place, using rubber straps to strap them down in case of earthquake.
All that and no downtime! Go figure.
I’ve got a 1000 watt UPS, that’s on the floor right now, but I have a bid on eBay for a 1400 watt rack-mounted unit that’s quite inexpensive and would be easier to locate.
Racks rock. This is the first rack I’ve owned. The price has come down remarkably. Next, some rack-mounted equipment…
The Starbucks Oracle claims I’m a freak because I order decaf americanos as my drink of choice. I also apparently go to ski resorts in the summer and drink non-alcoholic beer. So it says.
Leander Kahney wrote an update about my averting the bandwidth disaster I’d faced. I just contacted Project Gutenberg to arrange for the donation, and they’re excited by the windfall. I need to wait for the PayPal and Amazon.com money to clear, which is another few days (when you hit 30 days, there’s no easy way to get a refund, so it’s easier to wait until that point).
The oil tank in the backyard has been removed, and the earth has been mostly healed, but a lot more reconstruction is still necessary. The bills arrived, and they were substantially higher than expected — not double, but a good 30 percent higher than the contractors prepared me for, and I’m trying to get a reasonable itemization.
I’m also working on an article for the Seattle Times Home and Real Estate section detailing how to deal with finding, deciding to remove, and ultimately removing a home heating oil tank. It’s a money pit. It’s unlikely that if there’s even a tiny leak from an old, unused tank that you could walk away for less than $7,000, according to sources. More likely you’ll spend more. Some homeowners have spent $100,000 to remove contaminated soil.
You don’t have to remove the bad soil, but if you leave it, you must disclose it as a material defect. Who buys a house with an unknown amount of contamination on the property? So we did the right thing.
____ ______ reports that the New York Times archives have returned for free through direct links. I have confirmed this through many of my own links. Was the linkrot a test of whether people would notice (and pay)? I have to think that thousands upon thousands of these links are clicked on every day.
Joel Spolsky has an accidental in-depth article on renting office space in New York. As anyone who rents commercial space finds out, you either become an expert like him or you get dicked over. End of story. There’s very very little intermediate space if you’re renting in office buildings or from serious landlords in which you get a reasonable lease (not talking about rent but lease terms) unless you behave like an SOB and really cross out parts of the lease and write in new parts.
I’ve negotiated three leases in the last several years in Seattle, each of which I’ve learned more than the last. My lawyer has educated me. We didn’t, in fact, use a lawyer on the last lease because we were (we still think) so well educated from the last lease, on which we spent a small fortune to have it reviewed and negotiated, but we think it paid off in the end.
Joel’s article is good reading and good advice. In Seattle, the situation is similar but much simpler. There’s not as much pressure, even during the dotcom period. Buildings tend to be newer. There’s either good spaces, decent spaces, or crap, and there’s not a lot of those old garment district and heavy industry buildings with lofts that Joel describes in New York.
I’ve been in Class A space (the Tower Building, owned by the Vance Corporation), and the lease negotiation was straightforward, the final lease fair, the landlord fantastic — treated us like kings despite only have .84% of the building’s space — and no hidden surprises.
I’ve been in junk space, a building on Green Lake that, when we moved in, we were told would be torn down to build condos at some point. Fortunately the market collapsed. When our lease was near up, we made a very nice offer to the landlord to pay him about 20 percent above market instead of the nearly 60 or 70 percent above market that we were paying. He thought he’d find tenants, he said, at that rate. (Instead, he moved in with some subletters he found.)
We moved to what’s probably accurately called Class C space: no elevator, but only two floors. Full heating, air conditioning, ventilation (just rebuilt from scratch — we lived through that, but we don’t mind now that it’s done). Renovated offices. All that included in the rent and no build-out costs because it was built just the way we needed it. And we’re paying about what we would have been willing to pay in our junk office that needed a lot of work just to stay in one piece.
Oregon is considering a bill that would put open source software as the preferred vendor, as it were, and require agencies to justify using closed, for-fee software.
It’s probably too broad, despite my love of and use of open-source, free, and similar software. A good response on it comes from a Microsoftee who identifies the costs of running any kind of software. It’s not free. You need hardware, staff, training, support, development.
Where open-source really shines at a municipal level has to be on the server front. Forget client licenses. You may or may not like Office, but it’s a universal product that’s used everywhere, and you have to fight like heck to find something comparable that’ll read and write its formats. Someday, there may be a real competitor, but I still use Word and Excel every day. (Fortunately, I’ve switched to Keynote for presentations from PowerPoint; PowerPoint has always lacked sophistication and ease of use.)
On the server end, though, the licenses to support even moderate numbers of users escalate fast. Customization requires in-house or outsourced programming where it’s possible, and it’s always tacked-on. With Red Hat Linux, MySQL, Apache, PHP, BerkeleyDB, perl, and other open-source and GNU projects, however, I can take a new Intel-based PC with a 20 or 30 Gb hard drive and within maybe two hours have the equivalent of a $10,000 to $50,000 set of licenses from Microsoft, Oracle, or other database/enterprise server software makers.
For many businesses, perhaps for most municipalities, anything outside core complexities, like managing pensions, budgets, and employee salaries, deploying an open-source alternative is much cheaper and easier, and free tech support is available from peers online, plus free adaptations to code and other benefits. If I’m building a database of 10,000 vendors that consumers could consider to hire as licensed sewer repair contractors, I need robustness, of course, and I have to build an interface to search and update it. But those interfaces already exist in many iterations on Sourceforge and elsewhere.
I wouldn’t push open-source products for business software replacement yet for governments. But on the back-end, it’s a way to shave a lot of licensing fees without increasing the necessary expertise or expense.
The New York Times has dropped free links into its archives, ____ _____ reported. This is bad news. The Times has been charging for searching and retrieving articles from its archives for some time, but seemingly always allowed direct links to persist to the free retrieval of an old article.
From looking at articles I’ve written for the Times, I can see that articles before February 2001 still work. In articles I wrote starting in November 2001, the links turn into abstracts with a link to pay $2.95 per article.
I’ve experienced many link breakages before, but never this wholesale pullback.
I’m working on a review of the new Apple 12-inch aluminum PowerBook G4s. The unit Apple sent me to review warped slightly on the top right front after a few weeks of using it. It’s clearly a warp, not a hinge problem.
A colleague pointed me to archives at Macintouch in which readers report their first-person experiences with either units they’ve had that have warped on the top and/or bottom, and walking into Apple Stores with obviously well-used 12-inch models that show warping.
Here’s the problem. I don’t want to report that the PowerBook warps. That’s unfair. I have about 10 accounts and my personal experience. That’s anecdotal. As a journalist, I can’t make an assumption about hardware based on that little information. It’s unlikely that unless Apple planned a recall and case replacement or other substantial move, or they acknowledged that a specific run of cases had a flaw, that there will be a way to gather comprehensive information on a statistical level.
So I ask the community: anyone have any ideas about either confirming or rejecting that warping is widescale? anyone have suggestions on how to cover this?
Of course, I may chart my own course, but it seems irresponsible for me to not report my first-person experience. Apple is sending me a replacement unit to look at. We’ll see what it does.
The deal between Amazon.com and Google to place Google search results on Amazon.com’s pages should be interesting — for me! It’s all about me, as you know.
Of course, there’s no way of knowing how Google will customize results for Amazon.com: will it be search only, or paid search listings, or “remove anything with an ISBN in it” results? It sounds like a combination of the first two.
I’d understand if Amazon.com didn’t want people leaving their site to my site where I’d direct them back: that costs them commissions that they shouldn’t be paying to me or anyone for that sort of quick redirect even if a couple of pages sit in between.
Worse, I might refer them to Half.com or Alibris or another bookstore offering used books at lower prices than Amazon.com Marketplace.
About ten days ago, I wrote about a bandwidth disaster that my records and the hosting provider’s information indicated could cost thousands of dollars — possibly as much as $15,000.
I got the good news today: it’ll cost me nada. Nothing. Zip. You heard me! Hurray!
The reason: the provider drops the busiest hours of the month to the 95th percentile; this eliminates any weird spikes and doesn’t penalize you for limited craziness. Because we halted the downloads soon enough, even though we served out 250 Gb of data over 36 hours and satured a 100 Mbps Internet pipe for nearly two hours, our 95th percentile utilization was only 0.2 Mbps.
I’ve been holding my breath for 10 days.
Meanwhile, kind souls donated nearly $1,800 to help defray the bill. I’d already promised to donate the remainder, if any, to Project Gutenberg. I’ve now emailed all donors giving them the chance to back out, since the bill is zero; I’ll be adding some of my own money to that donation as gratitude for the Internet’s generosity.
I’m also writing a couple of articles about the panic attack and some ideas for better coping with excessive limited bandwidth needs.
October 2011 | August 2011 | June 2011 | May 2011 | February 2011 | December 2010 | November 2010 | October 2010 | September 2010 | August 2010 | July 2010 | June 2010 | May 2010 | April 2010 | January 2010 | December 2009 | November 2009 | October 2009 | September 2009 | August 2009 | July 2009 | May 2009 | April 2009 | March 2009 | February 2009 | January 2009 | December 2008 | November 2008 | October 2008 | September 2008 | August 2008 | July 2008 | June 2008 | May 2008 | April 2008 | March 2008 | February 2008 | January 2008 | December 2007 | November 2007 | October 2007 | September 2007 | August 2007 | July 2007 | June 2007 | May 2007 | April 2007 | March 2007 | February 2007 | January 2007 | December 2006 | November 2006 | October 2006 | September 2006 | August 2006 | July 2006 | June 2006 | May 2006 | April 2006 | March 2006 | February 2006 | January 2006 | December 2005 | November 2005 | October 2005 | September 2005 | August 2005 | July 2005 | June 2005 | May 2005 | April 2005 | March 2005 | February 2005 | January 2005 | December 2004 | November 2004 | October 2004 | September 2004 | August 2004 | July 2004 | June 2004 | May 2004 | April 2004 | March 2004 | February 2004 | January 2004 | December 2003 | November 2003 | October 2003 | September 2003 | August 2003 | July 2003 | June 2003 | May 2003 | April 2003 | March 2003 | February 2003 | January 2003 | December 2002 | November 2002 | October 2002 | September 2002 | August 2002 | July 2002 | June 2002 | May 2002 | April 2002 | March 2002 | February 2002 | January 2002 | December 2001 | November 2001 | October 2001 |