Is the jig up for honest search results?
This column appeared July 23, 2001, in the Online Journalism Review. Here’s the version on the OJR site.
Many of us in the new media industry have watched in despair during the past few months as several major search engines have abandoned all pretense at editorial integrity by adopting deceptive, misleading advertising practices at the expense of their users.
Finally, someone has stood up and said, Enough is enough. And now it’s time for the rest of us to join the battle as well.
Commercial Alert, a 3-year-old consumer organization in Portland, Ore., founded by Ralph Nader, filed a formal complaint with the Federal Trade Commission last week, charging that eight of the major search engines were “inserting advertisements in search engine results without clear and conspicuous disclosure that the ads are ads.”
Many search engines have gone to great lengths to fuzz the line between editorial and commercial listings.
To which I say: Bravo! But also: It’s not enough. Better that the search engines clean up their act on their own by bowing to their users’ wishes rather than bend to government coercion. See below for how you can make your voice heard.
Why should this matter to journalists, researchers and other Net denizens? Because search engines have become indispensable to our online existence as we look for ways to sensibly navigate the Web’s 2 billion pages and 14 billion links. Seven of the 10 most visited Web sites are search engines. And a February survey by the Pew Internet & American Life Project found that Internet users’ top two activities are e-mail and online searches.
The Net is becoming central to Americans’ everyday lives. A nationwide survey by the Markle Foundation released this month found that by a huge margin, the leading metaphor in the public’s mind for the Internet is “a library” — not “a shopping mall.” But an increasing number of search engines don’t seem to understand that.
‘Crass commercialism over editorial integrity’
“The problem has become acute lately,” says Gary Ruskin, Commercial Alert’s executive director. “Search engines have become essential to the quest for learning and knowledge in the Internet Age, and we don’t want such an important platform to be used to deceive the public and skew search results on behalf of hucksters. They’ve chosen crass commercialism over editorial integrity.”
No one, including Commercial Alert, begrudges the search engines their right to turn a profit and make an honest buck. It’s the honest part that’s at issue.
Here’s what has happened: As the dot-com meltdown and economic slowdown have combined to pummel online ad revenues, Internet companies, including search engines, have scrambled to come up with other ways to staunch the flow of red ink.
Search engines have increasingly turned to two significant revenue streams:
• Paid placement: In addition to the main editorial-driven search results, the search engines display a second – and sometimes third – listing that’s usually commercial in nature. The more you pay, the higher you’ll appear in the search results.
• Paid inclusion: An advertiser or content partner pays the search engine to spider its site and include the results in the main editorial listing. The result? A site is more likely to see its pages appear in a search result, but there are no guarantees.
There’s nothing intrinsically wrong with either of these business revenue streams. A user who enters “star wars” into a search engine may be doing research on the Strategic Defense Initiative – or she may want to buy plastic action figures.
The trouble is that many search engines have gone to great lengths to fuzz the line between editorial and commercial listings. “This is a breach of the editorial-advertising line,” Ruskin says. “We don’t oppose advertising, of course. We just support editorial integrity.”
Commercial Alert’s complaint names Lycos, HotBot, AltaVista, LookSmart, MSN.com, Netscape, iWon and Direct Hit, owned by Ask Jeeves. All eight display paid listings in addition to objective, algorithm-based results, but they call them “Featured Links” or “Partner Results” rather than disclosing that they’re paid by sponsors. In addition, MSN, Alta Vista and LookSmart accept payments for inclusion in their editorial results.
So, for example, LookSmart returns three listings: Featured Listings (paid listings), Directory Topics (spidered results) and Reviewed Web sites (including paid listings). Clear as a bell, no?
Notably missing from the list are Google, Yahoo and Excite, three of the industry’s leaders, which clearly are wearing the white hats in this contest.
Chaste almost to a fault, Google won’t display search results for corporate partners because it doesn’t want to show favoritism to its business clients.
Google, especially, has consistently staked out the high ground, telling Interactive Week: “We think if we’re the only ones out there with a reputation for integrity in our search results, that can only be good for us.” A second Google spokesperson told Searchenginewatch.com: “Our search engine results represent our editorial integrity, and we have no plans to alter our automated processes.”
Google publishes paid listings, but clearly marks them as “Sponsored Links.” And it has bucked the commercial tidal wave by not using paid inclusions. Chaste almost to a fault, Google won’t display search results for tens of thousands of pages that it has already crawled and indexed for internal use by corporate partners such as Cisco and MarthaStewart.com – because it doesn’t want to show favoritism to its business clients. A rare virtue in these times.
Fittingly, last Wednesday Google Inc. won the Webby Award for best practices, a new category that honors a single site that serves as a model of overall excellence.
An industry observer’s take
Danny Sullivan, editor of Searchenginewatch, has been one of the more seasoned observers of the portal business in recent years, keeping tabs on paid placements and paid inclusions by all the major search engines.
Sullivan, speaking by phone from Shrewton, England, is no anti-business zealot. He observes, smartly, that search engines are still underutilized by advertisers seeking highly targeted audiences. “It amazes me that there’s been so little written about this aspect of search engines,” he says. “There are very few places where people tell you exactly what they want, in real time. Banner ads are a poor tool to respond to those requests. The ability to interact with the users right within a search results area – that’s gold to the advertisers.”
And it should be gold to users, too – if it’s done on the up and up. “We’re seeing sites all over the Web displaying skyscraper ads and popup ads that are untargeted, intrusive, in-your-face,” Sullivan says. “We should be clamoring for more targeted commercial search results, because they just might be relevant to my needs. People need to understand that if you don’t support the sponsored areas, the alternative is an all-commercial zone.”
That’s the big fear that Sullivan often hears from webmasters: that search engines will eventually offer nothing but paid listings. “That’s not why most of us are on the Internet,” he says. “It’s more than just an online Yellow Pages.”
Writing a feature article on Aimee Mann? A business backgrounder on Dell? A medical story on Propecia? A travel piece on Maui? Get ready to wade through a torrent of commercial come-ons first when researching your story.
“This is like one day opening your newspaper and finding it filled with nothing but ads,” says Sullivan, who has covered the subject in essays such as Buying Your Way Into Search Engines, Can Portals Resist the Dark Side? and The Evolution of Paid Inclusion.
Both Sullivan and Ruskin point out that many of the smaller search engines engage in similar practices, or worse. Dogpile used to be a fairly useful search engine, but search the Web on Dogpile today and you’ll find page after page of paid commercial links (from GoTo, FindWhat, Sprinks, ePilot, ah-ha, BrainFox and Kanoodle) before it displays anything remotely relevant or objective. (Studies show that the vast majority of users don’t bother looking past the top 30 results.) Also falling from grace into the pitfires of hell is Disney’s Go, which once operated one of the slickest search engines on the Web in Infoseek but now merely coughs up results from GoTo’s commercial engine.
The empire strikes back
What’s the reaction of search engines to all this? LookSmart and Alta Vista issued statements dismissing the charges as groundless. A few weeks earlier, LookSmart’s CEO, Evan Thornley, told the San Francisco Chronicle his managers did not even raise the subject of ethics when the company adopted a pay-for-placement program last month. “We can’t afford to have ideological debates anymore,” he huffed.
Sarah Lefko, a product manager for MSN.com, added that surveys by Microsoft show that consumers already assume that all search results are for sale. (How conveniently self-serving – and what rubbish.)
Matt Stoever, a vice president at Lycos, told the New York Times: “We thought long and hard and decided it doesn’t matter if we are paid for a link, so long as the results are what the user wants. … The industry has trained users to avoid anything that looks commercial. By calling them paid listings, it hurts the user.”
Such is the arrogant, Alice-in-wonderland, upside-down world of Internet executives these days. Black is white, profit is all, and the entire Internet is filled with callow users who are either (a) so cynical that they assume all search results are bought and paid for, or (b) clueless droids who have somehow been goaded into ignoring wonderfully useful commercial listings.
Says Ruskin: “Their arguments aren’t credible. If what they’re claiming is true, why wouldn’t they be straight with the users and reveal that their ads are ads? The fact that they’re hiding and obfuscating that fact proves that it’s balderdash.”
Have your say
Ruskin says it’s too early to tell whether the FTC will move aggressively to act on the organization’s complaint, noting that chairman Timothy J. Muris took over the agency’s reins only last month. But he notes there are extensive precedents for the FTC to intervene. “For example, they’ve repeatedly told makers of infomercials that they have to disclose to the public that they’re paid ads rather than independent programming,” he says.
Personally, I’m not counting on the Bush appointees on the FTC to defend my interests as a consumer. I’m voting with my mouse clicks by doing my searches on Google and Yahoo.
But I also think the out-of-touch search engines could use some interaction with their audience. Ruskin agrees: “Yes, we’re hoping that users will switch to sites that have more editorial integrity, and that they’ll make their voices heard. It’s a very simple request: We don’t want trickery.”
Here’s the message I hope the search engines will hear and begin to heed:
We want our search results clean and unsullied. Display your paid listings, too, in a separate area, but be honest and upfront about them. Tell us your practices in a straightforward disclosure statement, and make it easy to find. Don’t deceive us, and don’t belittle us by saying we’re too shallow to care about editorial integrity.
If you’d like to give the search engines a piece of your mind, here’s where to contact them:
And while you’re at it, toss a little praise to Google, Yahoo and Excite as well. Editorial integrity needs all the friends it can find these days.
