Here’s how two European search upstarts aim to go after Google—from the index up

Google’s dominance of web search—consistently above 85% of the market in the U.S.—is an unmissable fact of online life. But a key source of that power is much less obvious: the exhaustive index of web pages that Google has created and maintains, and which is so comprehensive that even alternative search engines often have to rent it out.

Now two of Google’s smaller rivals are launching their own indexing project, and the circumstances of their effort may give them better odds at success than past such attempts.

European Search Perspective (ESP), a joint project announced earlier this month by Berlin-based Ecosia and Paris-based Qwant, will start outside Google’s backyard by indexing French- and German-language pages so that those two search engines don’t have to pay as often to use Google’s index or the one Microsoft maintains for its Bing search site. Those expenses are nontrivial and can come with usage restrictions. 

The timing for European Search Perspective benefits from a lineup of favorable factors: a combination of server costs’ dramatic drop, growing demand for web indexes from generative-AI projects, and Microsoft’s increase in access fees to its Bing’s index.

“A little bit of pressure, a little bit of opportunity,” says Ecosia CEO Christian Kroll. 

Ecosia itself switched from Bing to Google data after Microsoft announced those rate hikes, but Kroll says the search-costs wolf is now a little farther from the door thanks to the interest of EU regulators in a possible violation of the EU’s sweeping Digital Markets Act. 

“I think regulators understood our situation and made it clear that they would potentially support us,” he says.

The initial goal with ESP is not to zero out those Microsoft and Google bills but to reduce them, while also earning money from AI developers and other tech firms in need of search data. 

“We pay per query, and I hope that in a year from now a substantial amount of our queries are answered by our own index,” Kroll says. “It’s quite feasible to get to half of the queries answered.”

But even growth beyond that would still leave Ecosia and Qwant beholden to the top two indexes: “For the last 10% to 20%, we still need access to Google and Bing,” Kroll adds. 

Indexing English-language pages should follow (Ecosia gets about 8% of its traffic from the U.S.). But Kroll sounds doubtful about how soon the project will be able to build useful indexes of pages in less-common languages such as Portuguese, saying it won’t have enough user-response data to help train its index.

More pressingly, ESP seems likely to run into the same problem that other would-be Google rivals have complained about: sites that welcome only Google, Bing and maybe one or two other crawlers to index their content, leaving startup web indexes to decide if they should ignore these standardized instructions and risk getting blocked. 

“Not easy,” Kroll says, without elaborating on how ESP might solve it beyond suggesting that “this is maybe something that needs to be solved by regulation.”

(Kroll is not a fan of all of the EU’s regulatory initiatives. He would like to see European regulators rethink “link tax” laws requiring search sites to pay news sites for showing brief snippets of text below search-result links, which he says sock his company with minimum payments that leave a deeper dent than those paid by Google, the intended target of these rules.) 

Depending on two giant American companies to index the web is a problem for Europe in general, Kroll says—especially with President-elect Donald Trump’s incoming nationalist agenda, which he fears could lead to EU firms shoved aside in tech- and trade-policy discussions.

The inventor of the web agrees that having so little competition for indexing the web is problematic.

“When you have a very strong monopoly, it’s not good for innovation,” Tim Berners-Lee says. He allows that a search-index monoculture may not be a problem solvable by market forces alone.

“You can make the argument that there are some natural monopolies,” he says. “Some people in Europe might suggest that search should be a common good.”

But Google’s defeat in the search antitrust case brought by the U.S. government and every state but Alabama opens up a novel possibility on the left side of the Atlantic. 

While the Department of Justice’s proposed remedy has drawn attention for its requirement that Google sell off its Chrome browser and exit preferential deals with Apple and other third parties to keep its search engine the default, Section VI of the plaintiffs’ proposed final judgment could shape search competition much more directly.

That would require Google to offer its search index “at marginal cost” to competitors with the same latency and reliability as its own search site—plus free access to anonymized “User-side Data” revealing how people respond to search results. 

A separate provision would strike out Google side deals with publishers that give it exclusive or preferred access to their content, such as the agreement Reddit inked with Google in March. 

(The EU Digital Markets Act already requires “gatekeeper” companies to provide “fair, reasonable and non-discriminatory terms to ranking, query, click, and view data,” but competitors such as DuckDuckGo have objected to the limits Google has placed on that sharing.)

Kroll isn’t commenting yet on how well the proposed U.S. regime might address his concerns. 

But his take before the publication of the proposed final judgment speaks to how little the market for search can look like how it did when Google was the plucky startup taking on entrenched competitors. 

“I think the technological challenge can be overcome. I think there’s even enough money in there,” he says. “But we need the right regulations for things to work out.”

https://www.fastcompany.com/91237704/heres-how-two-european-search-upstarts-aim-to-go-after-google-from-the-index-up?partner=rss&utm_source=rss&utm_medium=feed&utm_campaign=rss+fastcompany&utm_content=rss