Hedge funds will spend $2 billion on web-scraping software to gain an edge, and it’s part of an investing gold rush

Hedge funds will spend $2 billion on web-scraping software to gain an edge, and it’s part of an investing gold rush

11th February 20190Byadmin

Despite information pouring in from billions of websites, poor performance plagued the hedge fund industry in 2018 — pushing investment managers to increase their already-massive web scraping programs.

One out every 20 web page visits last year was done by a hedge fund or sell-side research institution scraping websites for information, according to a new report by Opimas Analysis. This comes out to roughly 10.2 billion page visits a day, equal to the daily users of Google’s search function, and expected to “grow rapidly.”

By 2020, managers’ web page visits for the purpose of scraping, or extracting information from a website using an automated software program, will eclipse 17 billion and cost more than $1.8 billion — nearly double what it currently costs — as managers invest in software, talent and outside vendors to clean and store the loads of data.

While the alternative data scene is exploding with new providers offering obscure info, research firm Opimas calls the web “the ultimate dataset.”

Read full story...