Open data is dead
For open data to remain a viable option, it must create value
I was recently browsing APIs for a project, and noticed how restrictive they’ve become. Is the promise of open data dead?
Reddit API key takes several weeks for approval
OpenTable API cites 3-4 weeks for approval
Twitter API costs $100/month for reading 10,000 tweets
Yelp API limits to 500 API requests/day, with only partial text from user reviews
Open data began with a promise
When you have a popular platform, making data open and easy to access incentivizes additional applications to be built on top of your platform. Think of the early days of Reddit or Twitter and how vital their APIs were to building their communities. Twitter had an especially large ecosystem of apps built on top of their API. Reddit also enjoyed a healthy ecosystem of readers, search apps, bots, and more.
A healthy app ecosystem around your product means more users, more traffic, and a larger business, until..
More pressure to monetize
The open data party started to wind down as platforms faced increased pressure to monetize. Third party apps which didn’t help with monetization found their access threatened, especially since the users were already there.
Thanks for getting us to 1 billion users!
Profitable businesses needed to be built and growth was slowing down so giving away the core component of your platform (your data!) was no longer an attractive practice. This was the first phase of the death of open data.
LLMs and the next phase of locking down data
The nail in the coffin for open data was LLMs, especially as OpenAI created a large business based on open data from other platforms. Unlike opening your site up to Google via sitemaps and using the increased search engine traffic to grow your business, LLMs take your data with no attribution to you and build their own healthy business doing it. LLMs spell the end of open data.
Open Data needs to create value to work
For open data to remain a viable option, it must create value to either a group or a business. Without business support, groups must rely on donations (e.g. Wikipedia) to pay for hosting and infrastructure for open data.
Business support for open data could mean a product catalog, since it will drive more users to purchase their products, even if integrated into third-party applications. Open data can be made available to select contributors to look for errors or help with validation, and used to strengthen business-critical data. If you own a telecommunications company and support open data for users to report cell phone service problems by location, this data will be valuable in improving your networks.
There’s hope for the future
I’m confident there will be tools in the future to support open data, as well as business cases that justify the costs. The next generation of applications shouldn’t require paid APIs to show interesting data, and I’m hopeful we’ll find the middle ground between truly open data and protecting the interests of a business.
At Sort, we’re building a supercharged database collaboration platform for Postgres and Snowflake. Learn more and sign up to get early access at sort.xyz today.