Over the last half-decade, data marketplaces have developed with extraordinary speed. They are incredibly valuable and their products drive many other technological verticals from AI to advertising to product development and marketing. 

Taken as a whole, the data economy is also a very fractured ecosystem. According to the EU’s own figures there are some 290,000 suppliers of data in Europe and the UK generating some €84bn in value between them. Despite all of this frenetic activity, little is known about data buyers themselves. What is on the minds of the people who are buying data and generating all this demand? How do they operate? And how are they responding to the challenges of evolving regulation and an ever more complex landscape? These questions are even more pressing because third party cookies, a big pipeline supply of information, are being terminated by Google at the end of next year.

Eager to find answers in this often opaque and even secretive economy, Wilsome recently led a research project, funded and commissioned by the ethical data startup Pool, to talk with ten senior data professionals from across a range of industries including advertising, finance (consumer and hedge funds), media and analytics. 

We asked them how they buy, source or commission data to discover more about their needs and challenges around sourcing, procuring, validating, and analysing data. Speaking to them in 1-1 interview format each lasting 45-75 minutes, here’s what we found. 

Businesses have many legitimate needs for data and are using data-led insight to develop increasingly sophisticated and smart ways to anticipate customer preferences, be competitive and operate efficiently. Inevitably this leads to a diverse set of requirements for sources and forms of data. The use cases are as numerous and varied as human interactions and behaviours.

However in this research we heard common themes about foundational needs in terms of data procurement. Everyone we spoke to articulated a simple set of basic requirements. They wanted data that is clean, compliant, easy to access, manage and manipulate, and in the right form to be successfully connected with first party data, which is increasing in importance for all.

There’s a mixed need for behavioural and attitudinal data - where buyers perceive both pros and cons when it comes to collected (e.g. analytics) vs reported (e.g. survey) data that most are wrestling with.

Perhaps surprisingly, given the straightforward nature of these needs, finding this data is the biggest challenge described by participants in this research. The professionals we spoke to describe a data landscape that is complicated, where the right data in a workable format is hard to come by. When it is found, it can be expensive. 

“Finding out where the data is can be a huge challenge for some people. And in making sure they're bringing in the right thing, because there's a lot of snake oil out there.” 

“There's more data available than we can possibly afford to buy, the cost of data is too high. Because the price that the data vendors want to charge for it does not reflect the business value.” 

There is also a lack of transparency in the data landscape. This secrecy - ostensibly to protect commercial advantage - means it’s hard to know what data is available and who it came from. We heard about a “spaghetti junction” of data relationships. Many organisations are data buyers, data intermediaries and data partners with each other. Data is being shared, processed and shared again in a complex web of commercial relationships and partnership deals. But overall, the body of suppliers we heard about was remarkably limited.

“Just finding out who's got data?  What are my options? What are my opportunities for using the data? You do tend to default to the same old providers, because that's who you know.” 

We heard in detail about the efforts being made to ensure that data sets are legally compliant and come from individuals who have given their knowing consent. But this can come at a high price and leads to limited choice in data suppliers, and a lack of confidence in data gathering practices for some data sets.

Nobody we spoke to wants things to be this way. Many people voiced ethical questions relating to the way that data currently flows through the digital landscape and businesses and the potential for ever more intrusive practices in the private lives of individuals. However there was a sense that it is hard to see what alternatives are available to businesses and data professionals.

“I'm not sad that they've been locking down the privacy there, because I think that's important. It’s always kind of weird to tell people what I do for a living, and they ask, “so you have data on me?” and I think, yes, probably more than you'd like to know.” 

The “elephant in the room” is that the biggest data pools are held by businesses such as Google and Facebook who “mark their own homework”, as one participant put it, when it comes to data validity, quality and individual privacy rights. Though some might say these businesses are the primary “winners” in the existing data paradigm, some of the professionals we spoke to pointed out that both Facebook and Google are starting to see significant damage to their brand reputations. This isn’t just with consumers and individuals. We heard that both organisations are being tainted by professional concerns about poor ethics and harmful business practices in relation to privacy and how data is being used and shared.

“If data buyers were provided a way of getting better data on people that didn't hit all those red flags of complying with legislation, then I think they would snap it up. I do think there is a market for that.” 

The research points to a real opportunity for ethical, transparent data suppliers who can shorten the buying cycle and offer flexible, user-friendly data products. The need and the commercial appetite are clearly there.

Wilsome offers user research and content strategy to people who are making their own path. Find out more here.