Digital Assets Report

Newsletter

Like this article?

Sign up to our free newsletter

Everybody has a data problem – and that problem is evolving

Related Topics

Hedgeweek exclusive: Gary Collier (pictured), the CTO of Man Group’s front office tech platform, Man Alpha Technology, discusses the evolving challenge of extracting value from data, and the technology Man Group has created to help.

  • Many hedge funds are managing an increasing amount and complexity of data 
  • Data is always evolving, problems are always evolving – and firms’ tech must, too 
  • This is creating opportunities for technologists to disrupt the way firms operate

By Gary Collier
CTO, Man Alpha Technology


Ten years ago, when we first started work on ArcticDB, problem-solving in the data space was more focused on time series data. 

In practical terms, this meant that data frames – the primary method for organising data and a very natural way for data scientists to think – were very long and very narrow. Within Man Group, we had a growing need for something that could scale horizontally in a way that was a challenge for the third-party vendor solutions we were using at the time.

Initially, we tried to add more computer power to process in parallel. But the third-party technology we were using for time-series data had fundamental design limitations that meant there were bottlenecks in data transfer. We evaluated alternatives, but concluded that no vendor or open source product would give us the performance, scalability and flexibility we required. At that point, we decided to build the technology in-house.   

ArcticDB is the result – a Python-native DataFrame database and our response to the ever-increasing amount of data and complexity of front-office research. 

At Man Group, we’re often dealing with asset classes that encompass an enormous universe of individual instruments – hundreds of thousands, in some cases – and the ability to represent such a large universe, and its evolution over time, in a single prime dataset and make decisions around that data quickly is invaluable. This technology is now driving investment decisions across Man Group’s front office, including product innovation.

Of course, the challenge of extracting value from ever-increasing datasets is one shared by the entire investment management industry. The first public-facing version of ArcticDB was launched on GitHub in 2015. So far, it has had over a million downloads. 

The challenge facing funds 

Today, the data frames used by many investment managers look a bit like an Excel spreadsheet – just much, much bigger. For example, quantitative analysis of a contemporary corporate bond universe might require a data frame 500,000 columns wide and millions – even billions – rows deep.

But even that’s a simplistic view. Those investment firms running complex back tests may require a model that uses 1,000 or even 10,000 data frames, with data that changes in shape and size. Corporate bond or tick data, for example, can be sparsely populated, meaning the whole shape of the data is time varying. Veracity and provenance are other important characteristics. That is, it’s important to know how data looks at the point in time, at the point it was issued, and what changes were made and when, not just after it’s been corrected. 

There’s a finite set of truths about financial assets and the more relevant data you have and can process effectively, you can get incrementally closer to understanding what those fundamental truths are, which in turn helps you understand how a particular asset might behave in future.

You need technology that accounts for all of this.  

Where next? There are many exciting items on our immediate roadmap. We’re talking with firms across different industries and sectors – from data vendors to investment banks and technology companies – and everybody seems to have a data problem and is reimagining their data platform. 

Is it just coincidence that these data platform problems are emerging at the same time? If I’d asked each institution every year over the past 10 years, would they have said the same thing? Data is constantly evolving, so the problem is always evolving. That creates huge potential for technologists to go in and disrupt the way many firms, in many industries, operate. This is just the beginning. 

Fundamentally, every investment firm on the planet deals with data. If you’re in the business of dealing with problems, you’re in the business of dealing with data – and you need technology that can help your process that data and extract value as efficiently as possible.

Gary Collier is the CTO of Man Alpha Technology, the front office technology arm of Man Group, a global active investment manager. His insights feature in Hedgeweek’s April research report, Turning Point: The New Technologies Helping Hedge Funds Evolve.  

Like this article? Sign up to our free newsletter

Most Popular

Further Reading

Featured