Big data accelerates the rush for the cloud

John Kain, AWS

There is one giant reason why asset managers have been accelerating their shift into the cloud in recent years: big data. The sheer scale of the data sets fund managers now use every day to shape their investment decisions is so large, that cloud-based data management and analytics solutions have become the most obvious choice.

John Kain (pictured), head of business and market development for banking and capital markets at Amazon Web Services (AWS) Financial Services, understands the technology needs of the industry better than almost anyone.

Since the industry first grasped the opportunity offered by cloud technology of virtually unlimited, scalable storage and compute that is available on demand, fund managers have been embracing cloud-based solutions in growing numbers.

“The primary trend driving adoption of cloud has been the increasing use of data, particularly for investment modelling and analytics,” says Kain, a Wall Street veteran who formerly worked for JP Morgan. 

“Not only have we seen the rapid growth in the volume of traditional financial data, whether it’s pricing or news, but also the availability and use of alternative datasets within the investment process.”

In the four years since he joined AWS, Kain says there has been a marked increase in both the volumes of data being placed in the cloud but also the sophistication of the cloud-based tools being applied by fund managers.

Machine learning

The rapid advancement of cloud technology means highly specialised services that were once the preserve of a handful of industry giants, with deep enough pockets to build their own AI and machine learning-based tools, are now being deployed by managers across the board.

“When I first joined AWS, we were already seeing the more quant-oriented funds taking advantage of the compute capacity of cloud to do backtesting and research against the various trading models that they were building,” he says.

“So, whenever their researchers had an interesting thesis that they wanted to model, they could immediately get the capacity to go do that – and they could do it at scale. And since with many of these financial models, the more compute you throw at it, the faster it goes, they could actually get their investment research back more quickly.” 

Attractive economics

Adding to the allure of a cloud-based solution is the lower cost. The extreme flexibility of the technology allows fund managers to access extreme computing power based on huge data sets as and when they need it – a far preferable system to the old days when fund managers stored most of their data on their own servers.

“It doesn’t matter whether you’re running 1000 servers for an hour, or one server for 1000 hours, the economics are the same in the cloud,” says Kain. 

“More importantly, they’re able to take advantage of some of the unused capacity. We have special pricing models to make that incredibly cost effective.”

Kain says that this approach, which was pioneered by the biggest quant funds, is now being adopted across the industry. 

“Many of the large quantitative firms have been running their research in the cloud for a while. What that’s done is create enough of a magnet for the industry that you’re increasingly seeing the availability of both traditional and alternative data sets within the cloud, for asset managers to leverage.”

That means that instead of having to build ingest pipelines for multiple vendors, and figure out how to pull data into their infrastructure, these data sets are now easily available within the cloud environment. 

The fact that so much data is now available on the cloud is allowing asset managers to take the obvious next step – automating more of the processes involved in running a fund to further streamline their operations.

Research environments

To effectively work with this growing volume of data, analysts need access to a secure and managed research environment which is connected to their data, and provides self-service access to the compute necessary to quickly perform their data sampling, preparation, and analytics. 

AWS recently announced FinSpace, a data management and analytics service that helps asset managers setup a research environment for their analysts so they can store, catalogue, prepare, and analyse financial industry data at scale in minutes. 

Kain said, “using FinSpace customers can reduce the time it takes for analysts to find, prepare, and analyse data from months to minutes.”

Natural language processing

Kain also offers the example of natural language processing tools which use machine learning to pick out themes or trends which can be incorporated into investment decisions.

The reduced costs and the ease with which asset managers can now access them are opening up a host of new opportunities formerly only available to the world’s very biggest firms.

He said these tools were, for example, giving portfolio managers “the ability to move from using standard growth measures for Chinese GDP, to instead look at things like satellite imagery, counting the number of ships entering and leaving a port, to website searches for holidays in Hong Kong to gauge consumer sentiment – and to find ways to get that data into the investment process.”

He continues: “In the past, you would have traditionally had to have your own machine learning capability as a firm and a dedicated team of data scientists to understand how to build that out.” 

These days, such advanced tools are increasingly available off the shelf from cloud providers including AWS, the world’s biggest cloud services provider owned by Amazon. 

For example, AWS offers SageMaker, which allows firms to analyse data sets using advanced machine learning techniques but without the need to develop bespoke technology.

Last year, AWS launched a service called Comprehend Events, which serves as a natural language processing service to extract details around real-world events from unstructured text.

“It can go through a news article to detect an acquisition; it can identify who the acquirer was as well as other parties mentioned in the story and pull that out in an automated fashion. In the past, that would have been a major investment for any firm to be able to get those things. 

So the ability to get that kind of insight from unstructured data has gotten much easier.”

High frequency holdouts

It’s true that a handful of niche areas in the fund management industry – such as high-frequency trading – are resistant to move everything into the cloud.

That’s because their strategy hinges on keeping the computers that make their trading decisions as close to the execution venue as legally possible – a critical capability for reducing latency and allowing them to execute trades before the competition.

Kain admits that it’s not easy to see an obvious alternative for these kinds of players that would work purely in the cloud.

“I think that’s an obstacle for the near future. When you’re measuring nanoseconds, it’s challenging to do that in a dynamic, on-demand environment.”

Nevertheless, this remains a niche area and more broadly, there is little doubt that asset managers are increasingly turning to cloud-based solutions. 


John Kain, Head of Banking & Capital Markets, AWS Financial Services
John Kain leads Worldwide Business & Market Development for Banking and Capital Markets at Amazon Web Services (AWS). He works with customers to help them transform their existing businesses and bring new, innovative solutions to market by leveraging AWS services. John has more than 20 years of experience developing solutions for financial institutions. Prior to joining AWS, he led key programmes for JP Morgan, Nasdaq, and two venture-backed financial technology companies.

Author Profile