Cloud and Open Source: Engines of Disruption
The sea change that many quants and data scientists are witnessing is that the processing power needed for high end computing tasks is now much more affordable and increasingly available via private cloud infrastructure and public cloud providers. This, combined with Open Source Software (OSS), has emboldened a new generation of entrepreneurs to challenge financial business as usual with only an algorithm, a business model and a dream.
The explosion of interest in OSS machine learning and big data platforms is also a clear indicator of the nature of the disruption to the old-world order in financial services. Open source platforms such as Apache Spark, TensorFlow and other machine learning platforms are attracting a growing number of developers who want to turn OSS big data innovations into new services. These OSS platforms are designed to scale to handle the huge data volumes that make machine learning and big data analytics come to life. The scale of these projects puts significant pressure on IT operations and DevOps to maximize the efficiencies and performance of their computing resources.
Use Case: Risk is Predictable
Risk continues to be of critical importance across financial services segments. While there are many forms of risk, the most common form of risk across all financial segments is surrounding cybercrime and fraud. There is also a post-financial crisis regulatory aspect of risk management that forces lenders to know precisely how much capital they need in reserve. Keep too much and you tie up capital unnecessarily, lowering profit. Keep too little and you run afoul of Basel III regulations.
Figure 2: Big Data in Risk Management
There is great promise in new big data and machine learning technologies to enable lenders to tap into an ever-deepening pool of new data to analyze all aspects of risk and fraud. Identifying risk and fraud can require huge data volumes and large compute clusters which are typical of modern big data systems.
Fraud detection is a classic example of predictive analytics at work. For data scientists, fraud can be determined precisely by building the right scoring model and associating the scoring model with actual business costs. These fraud models identify the rules of what constitutes the fraud, and then those models crunch through the relevant data sets to identify the cost of the fraud versus the cost to detect the fraud. Therefore, cluster performance of the big data platform is becoming increasingly critical.
Use Case: Algorithmic Trading and Analytics
While using advanced algorithms to make informed trades is not new, its widespread adoption and applicability to a broader profile of traders is noteworthy. Today, asset managers, fintech companies and even retail banks are looking to provide richer analytics, daily forecasts, market advisories and recommendations to both industry and consumers. Both disruptive startups and well established financial firms invest heavily on adding speed and sophistication to automated system trading. This is putting increased stress on cloud and big data infrastructure.