Hydra Platform, removing complexity from bespoke solutions

Over the past 9 months, the Hydra team have been working on iterations of Hydra Platform in collaboration with both internal delivery teams and client development teams that have started using the Hydra Platform in 2020.

In this blog post, we are going to explain what has been happening in Hydra Platform, what we are planning to do next and how these changes benefit our clients.

What is Hydra Platform

Hydra Platform is one of our accelerators that takes the most complex elements of real-time trading and makes them simple to develop, deploy and operate. It is a core platform and framework of components that we use as a delivery accelerator to build differentiating products and business logic for our clients to stand out from the competition.

Hydra Platform allows our engineering teams and our client development teams to focus on building a differentiating product by providing a platform accelerator that takes care of the non-differentiating, architectural and fault-tolerant aspects of low latency trading platform development.

The platform is made up of a fault-tolerant, event-sourced, clustered transaction processing engine, highly available and fully scalable FIX, Web and messaging gateways complemented with a SQL database for quick and easy retrieval of historical data and a complete disaster recovery solution.

Out of the box, Hydra Platform comes with an array of non-functional, architectural and fault-tolerant features implemented and ready to be deployed, including:

• Connectivity between the clustered engine and the various components and gateways of the platform.
• Clustered failover of the core transaction processing engine.
• Message persistence & sequencing; allowing graceful recovery of a gateway or component in the event of connection failure.
• Very low latency business logic processing using allocation free code to avoid processing jitter caused by JVM garbage collection. Yielding systems capable of over 100k transactions per second and sub 40 microsecond response times.
• Low latency logging, auditing, metrics monitoring and snapshotting via code generated off-heap repositories.
• System health and latency monitoring.

The platform also comes with:

• A complete suite of developer tooling and operational tooling to accelerate development, automate the operation of the platform and troubleshoot the application.
• A suite of test frameworks for functional, non-functional & performance testing to ensure the solution continuously meets standards set for functionality, speed and reliability.

Adaptive has accelerated the build and delivery of multiple full-stack, low latency, trading systems on top of Hydra Platform; ranging from:

• Fixed Income RFQ platforms,
• Enterprise-grade venues for crypto and energy exchanges,
• Retail brokerage trading platforms,
• Corporate bond marketplaces, and
• Buy-side OMS & compliance systems.

Our low latency cloud based trading environments have also won awards in Europe and North America for the best cloud-based trading environment.

Accelerated delivery

Increased efficiency

Reduce the delivery risk

Higher quality

Control and flexibility

Accelerated delivery

We want to increase the product delivery efficiency of our clients by providing a platform that takes care of the non-differentiating, architectural and fault-tolerant aspects of low latency trading platform development. This, combined with our small and highly experienced engineering and product delivery teams, allows our clients to focus on accelerating the development of the differentiating aspects of their trading platform while Hydra Platform looks after the underlying, non-differentiating aspects.

Increased efficiency

Increasing product delivery efficiency translates into:

• Reduced time to market: the acceleration of Hydra Platform combined with an experienced team delivers projects 40-50% quicker than standard bespoke builds. This significantly improves time to market and agility in responding to market demands.
• Reduced capital investment: by reducing the delivery timeline we reduce the capital investment required to build the system.

Reduce the delivery risk

We want to reduce the delivery risk on the project as much as possible. In our experience building trading systems that are resilient, efficient and functionally correct is hard because it requires a wide range of engineering skills that are hard to acquire and retain within a team. Hydra Platform covers the non-functional requirements of a trading system (the most challenging engineering part of the trading system) so the Adaptive delivery team can focus on delivering business value only.

Higher quality

We want to deliver trading systems of a higher quality than would be possible with a bespoke build. Our continuous investment in Hydra Platform allows us to provide trading systems that are fault-tolerant and highly efficient out of the box. We also provide a large range of tooling such as an end-to-end testing framework, performance and resilience testing frameworks which are generally unseen on bespoke projects because they would just cost too much to engineer for a single project. This greatly lowers the total cost of ownership of the platform itself through reduced operational and on-going development costs.

Control and flexibility

We want our clients to retain as much control and flexibility on the solutions we deliver for them. We think vendor products are not the solution: clients are locked into a product vendor and its roadmap. Instead what we do is build bespoke trading applications on top of Hydra Platform. Our client retains the IP of the application, they can embed their own developers in our delivery team (hybrid delivery team) and once the project is complete our client can take full ownership of the application and customise it further.

Hydra Platform Features

We are putting the finishing touches to the core features of Hydra 2.0, or Hydra Platform, which includes a higher-level domain-specific language (DSL) that allows Hydra platform to code-generate much more of the boiler-plate code required to stand up a multi-component and distributed trading system and presents developers with higher-level typed APIs.

This version has these components out of the box:

Clustered transaction processing engine

The clustered engine is a high performance and fault-tolerant container and is a core building block for the application logic of a trading system. It houses the trading functionality and business logic that manages order placement, pre- and post-trade risk, fees, trading permissions, market, product and counterparty models as well as order matching algorithms. Application developers author bespoke logic and host it in the clustered engine. The clustered engine provides out of the box connectivity to the various services of the platform, fault tolerance, message persistence and other key capabilities such as auditing, logging and metrics monitoring. Check out this blog post for more details.

Web Connectivity module

The web connectivity module is used to provide bi-directional connectivity to UIs and Web API clients. It is composed of a server-side part called the Web Gateway and a client-side library for UI connectivity. The Web Gateway is typically deployed in an internet security zone and scaled out horizontally to allow a large number of clients to connect to the trading system built on top of Hydra Platform. Application developers can host custom logic inside the Web Gateway, to apply application specific logic for inbound and outbound messages. Out of the box, the Web Gateway provides user authentication, permissioning, slow consumer management, rate-limiting, fault tolerance, latency monitoring and connectivity to the clustered engine and historical service.

Event Streaming Gateway

The event streaming gateway provides a convenient API to subscribe and consume events flowing through the trading system built on Hydra Platform. The event streaming gateway provides an at-least-once semantic and allows subscribers to replay messages, for instance in case of disconnection.

It can be used for many use cases, for instance to replicate data out of the trading system for data analytics such as business intelligence, market surveillance or to store market data produced by the trading system. Application developers can easily extend the event streaming gateway to expose application specific events and data.

FIX gateways

Application developers can expose any number of FIX gateways using Hydra Platform. For instance if the platform is used to implement an exchange (central limit order book), it is possible to expose an order management FIX gateway, a market data FIX Gateway and a drop copy gateway - other types of systems may require a different number of FIX gateways any configuration is possible.

FIX gateways are connected to the clustered engine and require minimal development to expose the trading system via FIX. The FIX gateway can be used as an acceptor or an initiator. Application developers can host custom logic into the FIX Gateway to handle inbound and outbound messages and out of the box it provides authentication, slow consumers management, per user rate limiting, fault tolerance, latency monitoring and connectivity to the clustered engine.

History Service

Most trading systems need to be able to provide a large amount of historical transaction data to meet a number of business needs ranging from customer transaction statements, regulatory reporting requirements to historical charting services.

To achieve this, the Hydra Platform History Service provides a toolkit to easily set-up and deploy a set of fault-tolerant database instances to allow reliable querying of large data sets off the latency path of the clustered trading system.

Disaster Recovery solution

This module provides disaster recovery capabilities to the platform. The module is deployed to a disaster recovery site and asynchronously replicates the state of the trading system. In case of failover to DR, the replicated state is used to seed the system in DR within tight recovery time and recovery point objectives. For Cloud based deployment the architecture of Hydra DR allows the DR environment to be provisioned on demand. That can reduce infrastructure costs by around 50%.

Operational tooling

The platform provides a set of tools for technical operators, allowing them to automate the operation of the platform and troubleshoot issues. The following capabilities are included as part of the core platform:

• Time Travel: the ability to replay any production transaction in a development environment as it happened for troubleshooting purposes.
• Monitoring Agents: to collect metrics and component health for all platform components in low latency settings.
• Snapshot Tooling: ability to export binary snapshots into a SQL database, modify and re-import.
• Backup and Restore: this can be used to backup an environment before a release or snapshot an environment to seed another.
• Poison Message Handling: tooling to investigate and fix the state of the system after an application error.
• Data Maintenance: archiving and truncation of old transaction logs and snapshots

Developer tooling

Along with the Operational tools above which application developers can use to debug applications build on Hydra Platform, we have also developed a suite of developer specific tools for application developers:

• Hydra Platform supports patch-based snapshot migration that simplifies changes to persistent structures in the clustered engine, especially when managing feature branches.
• Stream introspection tooling: this allows developers to see in human readable form what messages are passing through the system at a given point in time. This simplifies the debugging process of a system in development.
• Patch-based snapshot migration that simplifies changes to persistent structures in the clustered engine, especially when managing feature branches.
• IntelliJ plugin for syntax highlighting of Hydra Platform files.
• PMD Code analyzer for static code analysis that checks the application behaviour is deterministic, common implementation mistakes related to threading and concurrency and rules related to high performance optimizations to ensure the fastest processing possible.

E2E testing

The platform provides an advanced testing framework allowing application developers to author end to end tests of their application. The testing framework is able to start the full Hydra application (i.e. the clustered engine and all gateways and services) on a single server or laptop and run a full suite of end-to-end tests via any of the gateways in the system. See our blog post on Acceptance Testing framework for more information.

Perf testing

A key element of delivering low latency trading platforms is the measurement, profiling and benchmarking of the entire stack to ensure we provide our customers with outstanding performance. Applications built on Hydra Platform can achieve sub 40 microsecond response times and are capable of processing over 100k transactions per second.

Our performance testing strategy for the platform has two dimensions;

1. Write targeted benchmarks (using Java Microbenchmark Harness) on critical components (e.g. message codecs, which are executed for every message handled by the platform); and
2. Write larger end-to-end tests to exercise multiple components representative of a customer system. We use the end-to-end tests to enable system profiling, helping us to understand areas that would benefit from optimisation effort.

Both of these test elements are run continuously on machines in the Adaptive performance lab. Test results are recorded in a database, and regressions in throughput or response-time cause a build to fail. The out-of-the-box performance testing framework that comes with Hydra Platform allows customers to apply the lessons learned in Continuous Delivery (CD) pipelines to performance testing for rapid feedback on platform latency and throughput.

Hydra applications can be deployed on either bare metal or cloud infrastructure. In both scenarios, Kernel bypass is possible, delivering very low latency response times. For example, a low latency exchange venue developed in partnership with ErisX deployed on physical hardware equipped with Solarflare NIC, connected to an Arista 7150S switch and run as Docker containers using kernel bypass, was capable of delivering sub 100 microsecond response times at the 99th percentile for simple and multi-level order fills.

For further information please see our blog post on performance engineering.

Trading Platform Reference Implementation

The platform reference implementation that comes with Hydra Platform has examples of all of the above and can be used as a starting point for new projects. It is the reference project for Hydra Platform, and we’ll continue to improve it as we improve Hydra Platform.

What’s coming up

We are currently working on improvements to the SQL database “History Service” which makes it easy to drive data grids with Hydra Platform. It includes a component to listen to high throughput events from the cluster and persist them to a database and another query component, which can scale horizontally, query data and return it to the web gateway. These components are well-suited to drive data grids or blotters throughout the system and will reduce the development time for a blotter by between 60 - 70%. For systems where blotters are used heavily, this can represent a saving of around 10-12 weeks of development effort.

In addition, next on the list are:

• A Fixed Income RFQ workflow reference implementation to accelerate the development of custom Fixed Income RFQ workflows.
• Risk Accelerator: a module for implementing pre and post trade risk management strategies which can be configured, adapted and extended to meet business needs. We expect the Risk Accelerator to reduce development time for risk management modules by 40 - 50%.
• A Kafka integration gateway to enable integration with existing system components in our clients technology stacks that communicate over Kafka.
• Enabling support for active/passive gateways out of the box. This will be entirely configuration based and will require no code changes to turn on.
• Adding support for testing snapshots in the acceptance testing framework, giving confidence that all necessary state is captured, reducing the possibility of a class of errors.
• Improving our training materials and documentation to support developers building systems with Hydra Platform.

Feel free to contact us for a chat if you have any questions or needs you would like to discuss.

Sign up here for regular updates.

Get regular updates