Messaging Protocols

This is the third in the sequence of blog posts regarding some of experiences in creating Trading and eCommerce platforms at Adaptive. The previous post can be found here.

In this post we discuss Messaging Protocols that are commonly found in these platforms.

We have found that the vast majority of messaging interactions in Trading and eCommerce systems can be encompassed with a relatively small number of messaging protocols. We have defined these are as follows:

• Request Response
• Request Only
• Streaming
• Request Stream
• Publish Subscribe

The following sections discuss each of these protocols with a description of the protocol, an example use-case and a sequence diagram and pseudocode interfaces for how the protocol is realised on both Client and Server.

Request Response

Request Response is probably the best known and most intuitive interaction. It consists of the Client initiating a Request by sending a message with a payload and a correlation ID. The Server then receives the message, uses the sender's SessionID to resolve a Session, deserialises the payload and processes the request. It then responds to the Client by serialising the response and sends the message to the client stamped with the correlation ID. The Client receives the response message, checks the correlation ID and, if matched, deserialises the response payload and invokes the response callback. Should the response not be received within a given time period, the Client should call back the response handler with a timeout.

An example use-case for this protocol is to execute a Trade for a received Quote. The Request payload would include a Quote ID and the quantity to be executed and whether to buy or sell. The Response payload would indicate whether the request was successful and, if so, the resulting Trade ID.

The figure below shows a Sequence Diagram for the Request Response protocol. As the Middleware is a direct pass-through of messages, it is omitted from the diagram

The Client-side interfaces look something like this:

    interface RequestResponseClient<TRequest, TResponse> {
        // Initiate a request
        void request(TRequest request, ResponseHandler responseHandler);
    }

Where:

    interface ResponseHandler {
        // Successful response
        void onResponse(TResponse response);
        // Failed to get response
        void onError(Exception error);
    }

The Server-side interfaces look something like this:

    interface RequestResponseServer<TRequest, TResponse> {
        // Start processing requests
        Subscription respond(RequestResponseHandler<TRequest, TResponse> requestHandler);
    }

Where:

    interface RequestResponseHandler<TRequest, TResponse> {
        // Called back for each request
        void onRequest(Session session, TRequest request, ResponseHandler responseHandler);
    }

And:

    // Generic interface for cleaning up resources
    interface Subscription {
        // Clean up resource
        void unsubscribe();
    }

Request Only

The Request Only protocol consists of the Client initiating a Request by sending a message with a payload. The Server then receives the message, uses the sender's SessionID to resolve a Session, deserialises the payload and processes the request.

An example use-case for this protocol is for sending Log information back to the Server. In this case the Client doesn't care when the request is actually processed, it is sufficient to know that the message has hit the Middleware.

The figure below shows a Sequence Diagram for the Request Only protocol. As the Middleware is a direct pass-through of messages, it is omitted from the diagram.

The Client-side interfaces look something like this:

    interface RequestOnlyClient {
        // Initiate a request
        void request(TRequest request);
    }

The Server-side interfaces look something like this:

    interface RequestOnlyServer {
        // Process requests
        Subscription process(RequestOnlyHandler requestHandler);
    }

Where:

    interface RequestOnlyHandler {
        // Called back for each request
        void onRequest(Session session, TRequest request);
    }

Streaming

The Streaming protocol is initiated by the Server sending update messages to the Middleware. If a Client is interested in this stream, they subscribe to the Middleware and subsequent update messages are received, the payload is deserialised and the callback is raised.

This protocol can is useful for broadcasting data to a large number of Clients. A common use-case for this is the dissemination of System Updates.

The figure below shows a Sequence Diagram for the Streaming protocol.

The Client-side interfaces look something like this:

    interface StreamClient {
        // Consume updates
        Subscription consume(StreamHandler streamHandler);
    }

Where:

    interface StreamHandler {
        // Called back for each update
        void onUpdate(TUpdate update);
    }

The Server-side interfaces look something like this:

    interface StreamServer {
        // Produce updates
        Subscription produce(StreamHandler streamHandler);
    }

Request Stream

The Request Stream protocol is initiated by the Client by sending a request message with a payload and a Stream ID. The Server then receives the message, uses the sender's SessionID to resolve a Session, deserialises the payload and establishes the stream. The Server then responds to the Client with either an acknowledgement or an error. Subsequently, the Server can send any number of updates. Finally, it can terminate the stream with either a completion or an error. In all cases messages are stamped with the Stream ID. If the Client fails to receive an acknowledgement within a set time period, a timeout will be raised. At any time the Client can unsubscribe from the Stream by sending an unsubscription message.

An example use-case for this protocol is the classic Request For Quote (RFQ) exchange where a stream of Quotes is established for an individual client for a period of time.

The figure below shows a Sequence Diagram for the Request Stream protocol. As the Middleware is a direct pass-through of messages, it is omitted from the diagram.

The Client-side interfaces look something like this:

    interface RequestStreamClient<TRequest, TUpdate> {
        // Request a stream
        Subscription request(TRequest request, StreamHandler streamHandler);
    }

The Server-side interfaces look something like this:

    interface RequestStreamServer<TRequest, TUpdate> {
        // Stream requests
        Subscription stream(RequestStreamHandler<TRequest, TUpdate> requestHandler);
    }

Where:

    interface RequestStreamHandler<TRequest, TUpdate> {
        // Called back for each request
        void onRequest(Session session, TRequest request, StreamHandler streamHandler);
    }

Publish-Subscribe

The Publish Subscribe protocol is the most complex of the protocols. It is instigated by the Client sending a request message with a Subscription ID. The Server receives this message, resolves a Session based on the sender's Session ID and deserialises the payload. It then calls back into user code to check that the user is entitled for the request. If so, it will see if there is an existing stream for an equivalent request. To do this it must check requests for equality. If a matching stream is found, it will be reused and the associated reference count incremented. If not, a new stream will be established and the associated with a Stream ID. A subscription message is then sent to the Client with both Subscription and Stream IDs. The server then streams updates, completion and errors in the same way as the Request Stream protocol. If the Client unsubscribes, it does so by specifying the Subscription ID. The Server then decrements the reference count of the associated stream and, if zero, unsubscribes from the underlying stream.

A sample use-case for this protocol is for streaming Executable Prices to a large number of Clients where many users are interested in the same instruments. This protocol is very efficient in that the Server need only hold-open a relatively small number of underlying streams and leaves the Middleware to do the fan-out.

The figure below shows a Sequence Diagram for the Publish Subscribe protocol

The Client-side interfaces look something like this:

    interface PublishSubscribeClient<TRequest, TUpdate> {
        // Subscribe to stream
        Subscription subscribe(TRequest request, StreamHandler[TUpdate] streamHandler);
    }

The Server-side interfaces look something like this:

    interface PublishSubscribeServer<TRequest, TUpdate> {
        Subscription publish(PublishSubscribeHandler<TRequest, TUpdate> requestHandler);
    }

Where:

    interface PublishSubscribeHandler<TRequest, TUpdate> {
        bool isEntitled(Session session, TRequest request);
        bool areEquivalent(TRequest left, TRequest right);
        Subscription stream(StreamHandler streamHandler);
    }

Summary

In this post we have introduced a number of messaging protocols and explored how they work and sample use-cases. In our experience, where we have required more complex protocols, we have always been able to decompose them in terms of the lower-level protocols presented.

One thing to note about the interfaces for all of the protocols is that the APIs are completely asynchronous on both Client- and Server-side. This has a huge impact in terms of performance in that neither the component will block a thread waiting for a message to arrive. In the past where we have offered both synchronous and asynchronous variants of these APIs, developers have naturally elected to use the synchronous variants without necessarily understanding the performance issues as they are easier to consume. For this reason, we no long offer the synchronous variants.

The next post can be found here.