Acceptance Testing with Adaptive’s Accelerators

At Adaptive we believe that QA and Testing are at the core of any agile delivery and your testing approach cannot be truly agile without test automation. So when we invested in our accelerator technology, we realised that an acceptance testing solution must be a part of it and it must be done right.

What we needed was a test framework which:

  • Promotes collaboration within the team
  • Makes it easy to concentrate on testing business features the same way our accelerators allow us to concentrate on writing business code
  • Helps us write fast and reliable tests

In this blog post, I would like to share some of the most exciting features of the framework that allow us to achieve these goals.

But first, some technical information.

Our framework is written in Java and is based on Cucumber to make collaboration between QA, DEV and BA easier, and to decrease the risk of getting complex business functionality wrong.

Now let’s talk about the test framework features.

Test Execution Mode

Test Execution Mode is one of the most powerful features for me as a QA! You can use the framework for any kind of test: component or integration, but you get the best value when using it for end-to-end (e2e) tests, which are notoriously most useful from the business perspective and at the same time least performant and least stable from the execution perspective.

But what if we could make e2e tests really fast and stable? Now we can!

The framework allows you to run tests in 3 modes:

  • Existing - this is the usual way that e2e tests are executed in the industry - against a previously deployed environment.
  • Embedded real - in this mode the tests and the services under test run on separate threads within the same JVM, communicating using real protocols, like HTTP, FIX, and Aeron.
  • Embedded direct - this is the e2e supermode, where the tests and the services not only run in the same JVM, but are wired together directly, bypassing the underlying network protocols, and running in a single thread. This makes the tests completely consistent, and lightning fast, allowing us to execute 107 scenarios with 1182 steps tests in just 44 seconds on one of our projects.

Now the usual test pyramid can be extended and e2e layer refined:

We only need to run a subset of e2e tests against a deployed environment - those that could actually be affected by production-like wiring and topology.

One of the reasons that e2e tests have a bad reputation is their reliability. It is often not just due to asynchronous behavior of the system which the test execution mode deals with, but also the stability of the environment: does it have all the pieces that I need? Does it have all the test data that I need?

In the embedded real and direct modes, the test framework spins up a new “environment” for you containing just the pieces necessary for the tests. You can inject the test data at the start of each test case saving us from the “mystery guest” antipattern.

Here is how it looks in a feature file:

Scenario: CreateAsset
Given the services 'engine, admin-gateway'
And admin session Foo successfully logs in with username 'lidia'
When admin session Foo successfully creates asset 'USD' 'US Dollar'
And admin session Foo successfully creates asset 'GBP' 'British Pound'
Then admin session Foo can see assets:
| Code | Name |
| USD | US Dollar |
| GBP | British Pound |

Test clients and utils out of the box

One of the key features of our technical accelerators is code generation, and we leverage it for the test automation solution as well. When you start a project, you get generated test clients for each gateway (like FIX or Web) out of the box covering all the API calls.

For example, if we look at the step from the scenario above:

Then admin session Foo can see assets:
| Code | Name |
| USD | US Dollar |
| GBP | British Pound |

The implementation that DEV or QA created looks like:

@Then("admin session {adminSession} can see assets:")
public void awaitAsset(final AdminGatewaySessionDriver session,
final DataTable expectedAssets)
{
final AssetServiceProxy assetServiceProxy = session.services().getAssetServiceProxy();
final AssetServiceClientRecorder assetRecorder = session.services().getAssetServiceClientRecorder();

final long correlationId = assetServiceProxy.generateCorrelationId();
assetServiceProxy.getAllAssets(correlationId);

deployment.expect(assetRecorder.getAllAssetsResponse())
.withCorrelationId(correlationId)
.toHaveCompleted()
.toContainExactly(expectedAssets);
}

Every line in this step implementation code includes either a call to generated code; for example:

session.services().getAssetServiceClientRecorder();

or a platform util; for example:

deployment.expect()

The deployment.expect() accepts both objects and datatables for verification and, in the latter case, the datatable transformation and comparison is happening under the hood.

FIX protocol support

A full-featured FIX gateway is part of our accelerator set, so it’s only natural that its test automation framework allows you to send and verify FIX messages.

Over the years I have worked with numerous solutions for FIX testing, the protocol versions and messages vary a lot between our clients so we had to find a generic solution that would fit if not all, then most cases. The easiest approach was to expose FIX message at the gherkin layer, here is an example test:

Scenario: FIX session subscribes to market data on connect and sends a snapshot request
Given the services 'engine, market-data-gateway, ref-data-injector'
When the market data provider session 'SPOT-FX' connects
Then the market data provider session 'SPOT-FX' receives a message containing:
"""
[MsgType] 35 = V [MARKET_DATA_REQUEST]
[MDReqID] 262 = CAPTURE_AS: MD_REQ_ID_1
[SubscriptionRequestType] 263 = 0 [SNAPSHOT]
[MarketDepth] 264 = 1
[NoMDEntryTypes] 267 = 2
[MDEntryType] 269 = 0 [BID]
[MDEntryType] 269 = 1 [OFFER]
[NoRelatedSym] 146 = 1
[Symbol] 55 = GBP/USD
"""

This approach makes it exceptionally easy for the team to start using the test framework for any specific FIX setup on a project and then later on they can adjust it if needed.

Time Travel

The last but not least feature that I would like to share in this article is ability to time travel (again, another case where e2e tests can be really cumbersome). This feature is, of course, only supported in embedded modes.

Here is how it looks at the gherkin layer:

Scenario: Scheduled Job
Given the services 'engine, admin-gateway'
Given admin session Foo successfully logs in with username 'lidia'
When admin session Foo successfully creates a reminder in 60 seconds saying 'Hi there'
And time advances 1 minute
Then admin session Foo sees these reminders:
| Type | Body |
| Reminder | Hi there |

And here is the implementation for the “time advances” step:

@Given("time advances {int} {timeUnit}")
public void timeAdvances(final int seconds, final TimeUnit timeUnit)
{
   timeMachine.advanceBy(timeUnit.toMillis(seconds));
}

The time machine is another util supplied by the framework that allows to manipulate the cluster clock and avoid waiting in this kind of tests.

Summary

When our clients choose an accelerated solution delivery from Adaptive, they not only get tools that support development but also tools that support QA (and UX and DevOPS). This allows us to not just write code faster but to do so with the confidence that business requirements are met and any risk of regression is minimized.

What’s next

Our test automation solution is already quite mature and helping us to significantly speed up delivery for several of our clients, but there are always more things to add:
Web UI test automation.

Further improvement of platform utils to streamline expectations and assertions.

If you would like to know more about it, please don’t hesitate to reach out to me on LinkedIn or click the button below.

 

Lidia Sinitsyna

Head of QA,
Adaptive Financial Consulting Ltd

×

Contact us

    By pressing "Send" I agree that I am happy to be contacted by Adaptive Financial Consulting. I can unsubscribe at any time. You can read more about our privacy policy here.