New market data measure starter guide

Our latest Market Risk release has a new Market Data API.

In previous versions of Market Risk, the market data model was constrained to one single table with a fixed set of fields. With the release of Market Risk solution version 5.4.0, we’ve introduced a new Market Data API that allows the easy implementation of custom market data models. In this new market data measure starter guide, we will walk through how to use this new Market Data API.

Let us imagine that we have equity sensitivities defined for a date, instrument, market data set, expiration, and moneyness, and we want to retrieve the associated market data. To implement this use-case, we need to:

• Convert the moneyness value into a strike by adding 100 to it, multiplying the result with the spot price of the instrument, and dividing by 100 — for the example described in this article, a moneyness of 0 represents an at-the-money option.
• Retrieve equity volatilities by date, instrument, market data set, expiration, and strike.
• Perform linear interpolation along the strike axis if there is no data point for a pair of expiry and strike.

Create a new market data store

If we need to create a new store, here’s how we can do it. For example, let’s assume that we need to create a store that contains volatility market data with the following data model:

• Date
• Market Data Set
• Instrument Id
• Expiration
• Strike
• Quote

The resulting data file could look like the following:

AsOfDate,MarketDataSet,SurfaceId,Expiration,Strike,Quote
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,8.8,29.58502
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,17.6,31.3017
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,26.4,32.17177
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,35.2,22.60426
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,44,29.04278
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,52.8,20.62918
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,61.6,19.31745
2023–09–28,Official EOD,BMW_Implied volatility,0.5Y,70.4,27.66252
…

💡Note: For the sake of this blog and succinctness, we’ve shown a small sample set of generated data. For a real world use case, you may have many more lines of data.

Next, we can create the store in the following way:

public class EqVolMarketDataStore extends AConfigurableSchema {


   public static final String VOLATILITY_MARKET_DATA_STORE = "VolatilityMarketData";
   public static final String EXPIRATION = "Expiration";
   public static final String STRIKE = "Strike";


   @Override
   public void createStores() {
       configurator.addStore(volatilityMarketDataStoreDescription(GLOBAL));
   }


   public IStoreDescription volatilityMarketDataStoreDescription(String schema) {
       return configurator.storeBuilder(schema)
               .withStoreName(VOLATILITY_MARKET_DATA_STORE)
               .withField(AS_OF_DATE, STORE_DATE_FIELD_FORMAT, LocalDate.MIN).asKeyField()
               .withField(MARKET_DATA_SET).asKeyField()
               .withField(INSTRUMENT_ID).asKeyField()
               .withField(EXPIRATION).asKeyField()
               .withField(STRIKE, DOUBLE).asKeyField()
               .withField(QUOTE, DOUBLE, 0.0)
               .withDuplicateKeyHandler(DuplicateKeyHandlers.LOG_WITHIN_TRANSACTION)
               .updateOnlyIfDifferent()
               .build();
   }
}

The AConfigurableSchema abstract class allows us to use the Datastore Helper to customize our application datastore schema. We also add an input data file Spot_Market_Data_additional.csv that contains the following entry:

AsOfDate,MarketDataSet,InstrumentId,Quote
2023–09–28,Official EOD,BMW_Implied volatility,88.02

The content of that file will be loaded into the store containing Spot prices — we will need that data to convert the moneyness that comes from the sensitivity entry into a strike value.

Implement the Market data retriever

Once the store is defined, we’ll need to implement the logic of the market data retriever.

This is because the retriever allows us to query data in a store directly, so that we can use the retrieved data with our own logic in the market data post-processor.

As a result, there are three aspects that we need to consider here:

• Conversion from a moneyness value to a strike.
• A direct retrieval by key (date, instrument, market data set, strike, and expiration).
• If a direct retrieval cannot be done, linear interpolation must be done along the strike axis.

The direct retrieval by key is performed out of the box for us. When we implement our retriever, we just need to extend the SingleTableMarketDataRetriever class. If we do not find data for the exact retrieval, we need to implement interpolation logic. Let’s see how we can achieve that.

Out of the box, MR handles both linear and spline interpolation, which means that the desired interpolation mode can be chosen via the InterpolationMode class. In this example, we will use linear interpolation.

Therefore, to perform linear interpolation, we will need to retrieve the data along the strike axis for a given date, instrument, market data set, and expiration. As a result, we can write a datastore query and use it in a class extending SingleTableMarketDataRetriever:

public class EqVolTableMarketDataRetriever extends SingleTableMarketDataRetriever implements IEqVolMarketDataRetriever {

   protected final IPreparedListQuery eqVolCompiledQuery;

   public EqVolTableMarketDataRetriever(IDatabase database, String table) {
       super(database, table);
       eqVolCompiledQuery = database.getQueryManager()
               .listQuery()
               .forTable(table)
               .withCondition(and(
                       equal(FieldPath.of(AS_OF_DATE)).as(AS_OF_DATE),
                       equal(FieldPath.of(MARKET_DATA_SET)).as(MARKET_DATA_SET),
                       equal(FieldPath.of(INSTRUMENT_ID)).as(INSTRUMENT_ID),
                       equal(FieldPath.of(EXPIRATION)).as(EXPIRATION)))
               .withFieldsWithoutAlias(FieldPath.of(QUOTE), FieldPath.of(STRIKE))
               .compile();
   }

   public CurveMarketData<Double> getVolCurve(
           LocalDate asOfDate, String marketDataSet, String instrumentId, String expiration) {
       try (var cursor = database.getMasterHead()
               .getQueryRunner()
               .listQuery(eqVolCompiledQuery)
               .withParameters(Map.of(
                       AS_OF_DATE,
                       asOfDate,
                       MARKET_DATA_SET,
                       marketDataSet,
                       INSTRUMENT_ID,
                       instrumentId,
                       EXPIRATION,
                       expiration))
               .runCurrentThread()) {
           var strikeList = new ArrayList<Double>();
           var quotes = new ArrayList<Double>();
           while (cursor.next()) {
               var record = cursor.getRecord();
               var strike = (Double) record.read(1);
               strikeList.add(strike);
               quotes.add(record.readDouble(0));
           }
           if(quotes.isEmpty()) {
               return null;
           }
           return new CurveMarketData<>(strikeList.toArray(Double[]::new), quotes.toArray(Double[]::new));
       }
   }
}

Now that we have the code needed to retrieve a volatility value by key and by date, market data set, instrument id, and expiry, we will need to use that code to perform the retrieval.

Thus, we need to define:

• A coordinate translator
• A container

We can define both of these objects using classes that exist out of the box:

@Configuration
public class EqVolMarketDataRetrievalConfig {


   public static final String EQ_VOL_MARKET_DATA_RETRIEVER = "EquityVolMarketDataRetriever";


   @Bean
   public AConfigurableSchema eqVolMarketDataStore() {
       return new EqVolMarketDataStore();
   }


   @Bean
   public IDateRetriever eqVolDateRetriever(IDatabase database) {
       return new TableDateRetriever(database, VOLATILITY_MARKET_DATA_STORE);
   }


   @Bean
   public IContextualDateRetriever eqVolDateShiftingDateRetriever(IDatabase database) {
       return new DateShiftingDateRetriever(eqVolDateRetriever(database));
   }


   @Bean
   public IMarketDataCoordinateTranslator<List<Object>> eqVolDateShiftingCoordinateTranslator(IDatabase database) {
       return new DateShiftTranslator(database, VOLATILITY_MARKET_DATA_STORE, eqVolDateShiftingDateRetriever(database));
   }


   @Bean
   public IEqVolMarketDataRetriever volatilityRetriever(IDatabase database) {
       return new EqVolTableMarketDataRetriever(database, VOLATILITY_MARKET_DATA_STORE);
   }


   @Bean
   public IMarketDataRetrievalContainer<IDefaultMarketDataRetriever> eqVolMarketDataRetrievalContainer(IDatabase database) {
       return new MarketDataRetrievalContainer<>(
               EQ_VOL_MARKET_DATA_RETRIEVER,
               volatilityRetriever(database),
               eqVolDateShiftingCoordinateTranslator(database));
   }
}

Next, we need to implement the post-processor. The post-processor will make use of a ADefaultContextualMarketDataRetriever that will take care of the key translation for you. We will need to implement the executeRetrieval method. In this example, we convert the moneyness in the date shifted coordinates to a strike and attempt to retrieve the point on the volatility surface corresponding to the resulting coordinates.

// Get coordinates from list
LocalDate asOfDate = (LocalDate) translatedCoordinates.get(0);
String marketDataSet = (String) translatedCoordinates.get(1);
String instrumentId = (String) translatedCoordinates.get(2);
String expiry = (String) translatedCoordinates.get(3);
String moneyness = (String) translatedCoordinates.get(4);


// Retrieve spot price
Double spotPrice = spotRetriever.getMarketData(List.of(asOfDate, marketDataSet, instrumentId));
if (spotPrice == null) {
   return null;
}


// Compute strike from moneyness and spot price
var moneynessMapper = mapMoneyness(translatedCoordinates);
Double strike = (moneynessMapper.applyAsDouble(moneyness) + 100.0)* spotPrice / 100;


// Try to retrieve volatility market data without interpolation
Double marketDataSingle = volRetriever.getMarketData(List.of(asOfDate, marketDataSet, instrumentId, expiry, strike));


if (marketDataSingle != null) {
   return marketDataSingle;
}

If the point does not exist, we retrieve the strike curve corresponding to the expiry:

// If the volatility market data cannot be retrieved directly, retrieve the strike curve for the current expiry
CompositeKey volCurveCacheKey = CompositeKey.of(asOfDate, marketDataSet, instrumentId, expiry, "VolSurface");


ICurveMarketData<Double> volCurve = (ICurveMarketData<Double>) getQueryCache().get(volCurveCacheKey);
if (volCurve == null) {
   volCurve = volRetriever.getVolCurve(
           asOfDate,
           marketDataSet,
           instrumentId,
           expiry
   );
}
if (volCurve == null) {
   return null;
}


getQueryCache().put(volCurveCacheKey, volCurve);

and use it to perform linear interpolation to compute the correct market data value:

// Perform linear interpolation along the strike axis
CompositeKey interpolatorCacheKey = CompositeKey.of(volCurve, "Interpolator");


var interpolator = (IInterpolator) getQueryCache().get(interpolatorCacheKey);
if (interpolator == null) {
   try {
       interpolator = interpolationService.getInterpolator(
               InterpolationMode.LINEAR,
               ArrayUtils.toPrimitive(volCurve.getValues()),
               ArrayUtils.toPrimitive(volCurve.getTenors())
       );
   } catch (ActiveViamException exception) {
       LOGGER.error("[INTERPOLATING_SURFACE_RETRIEVER] Unable to create interpolator based on the coordinates {}.", translatedCoordinates);
       throw new ActiveViamRuntimeException(exception);
   }
   getQueryCache().put(interpolatorCacheKey, interpolator);
}


return interpolator.value(strike);

In addition, we can now use the previously defined container to instantiate this contextual market data retriever inside a post-processor (getMarketDataRetrievalContainer() will use the RETRIEVER_PROPERTY post-processor property in the market data retrieval container service):

@Override protected EqVolMarketDataRetriever getMarketDataRetriever(ILocation location) {
   return new EqVolMarketDataRetriever(getMarketDataRetrievalContainer().name(),
           getMarketDataRetrievalContainer().coordinateTranslator());
}
We also need to define the injections of attributes for that post-processor:

@Component
public class EqVolPostProcessorInjector implements IPostProcessorInjector {

   @Autowired
   public ISpotMarketDataRetriever spotMarketDataRetriever;

   @Autowired
   public IEqVolMarketDataRetriever volMarketDataRetriever;

   @Autowired
   public IInterpolationService interpolationService;

   @Override
   public void apManagerInitPrerequisitePluginInjections() {
       inject(IPostProcessor.class, EqVolMarketDataPostProcessor.PLUGIN_KEY,
               "spotRetriever", spotMarketDataRetriever);
       inject(IPostProcessor.class, EqVolMarketDataPostProcessor.PLUGIN_KEY,
               "volRetriever", volMarketDataRetriever);
       inject(IPostProcessor.class, EqVolMarketDataPostProcessor.PLUGIN_KEY,
               "interpolationService", interpolationService);
   }
}

Furthermore, we can now create a CopperMeasure, based on this post-processor, to use in our application. If we want to add the measure to the Sensitivities cube, we can define a Spring Bean. For more information about this mechanism, see how to configure measures using Spring Beans.

@SensitivitiesCopperContextBean
@Qualifier(EQUITY_VOL_MARKET_DATA)
public CopperMeasure equityVolMarketData(LevelsProperties levelsProperties) {
   return Copper.newPostProcessor(EqVolMarketDataPostProcessor.PLUGIN_KEY)
           .withProperty(EqVolMarketDataPostProcessor.LEVELS_PROPERTY,
                   new LevelIdentifier[] {
                           levelsProperties.getAsOfDate(),
                           levelsProperties.getMarketDataSet(),
                           levelsProperties.getRiskFactor(),
                           levelsProperties.getTenors(),
                           levelsProperties.getMoneyness()
           })
           .withProperty(EqVolMarketDataPostProcessor.RETRIEVER_PROPERTY, EQ_VOL_MARKET_DATA_RETRIEVER)
           .withFormatter("DOUBLE[#,##0.00;-#,##0.00]")
           .as("EQ Volatility");
}

We also need to add a Bean to define the logic for the initial load of our datastore:

@Bean
@Qualifier(SP_QUALIFIER__COMMON_TOPIC_TO_STORE_AND_FILE_MAP)
@Order(0)
public ChannelParametersHolderOperator equityVolMarketDataTopicToStoreAndFilePatternMapDefault() {
   return patterns -> patterns.addTopic(VOLATILITY_MARKET_DATA_STORE, VOLATILITY_MARKET_DATA_STORE, "glob:**EqVol_Market_Data*.csv", List.of(ALL_MARKET_DATA, ALL_FACTS));
}

Finally, we’ll need to update the topicToScopeToLoadConverter and topicToScopeToRemoveWhereConditionConverter method in the DataLoadControllerFileConfig class to add the VOLATILITY_MARKET_DATA_STORE topic in those methods.

Here is an example of a query for that new measure that we have created:

To see more information about the Market data API along with additional examples, check out the documentation.