Troubleshooting IPA & RD Python Library
This article is writen a little differently to the others I normally publish. It takes on more of a Blog form. For the original article, please go to the LSEG Developer Portal.
I was working on expanding/part 2 of this article on the LSEG Developer Portal, in which I send time-series data to a service that functions, in effect, as a calculator; this service is called ‘Instrument Pricing Analytics’, IPA. Specifically: I used the LSEG Data (LD) Python Library to send index price data and wanted to retrieve back Implied Volatilities.
While doing this, I once retrieved back a data frame filled with <NA>s and I had to troubleshoot it… But where to start?
In this short article, I look into my journey troubleshooting this step, in the hope that it’s of use to you!
Background
I used certain common libraries, such as Pandas, in addition to RD library functions:
import pandas as pd
import refinitiv.data as rd # This is LSEG's Data and Analytics' API wrapper, called the Refinitiv Data Library for Python.
from refinitiv.data.content import historical_pricing # We will use this Python Class in `rd` to show the Implied Volatility data already available before our work.
import refinitiv.data.content.ipa.financial_contracts as rdf # We're going to need thtis to use the content layer of the RD library and the calculators of greeks and Impl Volat in Instrument Pricing Analytics (IPA) and Exchange Traded Instruments (ETI)
from refinitiv.data.content.ipa.financial_contracts import option # We're going to need thtis to use the content layer of the RD library and the calculators of greeks and Impl Volat in IPA & ETI
# Let's authenticate ourseves to LSEG's Data and Analytics service, Refinitiv:
try: # The following libraries are not available in Codebook, thus this try loop
rd.open_session(config_name="C:\\Example.DataLibrary.Python-main\\Example.DataLibrary.Python-main\\Configuration\\refinitiv-data.config.json")
rd.open_session("desktop.workspace")
except:
rd.open_session()You can already see traces of that service I mentioned, IPA, in the 4th line. FYI: Here we are using the LD Library (named the RD Library) version 1.0.0b24.
For more information on the configuration proccess (as per line 9 in the code above), please read this quickstart.
I then gathered data on two Options, a live one (‘2HSI16200L8.HF’) and an expired one (‘HSI19300N3.HF^B23’).
HSI_test0 = rd.content.historical_pricing.summaries.Definition(
'2HSI16200L8.HF',
interval=rd.content.historical_pricing.Intervals.DAILY,
fields=['SETTLE'],
start='2023-10-01',
end='2024-01-09').get_data().data.df
hk_rf = 100 - rd.get_history(
universe=['HK3MT=RR'], # HK10YGB=EODF, HKGOV3MZ=R, HK3MT=RR
fields=['TR.MIDPRICE'],
start=HSI_test0.index[0].strftime('%Y-%m-%d'),
end=HSI_test0.index[-1].strftime('%Y-%m-%d'))
HSI_test1 = pd.merge(
HSI_test0, hk_rf, left_index=True, right_index=True)
HSI_test1 = HSI_test1.rename(
columns={"SETTLE": "OptionPrice", "Mid Price": "RfRatePrct"})
hist_HSI_undrlying_pr = rd.get_history(
universe=['.HSI'],
fields=["TRDPRC_1"],
# interval="1D",
start=HSI_test0.index[0].strftime('%Y-%m-%d'),
end=HSI_test0.index[-1].strftime('%Y-%m-%d'))
HSI_test2 = pd.merge(HSI_test1, hist_HSI_undrlying_pr,
left_index=True, right_index=True)
HSI_test2 = HSI_test2.rename(
columns={"TRDPRC_1": "UndrlyingPr"})
HSI_test2.columns.name = 'STXE42000D3.EX'
HSI_test2
Content Layer
There happen to be three layers to the Data Library. The 1st, called the Access Layer, is the most user-friendly; the 2nd, the Content Layer, is less so, and the Delivery Layer simply sends data requests to LSEG Databases and returns datasets in their ‘rawest’ form — with no/few filters. Below, we will use the Content Layer:
request_fields = ['MarketValueInDealCcy', 'RiskFreeRatePercent', 'UnderlyingPrice', 'PricingModelType', 'DividendType', 'UnderlyingTimeStamp', 'ReportCcy', 'VolatilityType', 'Volatility', 'DeltaPercent', 'GammaPercent', 'RhoPercent', 'ThetaPercent', 'VegaPercent', 'ErrorMessage']response = option.Definition(
buy_sell=option.BuySell.BUY,
call_put=option.CallPut.CALL,
instrument_code=live_inst,
strike=float(4200),
underlying_definition=option.EtiUnderlyingDefinition(instrument_code=live_inst),
underlying_type=option.UnderlyingType.ETI,
fields=request_fields,
pricing_parameters=option.PricingParameters(
pricing_model_type='BlackScholes',
volatility_type="Implied",
market_value_in_deal_ccy=float(HSI_test2['OptionPrice'][0]),
report_ccy="HKD",
risk_free_rate_percent=float(HSI_test2['RfRatePrct'][0]),
underlying_price=float(HSI_test2['UndrlyingPr'][0]),
valuation_date=HSI_test0.index[0].strftime('%Y-%m-%d')
)).get_data()I used VSCode and Intellisense to figure out which arguments were needed in `option.Definition`:
Intellisense is great to quickly write functions like this, however, as we will see soon, it’s not enough.
My code did not work… I kept getting errors:
How can I investigate such an issue?
What this Content Layer does is send request to LSEG’s back-end to a specific endpoint. One can see these endpoints on the API Playground:
How can you find the endpoint in question that you used? this is where ‘Logs’ come in:
Switching `Logs` On
You can switch on logs with the following code in Python:
rd.get_config().set_param(
param=f"logs.transports.console.enabled", value=True
)
session = rd.open_session()
session.set_log_level("DEBUG")You can also switch on logs with the configuration file. If we look into the configuration file used to authenticate ourselves to RD, we can change the “logs”>“transports” >”file”>”enabled” to true, giving us a log file in the path where we are running the Python code. The one produced from this on my machine was called 20240109–1353–11136-refinitiv-data-lib.log. It started with
[2024-01-10T10:59:21.083581+01:00] - [sessions.desktop.workspace.0] - [DEBUG] - [4772 - MainThread] - [_session_provider] - [session_provider] - + Session created: DesktopSession
name = 'workspace'
connection = DesktopConnection
stream_auto_reconnection = True
handshake_url = http://localhost:9000/api/handshake
state = OpenState.Closed
session_id = 0
logger_name = sessions.desktop.workspace.0You can check which session you are using, out of the Desktop and Platform session, on lign 8. More on these sessions here.
From there, I was looking for the JSON message sent to the IPA service. To find those, I looked for the last instance of api/rdp/data, since we’re looking into our last usage of the RDP API in our code. I looked for the last instance of this and found the request which started with the following:
url = http://localhost:9001/api/rdp/data/quantitative-analytics/v1/financial-contracts
method = POST
headers = {'Content-Type': 'application/json', ... }
json = {'universe': [{'instrumentType': 'Option', 'instrumentDefinition': ... 'ErrorMessage']}
[2024-01-10T10:59:21.083588+01:00] - [RetryTransportBase] - [DEBUG] - [13552 - MainThread] - [_retry_transport] - [_handle_request] - Sending request to http://localhost:9001/api/rdp/data/quantitative-analytics/v1/financial-contracts
[2024-01-10T10:59:23.670464+01:00] - [sessions.desktop.workspace.0] - [DEBUG] - [13552 - MainThread] - [http_service] - [request] - HTTP Response id 9
status_code = 200
text = {"headers":[{"type":"Float","name":"Strike"}, ... , {"type":"String","name":"ErrorMessage"}],"data":[[4200.0,4804.0,1.146,17195.84,"BlackScholes", ... ,"NaN","NaN","NaN",,"NaN","Unable to calculate the Implied Volatility."]]}It is too long to show in full, but in line 1 & 5, you can clearly see the endpoint that was pinged. Here it’s `quantitative-analytics/v1/financial-contracts`.
API Playground
Coming back to the API Playground, we can now find all the documentation we’re after:
Lookinginto the examples given in the ‘Playground’ tab, I managed to construst a ‘body’ that gave some data similar to what I was after by ommiting data that I originally thought was useful. I could see the data returned by pressing on ‘SEND’:
The body that seem to give me some data was:
{
"instrumentType": "Option",
"instrumentDefinition": {
"buySell": "Buy",
"underlyingType": "Eti",
"instrumentCode": 'STXE42000D3.EX',
},
"pricingParameters": {
"marketValueInDealCcy": 4804,
"riskFreeRatePercent": 1.1460,
"underlyingPrice": 17195.84,
"pricingModelType": "BlackScholes",
"dividendType": "ImpliedYield",
"volatilityType": "Implied",
"underlyingTimeStamp": "Default",
"reportCcy": "HKD"
}
}But what do we do now? Well, there are two options, you could go back to the Content Layer and change the relevent arguments, or use the Delivery Layer. I haven’t shown you how to use that yet, so let’s try it:
Delivery Layer
live_universe = [
{
"instrumentType": "Option",
"instrumentDefinition": {
"buySell": "buy",
"underlyingType": "Eti",
"instrumentCode": live_inst,
},
"pricingParameters": {
"marketValueInDealCcy": float(HSI_test2['OptionPrice'][i]),
"riskFreeRatePercent": float(HSI_test2['RfRatePrct'][i]),
"underlyingPrice": float(HSI_test2['UndrlyingPr'][i]),
"pricingModelType": "BlackScholes",
"dividendType": "ImpliedYield",
"volatilityType": "Implied",
"underlyingTimeStamp": "Default",
"reportCcy": "HKD"
}
}
for i in range(len(HSI_test2.index))]
def Chunks(lst, n):
"""Yield successive n-sized chunks from lst."""
for i in range(0, len(lst), n):
yield lst[i:i + n]batchOf = 100 # 100 is the max
for i, j in enumerate(Chunks(live_universe, 100)):
print(f"Batch of {batchOf} requests no. {str(i+1)}/{str(len([i for i in Chunks(live_universe, batchOf)]))} started")
# Example request with Body Parameter - Symbology Lookup
live_troubleshoot_request_definition = rd.delivery.endpoint_request.Definition(
method=rd.delivery.endpoint_request.RequestMethod.POST,
url='https://api.refinitiv.com/data/quantitative-analytics/v1/financial-contracts',
body_parameters={"fields": request_fields,
"outputs": ["Data", "Headers"],
"universe": j})
live_troubleshoot_resp = live_troubleshoot_request_definition.get_data()
headers_name = [h['name'] for h in live_troubleshoot_resp.data.raw['headers']]
if i == 0:
live_troubleshoot_df = pd.DataFrame(
data=live_troubleshoot_resp.data.raw['data'],
columns=headers_name)
else:
_live_troubleshoot_df = pd.DataFrame(
data=live_troubleshoot_resp.data.raw['data'],
columns=headers_name)
live_troubleshoot_df = live_troubleshoot_df.append(_live_troubleshoot_df, ignore_index=True)
print(f"Batch of {batchOf} requests no. {str(i+1)}/{str(len([i for i in Chunks(live_universe, batchOf)]))} ended")Batch of 100 requests no. 1/1 started
Batch of 100 requests no. 1/1 ended
live_troubleshoot_df.columns.name = live_inst
live_troubleshoot_df.describe(include='all')As you can see, we found what we’re after!
rd.close_session()References
Do not hesitate to ask any technical questions you may have about LSEG APIs on the LSEG Developer Comunity Q&A Forum. Please note, however, that this is a Forum aimed at answering technical questions onnly; for others (e.g.: content-related questions), please reach out to my.refinitiv.com.
For documentation, training articles and videos and more on LSEG APIs, please visit the LSEG Developer Portal.
