C++ Backtest Environment — Part 3: Bloomberg API Data Interface
I’ve recently gained access to a Bloomberg Terminal through my school at last. Thus, from here on out, this series will be using data pulled from the Bloomberg API rather than Yahoo Finance. However, the structure will be similar and it shouldn’t be too difficult to switch between the two sources. The topic for the rest of this post will be focused solely on interfacing with the Bloomberg API through a Bloomberg Terminal.
The Bloomberg API is a formidable resource. It allows users to pull historical EOD data from many years past, intra-day tick data for the past 140 days, and even real-time market data through a subscription. My focus at the moment is just on the EOD historical data, but I will try to structure my code so that it won’t be too difficult to implement the other two types sometime in the future.
Before we begin, we need to install and link the API libraries themselves. On the Terminal, login and enter the command WAPI <GO> to be taken to the download page for all things API-related. Find the package for your system and C++ and then download it. You’re going to then want to move the folder inside the package named “C++” into somewhere you will be able to link to. For me, that was in “C://blp” with the rest of the Terminal’s code. Then, in your project, you want to provide the include directory and link to the .lib files. The necessary lines in my CMakeLists.txt look like this:
# CMakeLists.txt# Bloomberg library includes
include_directories("C:/blp/C++/include")# Build the backtester executable named Backtester
add_executable(Backtester main.cpp ${BACKTEST_SRCS})# Now link to the Bloomberg libraries
file(GLOB BLPAPI_LIBRARIES "C:/blp/C++/lib/*.lib")
message(STATUS "Bloomberg Libraries: ${BLPAPI_LIBRARIES}")
target_link_libraries(Backtester ${BLPAPI_LIBRARIES})
After linking and providing the includes, you will be able to utilize Bloomberg API libraries inside your C++ code. In the header for your classes that utilize the Bloomberg libraries, you should include these files, as they are the ones you will mostly use:
// bloombergincludes.hpp// Bloomberg API includes
#include <blpapi_defs.h>
#include <blpapi_correlationid.h>
#include <blpapi_element.h>
#include <blpapi_event.h>
#include <blpapi_exception.h>
#include <blpapi_message.h>
#include <blpapi_session.h>
#include <blpapi_subscriptionlist.h>
Finally, it is useful to use the Bloomberg::blpapi namespace in your files to increase readability. I will be using this throughout the code blocks:
using namespace Bloomberg::blpapi;Before we actually get coding, we also need to understand how Bloomberg API requests work. The first component is the Session, which uses a host name and a port to run a local server which connects to Bloomberg. The default port is 8194 and is the easiest to work with.
Important: Before enabling the connection, Bloomberg also requires running bbcomm.exe to open up the connection. Having a logged in session on the Bloomberg Terminal application is sufficient as well, and is what I usually do. If you have trouble connecting with the API, you can run the API Diagnostics Tool program that came with the Bloomberg Application to try to figure out your problem and repair it.
Anyways, with an opened Session, we can then open a Service depending on what type of data we want to use. The final component is a Request built using the Service that specifies the parameters of the data we want returned. By sending our Request through the Session, we can receive events on an EventQueue and handle them as they come in. We need a queue for two reasons: (1) large data responses are split up into multiple returned PartialResponses, and (2) an event queue enables us to make multiple asynchronous requests through a single session and not have them all put through the Session’s single event handler. We will build our own data handler for interpreting these Responses and putting them into a standard library structure for easier manageability. That’s pretty much it for a broad summary, now we can begin implementing the retriever.
We will organize our retrieval code in two classes: DataRetriever and HistoricalDataHandler, the latter of which will handle the events on the event queue. DataRetriever contains a single function pullHistoricalData() which returns an std::unordered_map with the retrieved data. HistoricalDataHandler contains three functions: processResponseEvent(), processExceptionsAndErrors() , and processErrors() . Each of these will be explained as we implement them.
In the constructor of the DataRetriever class, we want to build the session and start it running so that we don’t have to worry about restarting the session every time we call the history retriever. In this case, session is a std::unique_ptr to a session object, stored as a local member of the DataRetriever class. This allows us to retain the open session across function calls, and satisfy RAII. The instantiation of a session requires a SessionOptions object as well. Altogether, the constructor should look like this:
// dataretriever.cpp// Data Retriever constructor
DataRetriever::DataRetriever(const std::string &p_type) : type(p_type) {
// First initialize the session options
SessionOptions session_options;
session_options.setServerHost("localhost");
session_options.setServerPost(8194);
// Now use the options to build a session object
session = std::make_unique<Session>(session_options);
// Open up the session in preparation for data requests
if (!session->start()) {
throw std::runtime_error("Failed to start session!");
}
}
The historical data retriever is a little bit more complicated. In this function, we build the Service, Request, EventQueue, and Event Handler through which we receive data. The EventHandler will fill a std::unordered_map which it will return through std::move in a std::unique_ptr to save memory space. The parameters of this function will specify the parameters for the data. Finally, just know that SymbolHistoricalData is a custom data structure which contains a string for the symbol and then a further ordered map of dates and EOD data.
// dataretriever.cpp// Retrieves the historical data
std::unique_ptr<std::unordered_map<std::string, SymbolHistoricalData>> DataRetriever::pullHistoricalData(const std::vector<std::string> &securities, const Datetime &start_date, const Datetime &end_date, const std::vector<std::string> &fields, const std::string &frequency) {
// Ensure that this instance of DataHandler is historical
if (type != "HISTORICAL_DATA") return; // Open the pipeline to get historical data
session->openService("//blp/refdata");
Service serv = session->getService("//blp/refdata");
// Build the request
Request request = serv.createRequest("HistoricalDataRequest");
for (const std::string& i : securities) {
request.append("securities", i.c_str()); }
for (const std::string& i : fields) {
request.append("fields", i.c_str()); }
request.set("startDate", get_date_format(start_date).c_str());
request.set("endDate", get_date_format(end_date).c_str());
request.set("periodicitySelection", frequency.c_str());}
Note: get_date_format() is a function which simply returns the YYYYMMDD format of a Bloomberg Datetime.
Now that we have our request built, we need to send it through the session and attach an event handler to it to process each response.
// Build an Event queue onto which the data will be put
EventQueue queue;
// Perform request with the queue
session->sendRequest(request, CorrelationId(1), &queue);
// Now handle events as they come into the queue
HistoricalDataHandler handler;
bool responseFinished = false;
while (!responseFinished) {
// Handle events until responseFinished becomes true
Event event;
if (queue.tryNextEvent(&event) == 0) {
responseFinished = handler.processResponseEvent(event);
}
}
// When the event finishes, return the handler's data map
return std::move(handler.target);
}Now, we have a way to retrieve events, but we need a class to verify them and notify the program when the responses have finished as well as use the data in those responses to build the data map. Such is the job of the HistoricalDatHandler, whose constructor simply initializes the unique ptr to the map it will return in a member variable target:
// dataretriever.cpp// HistoricalDataHandler constructor / initializer list
HistoricalDataHandler::HistoricalDataHandler() : target(std::make_unique<std::unordered_map<std::string, SymbolHistoricalData>>()) {}
With the data map built, we can finally provide a functionality to handle the events passed into its processResponseEvent() function. Before we can take data from an Event, we actually need to first verify that the Event contains data and is not an error or exception. That is the job of processExceptionsAndErrors() , which will return true if the given Event is an error or exception:
// dataretriever.cpp// Makes sure Events are not errors
bool HistoricalDataHandler::processExceptionsAndErrors(Message msg) {
// If there is no security data, process the errors
if (!msg.hasElement("securityData)) {
if (msg.hasElement("responseError")) {
Element response_data = msg.getElement("responseError);
// Log the error and return true
std::cout << response_data << std::endl;
}
return true;
} // There could also be field exceptions within the security data
Element security_data = msg.getElement("securityData");
Element field_exceptions = security_data.getElement("fieldExceptions");
// Ensure that the fieldExceptions element is empty
if (field_exceptions.numValues() > 0) {
Element element = field_exceptions.getValueAsElement(0);
// Log the exception
std::cout << element << std::endl;
return true;
}
return false;
}
Once we are sure we have no errors or exceptions, we can interpret the data from the Response received. Like I mentioned earlier, large Responses are split up into smaller PartialResponses. However, the last packet of data in the stream is always a Response object. We will use this fact to know when the data has stopped coming and then return the STL container.
// dataretriever.cpp// Processes responses, returning true once data finished
bool HistoricalDataHandler::processResponseEvent(const Event &event) {
// Make sure the message is a response
if ((event.eventType() != Event::PARTIAL_RESPONSE) && (event.eventType() != Event::RESPONSE)) { return false; }
// Iterates through the messages returned by the event
MessageIterator msgIter(event);
while(msgIter.next()) {
Message msg = msgIter.message();
// Make sure no exceptions or errors
if (!processExceptionsAndErrors(msg)) {
// Handle the fields and load their data
Element security_data = msg.getElement("securityData");
// Build our custom data type
SymbolHistoricalData shd;
shd.symbol = security_data.getElementAsString("securityName");
// Fill data map with field data
Element field_Data = security_data.getElement("fieldData");
if (field_data.numValues() > 0) {
for (int i = 0; i < field_data.numValues(); ++i) {
Element element = field_data.getValueAsElement(i);
Datetime date = element.getElementAsDatetime("date"); // Fill the SHD with the data
for (int j = 0; j < element.numElements(); ++j) { Element e = element.getElement(j);
shd.data[date][e.name().string()] = e.getValueAsFloat64();
}
}
} // Once all data has been entered, append SHD
if (target->find(shd.symbol) != target->end()) {
target->operator[](shd.symbol).append(shd);
} else {
target->operator[](shd.symbol) = shd;
}
} else {
// Exception occurred in message
std::cout << "Exception occurred." << std::endl;
}
} // If the event was a RESPONSE, then the data is finished
return event.eventType() == Event::RESPONSE;
}
Note: SymbolHistoricalData is a struct containing a std::string symbol denoting the symbol of security the data represents, a std::map<Datetime, std::unordered_map<std::string, double>> data holding all the dates and data fields for their respective dates, and finally a function append() which appends another SymbolHistoricalData’s data to the current data object (given they have the same symbol).
This should be all we need to pull historical EOD data! A quick recap: we created a class DataRetriever with a function pullHistoricalData which can be given the parameters for the request. This function then opens up a Historical Data service through Bloomberg and sends the requests, processing responses with our own HistoricalDataHandler as they come onto the Event Queue. Once the Event Queue encounters a Response event (denoting the end), pullHistoricalData() moves a std::unique_ptr to the data into the object being assigned to the function. We can test this functionality with a couple lines of code. Make sure you have a connection through a running bbcomm.exe or a logged in instance of the Bloomberg Terminal application active first!
// main.cpp#include "dataretriever.hpp"int main(int argc, char* argv[]) {
// Build a Data Retriever for historical data
DataRetriever dr("HISTORICAL DATA");
// Now pull historical data into the unique ptr
std::unique_ptr<std::unordered_map<std::string, SymbolHistoricalData>> data = dr.pullHistoricalData({"IBM US EQUITY"}, Datetime(2005, 3, 3, 0, 0, 0, 0), Datetime(2006, 3, 3, 0, 0, 0, 0), {"PX_LAST"});
// Print the retrieved data to console
auto i = data->operator[]("IBM US EQUITY").data.begin();
while (i != data->operator[]("IBM US EQUITY").data.end()) {
std::cout << "DATE: " << i->first << ", PX_LAST: " << i->second["PX_LAST"] << std::endl;
++i;
}
return 0;
}
When you run this, you should get 253 lines of the last prices on each market day between March 3, 2005 and March 3, 2006 (not 252 because it includes both endpoints). Feel free to test it out with other data fields.
With our data handling up and running (for the second time), we can now get around to building the new and improved backtester!
The source code for this post can be found at my GitHub. If you have any other questions, feel free to email me at kirkilese20@kent-school.edu!
