Want to join our team? Visit the Octane Careers page!

Meet the Octane Technology & Product Team was originally published in Octane — Instant financing to fuel your lifestyle on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve ever hesitated when making a decision, even one as simple as what to have for dinner, you know just how difficult it can be to make good decisions quickly. That’s especially true in lending and finance, where peoples’ very livelihoods may be on the line. Making the right decision takes preparation, information, and the ability to synthesize that information and its context into a plan of action. Accordingly, decision-making is a task we have historically left to well-paid experts, no matter the field.

But experts are by definition rare, and even an expert has only so many hours in a day. That means good human-level decision-making is something that doesn’t scale up easily when a business seeks to expand, no matter how many procedural manuals we write. Here at Octane Lending, however, we’re using machine learning and artificial intelligence to help us do exactly that: Scale our expertise and innovation to provide the quickest, most accurate, easiest, and lowest-cost loan process in the industry. At the core of how we do that is the Octane Lending Income Verification Model.

What Is the Octane Lending Income Verification Model?

The Income Verification Model determines whether we need to require proof of income from a loan applicant. It runs as a part of the machine-learning-powered Octane approval process, which takes less than ten seconds. By predicting which buyers are most likely to have the capacity to repay their loans, the Income Verification Model also helps improve Octane’s overall repayment rate and reduces the cost of lending, as savings are passed down the chain. Because proof of income is often waived by the Income Verification Model, it saves the customer the hassle and time of collecting documentation, which in turn saves our own staff the time spent verifying them. But even more important than our Income Verification Model’s benefits to speed, scalability, and savings is how it all works.

Senior data scientist Eva Xu and engineering manager Pat Paripoonnanonda, along with many folks in the Credit Risk, Engineering, and Product departments, spearheaded the development and deployment of the Income Verification Model, Octane Lending’s first ground-up model built entirely in-house.

“There’s zero third-party data,” Xu said. “All of the historical verified data used in training the model came from our app.”

But the Income Verification Model is not just about trawling data and identifying who has excess income, said Paripoonnanonda. “It’s determining whether the differential between reported income and real income would impact loan terms.”

How Does the Income Model Work?

“There’s a lot that goes into making the model,” Xu said. “Getting the data, making sure the features we’re using make logical sense, cleaning those features, making sure there’s not a lot of sparse data, determining the right subset of data to look at. Then we can add and remove features, apply a score, and set thresholds.”

Xu found the process of building the model rewarding, especially compared to other AI/ML work in the finance sector.

“Compared to banks, when you’re working with models or anything like that, most of the cool things are already built,” Xu said. “So you end up fine-tuning legacy models, fixing bugs, retraining with new data, and trying to make small improvements. That’s not nearly as exciting as saying, ‘I made this.’ ”

Drawing on over two-and-a-half years of Octane lending data as a training base, Xu built the Income Verification Model using state-of-the-art parallel tree boosting to quickly and efficiently evaluate a potential loan candidate.

“On our end, we compute necessary features,” Paripoonnanonda said [features are derived from the submitted application and credit profile], “which we then package into a private API call to the model, which returns a score, and the decision whether to request proof of income is made from there. This ensures greater security for the data and our model while also ensuring nearly instantaneous decisions.”

“We use recall and precision as our metrics,” for the model’s success, said Xu. “But there’s an inherent trade-off between recall and precision.”

That’s because, in machine learning, maximizing precision results in fewer positive instances passing the classifier. In this case, that means fewer loan approvals overall, but this helps ensure the few that are approved are good loans with a high likelihood of repayment.

On the other hand, maximizing recall results in more positive instances from the classifier. In this case, that means more loan approvals overall, but some of those loans may include a greater risk of default than would otherwise be desirable.

Balancing concerns like these is the art and science of AI/ML in the lending industry.

AI: Better Than Human?

While reducing friction, meaning the time and effort required, in the lending and buying process is a primary objective for Octane Lending as well as for the Income Verification Model, it’s not the only objective. The model itself helps ensure we’re not simply making as many loans as we can as quickly as we can, but that we’re making loans that are also right for the customer. That can mean making sure a customer doesn’t take too large a loan and end up in default, or it can mean ensuring our lending decisions are as free from bias as possible by carefully curating the model’s input features.

For example, we don’t use geographical or gender data when evaluating a loan candidate, said Xu, because they’re not relevant to the outcome. In fact, no personally identifiable information (PII) is sent to the model, according to Paripoonnanonda.

But Humans Are Still the Key to Our Success

Working to eliminate bias isn’t just something we do because it’s popular. It’s part of Octane’s culture. We are also proud to support a strong culture of innovation, and we seek to ensure everyone, even junior employees, has the opportunity to contribute.

“At Octane, we favor balancing our teams with both more experienced engineers and younger engineers,” Paripoonnanonda said. “It not only gives our senior engineers mentoring opportunities, but we find a lot of value in the more refreshing ideas of our younger engineers. Sometimes they’ll think of something a more experienced engineer would tend to overlook, or be familiar with a more modern tech stack.”

“If you put in the work, you get recognized here,” agreed Xu. “You can take a lot of initiative; you’re able to go through the entire process of designing a model, from getting the data to feature engineering. It’s exciting too… You’re able to present your results to stakeholders and the C-suite, and everyone in the company is very interested in what you’re doing.”

Octane’s Income Verification Model was originally published in Octane — Instant financing to fuel your lifestyle on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Octane, with the fast growth of our lending business, our lending and servicing operations required additional workflows in order to effectively do their jobs. So we decided to use Salesforce creatively to help with some of those needed internal workflows. However, with this new tool in play, it was soon necessary to integrate data generated by our sales and operations teams inside Salesforce with our data lake. It would enable us to produce complete and accurate insights and reporting for various stakeholders within and outside Octane.

This past spring, the Data Engineering team began brainstorming the best way to accomplish this goal. In this blog post, we want to share our findings of the various options available for Salesforce integration in 2021, our approach in evaluating these options, and our collaborative decision process. This is part one of a two-part series. The second part of this blog post will dive into the details of knowledge we gained and challenges we encountered while integrating the Salesforce data with the approach we’ve chosen.

Data Pipelines Overview

Like all other engineering teams at Octane, the Data Engineering team operates mostly in the Amazon Web Services (AWS) environment. We own a number of nightly batch data pipelines, including MySQL, PostgreSQL, and Salesforce, which extract operational data from stores on and off AWS.

Our current pipeline architecture roughly follows the Extract, Load, Transform (ELT) pattern. We run batch processes to “extract” data from our various sources and “load” to a staging area, and then “transform” the data for usability in our data lakehouse. After that, various data products are built on top. These data products are persisted in Parquet format on S3 and then made available to our community of internal users via AWS Athena, Redshift Spectrum, or directly on S3, all utilizing AWS Glue as an underlying metastore.

With Salesforce, we didn’t have direct access to the underlying Salesforce Oracle databases, so we instead had to rely on tools made available by Salesforce directly or third parties to help us with our goal. Salesforce as a new data source didn’t fit our established extraction pattern into our staging area. Our challenge was to select the best approach to balance the following need: make raw and processed Salesforce data available in our data lake in a timely, scalable, and maintainable fashion, preferably without significantly affecting our current ELT architecture.

Decisions, Decisions

To solve this challenge, the team evaluated multiple approaches of ingesting Salesforce data, from consuming the Salesforce bulk REST API directly to fully managed paid solutions like Stitch. Our evaluation list consisted of:

• Airbyte
• AWS AppFlow
• Direct Salesforce Bulk API consumption
• Fivetran
• Matillion
• Meltano
• Mulesoft
• SpringML and other Spark-based extract solutions (CDATA, Progress)
• Stitch

After some research, we narrowed this down to a short list:

• Direct Salesforce Bulk API consumption
• AWS AppFlow
• Meltano
• Stitch

Considerations that eliminated other options varied from nonavailability of S3 as a “target,” complexity of workflows, and lack of publicly available documentation. Notably, Airbyte was a strong choice, but unfortunately didn’t have S3 as a target available at the time (Airbyte has since added an S3 connector).

We further evaluated these options and discussed potential architectures for each, considering the effort that would be involved to implement the solution and, more importantly, to maintain it long-term within our existing data infrastructure and environment. Below are key summaries of each option we considered and the final selection we chose and implemented.

Direct Salesforce API Consumption

The default “bare” solution was to query the Salesforce Bulk API directly. Here, the two suboptions were 1) sticking as closely as possible to our existing extraction architecture and simply performing data pulls from ephemeral EC2 instances, or 2) adopting a more serverless approach with Step Functions. The team was already familiar with the first option, so we focused our evaluation on the serverless approach. Figure 1 shows an approximate proposed solution with this approach.

Figure 1. Direct Salesforce Bulk API consumption approach, a serverless architecture.

Regardless of the suboption, with direct Salesforce API consumption the team also considered the additional upfront engineering work needed to reliably consume gigabytes of data per night, with parallelization, Salesforce throttling, and Salesforce-specific bulk chunking rules in mind, and in light of our project delivery timeline goal.

Stitch

Stitch is a fully “hands-off” SaaS solution allowing one to easily set up scheduled pipelines with a few button clicks in their UI. Stitch extracts from more than 100 different sources, including Salesforce, Google Analytics, etc., and loads to supported destinations, including S3 and all major cloud data warehouses.

We were able to set up a full, scheduled, repeatable Salesforce data extraction pipeline to an arbitrary Octane S3 bucket in the production account in under 30 minutes.

The price of that convenience starts at $1,000 per year for up to 5M extracted rows per month, and goes up with the amount of rows in absolute terms (but goes down in relative terms of cost per row) to e.g., 50M rows per month for $5,000 per year. Additional perks are unlocked at “Enterprise” level, or above 5 registered users, at a presumably higher price.

Stitch uses open source Singer for its connectors and touts its extensibility and open source. Stitch is also the official sponsor of Singer, and it’s mostly Stitch engineers who contribute to Singer. Singer is the primary open source library around which other SaaS ETL startups and ETL tools wrap their own connectors and targets.

The downsides we considered included security and privacy; challenges of integration within our current overall pipeline architecture, especially notifications of completion; and non-open source and opacity of the process, among others. With the nature of our business, we must be SOC 2 compliant. While Stitch does claim SOC 2 compliance in regard to customers’ data, that still meant additional annual auditing was required by our Security team, adding to our annual cost basis with this solution.

AWS AppFlow

AWS provides Salesforce integration with AppFlow. This integration is similar to Stitch but with fewer features. The upside is obviously that it’s managed by AWS in the same account, so concerns raised about Stitch regarding extra auditing load on the Security team don’t apply. A walk-through blog post for this service is available here. Some notable characteristics for AppFlow include:

• UI driven configuration, some limited API available.
• No Terraform support yet, but there’s CloudFormation support.
• Manual creation of a “flow” (integration) for each object.
• No support for incremental extracts as far as we can tell.
• Very limited configurability (e.g., destination S3 keyspace naming, see next bullet) and doesn’t seem to play well with Glue crawler or Athena.
• Non-configurable S3 keyspaces. AppFlow injects its own keyspace (flow execution ID) inside a daily keyspace. Combined with the lack of incremental extracts, there appears to be no compelling reason for using daily S3 keyspaces.
• Discarding all data type metadata, encoding everything as a STRING.
• Easy, fully managed service, full raw Salesforce extract can be set up through UI within hours.

A viable serverless, event-based extraction architecture with AppFlow is shown in Figure 2.

Figure 2. Salesforce staging architecture with AWS AppFlow.

The downsides of using AppFlow were similar to Stitch, albeit much less pronounced. Since all of our infrastructure is already on AWS, integration within our larger data pipeline architecture was trivial and no additional auditing burden on our Security team was required. However, AppFlow was still an opaque, non-open source solution not under our control, which is always a consideration with managed solutions. In addition, there were other minor annoyances, such as lack of control over the exact S3 target keyspace (AppFlow always added an additional autogenerated keyspace “directory” at the end of the specified target keyspace).

Meltano

Meltano is an open source CLI tool, which like Stitch, also wraps around Singer connectors to expose its own “extractor” and “loader” interfaces, but in a much more pleasant way than using Singer configurations directly. Unlike Stitch, Meltano is not a SaaS offering, but a CLI tool to be run on your own infrastructure. Some key considerations are noted below:

• Meltano provides an intuitive and easy CLI-based management of Singer taps and targets, and more importantly, its own easy YAML-based configuration wrapping uglier Singer configuration requirements.
• Meltano supports containerization for production deployments.
• Meltano can be run as part of a Docker container on EC2 instances that we spin up during extraction pipelines.
• It’s open source and free, with development bankrolled by Gitlab, which spun it off as an open source project.
• Exposing Meltano as a library dependency in addition to just being a CLI tool is on the development team’s radar.
• Unlike Stitch, using Meltano allows for full control over pipeline architecture and infrastructure.
• Since Meltano just uses locally stored Salesforce authentication configuration, no Salesforce access to any third parties is provided.

However:

• Full control means full responsibility for a correct implementation.
• It’s a CLI, so calls to it will need to be shelled out of Python as part of extraction pipeline execution, and stdout and stderr fed back in.

Figure 3 shows the architecture we considered with Meltano, which was very similar to the architecture for direct Salesforce API consumption on EC2 instances (in light of our already existing ELT architecture).

Figure 3. Proposed staging architecture with Meltano.

Final Decision

In the end, after careful consideration and a cross-team deliberation process, the team voted to implement the architecture proposed in Figure 3 and to use Meltano as a tool to extract Salesforce data.

The factors that played a key role in the decision were:

• Meltano is an open-source project, which allows for full control and full visibility into how our data moves.
• The EC2-based architecture fits closest with our current framework and requires little architecture alteration.
• The Meltano tap-salesforce connector abstracts low-level interactions with the Salesforce API and makes extract job configuration easier than consuming the API directly.
• Meltano is built on the Singer spec, supporting the ingestion of other data formats in the future should the need arise.

Since then, we have implemented and released integration of Salesforce data into our data lake using Meltano. Be on the lookout for our follow-up post on our experience implementing Meltano into our architecture!

Extracting Salesforce Data at Octane was originally published in Octane — Instant financing to fuel your lifestyle on Medium, where people are continuing the conversation by highlighting and responding to this story.

Little things matter and can have a big impact.

Right before COVID hit in March 2020, the Octane Engineering team launched a process to bring 20% Time to the organization. 20% Time is a policy first started at Google (since retired) allowing engineers to spend 20% of their time on self-driven initiatives. However, when our team looked at our roadmap, we elected for something scrappier.

In October 2020 we launched Tools and Exploration Time. Every other Friday, the team would take four hours in the afternoon to experiment with different tools and frameworks in the data and engineering realm. We felt that it was enough time to produce substantive results without taking away from our team’s velocity and would allow us to find new solutions that could help us be more productive, especially given the exploding data engineering space.

Since then, we’ve rebranded the Tools and Exploration Time to Free-Range Fridays. Why? The new name just sounds more fun!

So … how does it work in practice and what impact has it had over the last nine months?

Free-Ranging Rules

Just as in the food industry, Free Range doesn’t necessarily mean free range. We have some rules. If the team has an urgent deadline , we can decide to forgo Free-Range Friday for that sprint. Participation is also optional; if a team member chooses not to participate, then the expectation is that they will take on sprint work. Participation is not limited to engineers because we believe that our product managers and data analysts should also be able to explore their interests and ideas.

We track the things we want to explore and who is working on what with a Google spreadsheet The spreadsheet is simply a list of tools and frameworks that anyone on the team can add to. Here is a screenshot of our sample spreadsheet.

During our free-ranging time, everyone must work on one of the tools/frameworks or collaborate with someone else. There are no expectations to produce or demo any work; however, knowledge sharing (verbal or written down on the spreadsheet) is expected.

Free-Ranging Results

Since we started Free-Range Fridays, our team has explored 26 different tools/frameworks. Of these 26, six have made it on our roadmap or are currently on the larger engineering roadmap. Two have been incorporated into the data engineering codebase and are something our team is currently using in production. Another two are in our broader engineering codebase and under active development. We have produced three knowledge-base/design documents and a proof of concept for PySpark declarative pipeline configuration framework. While we are very proud of these results, the most important one has been a very happy and engaged team!

Conclusion

So, will the data team participate in 20% Time once we have capacity?

At Octane, we are one of only two functional teams of specialists in the engineering org. Our stakeholders are all of Octane’s data-dependent teams (effectively the entire company!). Thus, we don’t feel like we will ever really have the capacity for full-on 20% Time without sacrificing on delivery or work-life balance. Fundamentally 20% Time lies on a spectrum and different teams may find different durations or cadences to fit their team or company better. Dialing down the time per sprint may be better than adding restrictions or stipulations around how the time is spent — two dimensions along which there are trade-offs; time spent and control over qualifying topics.

Free-Range Fridays is a compromise that’s enabled us to find new opportunities to take on and also helped create a learning-focused culture, which our team values. For now Free-Range Fridays works for our team and may be something that could work for your team(s) as well.

-Octane Data Engineering Team

Free-Range Fridays was originally published in Octane — Instant financing to fuel your lifestyle on Medium, where people are continuing the conversation by highlighting and responding to this story.