What’s LDV and Why Does it Matter for My Nonprofit or School? Part I

This is Part 1 of a three-part series on large data volumes (LDV), specifically in the nonprofit and education space.

In this part, we cover what we mean when we say large data volume, and why it is important to understand. Part 2 covers how to design an organization to be prepared for LDV, and Part 3 covers loading data into a LDV organization and best practices.

Introduction

LDV is a flexible term we use at Salesforce to describe a Salesforce organization that may have tens of thousands of users, tens of millions of records, or hundreds of gigabytes of total record storage. LDV can also be used to describe an organization where the number of records in any object exceeds 5,000,000. Though some organizations start with this volume, others reach this scale over time as the nonprofit or school uses the organization. Therefore, it’s a good idea to keep the best practices of designing for LDV in mind at the start of any project.

There are many resources available from Salesforce about LDV, but none, until now, specifically address scenarios that we at Salesforce.org see with our nonprofit and education customers. That’s why this series is focused on the scenarios we see in the nonprofit and education space and how to use LDV best practices to address them.

How do you know if you fall into one of the LDV scenarios we commonly see? Typical symptoms of LDV include:

• Related lists or Visualforce pages time out instead of displaying
• Reports or dashboards time out before completing
• Errors or degradation in the performance of Lightning components
• Error messages about reaching governor limits for SOQL, SOSL, and Apex
• Error messages about a row lock when updating records
• Nonprofit Success Pack (NPSP) recurring donation jobs take too long to run
• NPSP rollup calculation jobs don’t complete
• Third-party payment processing jobs don’t complete

Why is LDV an important topic?

The Salesforce platform is a scalable, robust solution built on established technologies that meets the needs of hundreds of thousands of customers of differing scales. Further, governor limits ensure the efficient use of resources on the multitenant platform. So why is LDV a concern?

It’s all about the customer experience. When dealing with LDV, it is important to ensure that the customer experience is not adversely affected. LDV can lead to performance problems and you may see that it’s taking too long to run reports, load list views, execute batch or transactional processes (e.g. nightly donation roll up calculations), or share calculations.

Where do we encounter LDV?

In our work with nonprofits and schools, we see LDV most frequently in certain commonly encountered situations. Here are some areas that you should keep a close eye on.

Nonprofit Success Pack (NPSP)

When using NPSP, pay attention to the following areas and objects, as these are where we typically see a large number of records.

• Households and supporters — Accounts and contacts (if using the household account model of NPSP, there will be, at least, a 1-to-1 relationship between accounts and contacts)
• Donations — opportunities
• Regular gifts — Recurring donations
• Payments or transactions
• Accounting identifiers — GAU allocations (if general accounting units are used)
• Campaign members
• Multiple addresses (if the address object is used)
• Potential or interested supporters — Leads
• Other custom objects or third-party apps

Other areas to look out for include:

• Rollups: Rollup jobs can fail when there are too many opportunities/opportunity contact roles associated with one contact or account. Typically this is a “bucket” anonymous record (covered under Bucket accounts). Rollup jobs can also fail if there are too many opportunities associated with the default GAU allocation. (This topic is covered in more detail in Part 2 of this series).
• Triggers: NPSP has a number of Apex triggers, which are fired either on create/update or as part of a scheduled process. If you have, for example, a large number of addresses (hundreds of thousands) you might see the nightly Seasonal Address job fail as a result.

Education Data Architecture (EDA)

If you’re using EDA, pay close attention to the following objects, which tend to accumulate a large number of records:

• Students / Faculty / Staff — Accounts and contacts (When using the Administrative Account or Household Account model of EDA, there will be, at least, a 1-to-1 relationship between accounts and contacts.)
• Prospective students — leads
• Applications — opportunities
• Organization-to-contact connections — affiliations
• Contact-to-contact connections — relationships
• Course enrollments
• Program enrollments
• Other custom objects or third-party apps

Note: EDA doesn’t build functionality around leads and opportunities, however, teams tend to use these objects for recruiting and admissions.

Like NPSP, EDA promotes the use of a 1-to-1 account-to-contact model (the administrative account model), which in most cases helps in avoiding LDV issues like data skew that are discussed in Part 2 in this series.

Salesforce Advisor Link (SAL)

With SAL, the most common LDV problem areas are events (activities), tasks, cases, and alerts

Marketing Cloud

If you’re using Marketing Cloud, be sure to monitor the individual email results object for potential LDV issues.

Bucket accounts

Many nonprofits, and even some schools, use bucket accounts. A nonprofit may, for example, use a bucket account for one-off anonymous donations. Because these donations (Opportunity records) are not related to a contact, they tend to be assigned to a single “Anonymous Donor” account for rollup, tracking, and reporting purposes.

Over time, a large number of Opportunity records will be associated with a single Account record, leading to performance issues with rollups and other activities. Both ownership skew and account data skew can be caused by the use of bucket accounts .

Object bloat

Object bloat occurs when an object has more fields than needed. This can happen due to a lack of governance, a lack of well-defined business processes, or whenever you hear someone say, “We might need it one day, so let’s just create another field”.

Object bloat can lead to the following issues:

• Degradation in performance of the object because of the volume of data being retrieved
• Degradation in performance of Lightning components, Visualforce pages, SOQL, SOSL, and Apex code
• Timeouts when retrieving standard list views
• Load failures when the object is referenced in a related list
• Timeouts of Visualforce pages
• Hitting governor limits for SOQL and Apex code

Focus on prevention

An ounce of prevention is worth a pound of cure. For nonprofits and schools that have yet to start a Salesforce implementation, the first area to be considered is developing a data management strategy. If however, you have completed your implementation, you should still develop a data management strategy. This does not need to be an onerous exercise; it could be as simple as a policy document that defines how data should be handled and regular meetings of a data governance group with a well defined submission, review, and approval process.

It is more important to have policies on how data is managed. Such policies will not only help with having a better understanding of the data being held and potentially avoiding LDV, but can also help ensure compliance with relevant data regulations and laws. At a minimum a data management policy should consider the following questions:

• What data needs to be captured and stored?
• Where is the data being stored?
• How much data is going to be stored?
• What is the potential growth in data?
• How is data ownership going to be managed, both physically in the platform, and also from a business perspective?
• What is the delete/archive strategy? When will redundant data be removed?
• How will you manage and resolve record duplication?
• How is data moving between, in, and out of systems?
• Who needs access to what data? What is your security model?

Answering these questions will help you have a better understanding of your data and assist in building a solution that meets business needs, while helping to prevent LDV.

Summary

Now that you have a better understanding of LDV issues and how they manifest in nonprofit and educational institutions, continue with Part 2 in this series, which covers how to design an organization with LDV considerations in mind.

About the Authors

Chris Rolfe is a Customer Success Architect at Salesforce.org in EMEA. As a member of the advisory team he ensures our EMEA customers in higher education and nonprofit areas are successful in their implementation of Salesforce technologies, enabling them to achieve their mission. Connect with Chris on LinkedIn.

Richard Booth is a Customer Success Architect at Salesforce.org in EMEA. He helps nonprofit and higher education organizations make the best possible use of Salesforce technologies to deliver value and support their mission. Connect with Richard on LinkedIn.

Marie van Roekel is a Customer Success Architect at Salesforce.org, based in the Netherlands. As a member of the advisory team, she works with nonprofit and educational institutions to ensure they are successful in their implementations of Salesforce technologies and best practices, enabling them to better achieve their mission.