Protective Factors Survey Q&A

Jessica Sprague-Jones, PhD, is an Assistant Director at the Center for Public Partnerships and Research, University of Kansas, where she works as a researcher and program evaluator in the content areas of child welfare, family policy, and social services for children and families.

As a researcher at the University of Kansas Center for Public Partnerships & Research (CPPR), I have been teaming with FRIENDS National Center for Community-Based Child Abuse Prevention and others to revise the Protective Factors Survey (PFS).

The PFS is the only peer-reviewed tool measuring multiple protective factors within one instrument. It’s widely used in a variety of human service settings, including prevention and family support programs — at last count, in 40 states, and in a few other English-speaking countries internationally. The PFS is a valid and reliable twenty-item questionnaire that provides valuable feedback to practitioners by measuring five protective factors in one instrument: Family Functioning, Social Support, Concrete Support, Nurturing and Attachment, and Knowledge of Parenting and Child Development.

PFS scores are established through a pre- and post-test administration of the survey. Many social workers, home visitors, and other practitioners in the field are familiar with the instrument, but if you are new to the PFS, you can find out more at the FRIENDS National Center and this article describing its development and validation.

There’s a lot to love about the tool — many organizations value it for its brevity, the fact that it’s self-administered, and the fact that it’s free to use, which is really important for a cash-strapped organization. However, we’ve also gotten persistent feedback from the field about ways it could be better.

So we’re making it better!

Below are the most common questions I get from the field about the PFS and our revisions, and what I have to say about them.

Why aren’t I seeing much change pre- to post-test?

By far the most pressing concern we hear is that programs just aren’t seeing change from pre- to post-test. A variety of factors may contribute to a small or non-existent change in pre- to post-test subscale scores, including:

• Social desirability bias. Participants may be overrating themselves at the beginning of services, because they aren’t sure how the information will be used.
• Response-shift bias. Very often, we don’t know what we don’t know. Participants may overrate their own strengths and abilities at the beginning of services, because they haven’t yet learned how they could be doing things differently.
• There could be not much change to detect. The program may not be implemented with fidelity, and therefore not result in the changes desired. Or, service intensity and duration may not be sufficient to result in significant changes.

What we’re doing about it:

• Reviewing and revising the PFS, and field testing the revised instrument, to improve scale sensitivity. CPPR staff are currently at work addressing this issue, including:
• Creating more detailed guidance for administering the survey geared toward encouraging people to answer honestly and assuring them that the information they share will be kept private.
• Adding new items to the Social Support and Nurturing and Attachment subscales in order to capture greater nuance and change, both between participants, and over time.
• Revising the Concrete Supports subscale to focus on immediate concrete conditions (in the last 30 days, has the respondent experienced hunger, homelessness, unemployment, etc.), which could capture more incremental changes in physical and financial well-being
• Developing a retrospective pre-post instrument in which respondents are asked to assess themselves on the protective factors both before and after having received services. We hope that such an approach will offer a number of advantages:
• Reduce response-shift bias by asking respondents to reflect back on their own skills and abilities before they began receiving services
• Save staff and participant time by administering the survey at one sitting rather than two
• Offer a better match for short-term, low-intensity services
• Yield 100% matched pre- and post-test scores

What are you doing to ensure the revised PFS is a culturally competent tool?

This is one we get a little less often, but we’re taking it very seriously because it is centrally important to what we’re trying to do with the PFS. Concerns are that some items, particularly in the family functioning subscale, display cultural biases around how a family should function, and do not necessarily have a protective element. For example, many of the items in this subscale focus on talking as a family. This is a behavior that has pretty pronounced cultural differences. Is a family that doesn’t sit around and talk about their problems with each other because it’s not a part of their culture more likely to engage in abuse or maltreatment? We’re thinking no.

It’s a really tough puzzle, close to my heart and fun and challenging to parse out:

What is true family functioning?

What is at the core of good family behavior that will minimize the risk of abuse and maltreatment?

How do you ask about it in a way that minimizes cultural bias, including presumptions about family structure?

Our research suggests that other instruments are not doing this well, so we’re really charting new territory.

That’s why our new items have been subject to careful scrutiny with diverse focus groups. We’ve also pilot tested new items with samples that are oversampled for African American and Latino respondents, and run analyses to ensure that no items in the final revised survey produce significantly different responses by racial or ethnic identity.

When will the revised PFS be available?

The next step in our revisions is a national field test, throughout 2017.

Although we have pilot tested dozens of new items with a nationally-representative sample of over 300 respondents, we feel the only way to be truly confident that we have improved upon the existing instrument is to test it on its intended population: families engaged in services designed to improve protective factors.

This spring, we are launching a national field test among Community-Based Child Abuse Prevention programs. Depending on the results of that field test, we hope to release a validated PFS-2 in 2018.

If you’d like to participate in the field test, email me!

Will there be a Spanish version of the PFS-2?

A Spanish translation of the PFS, the Spanish Protective Factors Survey (S-PFS), was developed in 2012 and validated in 2014. Now that we are revising the PFS, we will need a Spanish version of the revised instrument. It is a strong possibility that a simple translation of the new PFS-2 items into Spanish will not result in a culturally competent instrument. Notions of proper family functioning and child-rearing are deeply cultural, and existing scholarly literature indicates that protective factors in Spanish-speaking cultures are different from those that have been found in the dominant English-speaking US culture. Given the need in the field for a culturally competent Spanish-language PFS-2, we hope to field a multi-stage research effort geared toward:

1) understanding the critical differences between protective factors for English- and Spanish-speaking families,

2) learning the special challenges Latino populations have faced using the current PFS, and

3) gathering information about how Latino respondents may interpret and respond to PFS-2 items.

So, yes, there will be a Spanish PFS-2, but it will be released later, after sufficient research to ensure a culturally competent tool.

Will the PFS-2 have optimal scores or benchmarks?

It has long been a goal to identify optimal scores or cut marks on the PFS that will help practitioners make sense of their PFS data. We have held back on making recommendations until we have a revised instrument that shows greater sensitivity to change, but we do consider it one of our ultimate goals in this revision process.

Stay tuned

As I write this, we are putting the finishing touches on the PFS-2 and preparing to begin field-testing with social service programs throughout the country. I will periodically post updates about how it’s going and what we are learning.

Thanks to Jessica Sprague-Jones, PhD, for contributing to Ideas in Motion! To learn more about Jessica and the rest of the staff at CPPR, visit our website.

Jessica earned her PhD in sociology from Indiana University, in Bloomington, IN. She is an Assistant Director of Research and Evaulation at the Center for Public Partnerships and Research at the University of Kansas, where she leads a variety of research and evaluation projects. Jessica spends most of her time with her husband, children, parents, sister, brother-in-law, aunts, uncles, and their respective dogs.

If you’d like to learn more about our work at the University of Kansas Center for Public Partnerships and Research, visit our website or follow us on Twitter.