Moving Forward with Barnes VR: Lessons Learned and Measuring Impact

If you are just joining this discussion about this project, get caught up on previous posts. This one will detail what we learned, how we measured, and moving how we’re going to move it forward. Spoiler alert — this medium-touch approach successfully introduced new audiences to the Barnes, cut costs dramatically, and had what we consider is a high rate of return.

I’ll be the first to admit these results surprised me.

Measuring Impact

This program had a 31% return rate measured by those who we engaged at libraries then coming on the Barnes field trips. Our highest return rate — 56% — was from Durham in West Philadelphia where we had a super engaged librarian. Our lowest return rate — 22% — was from South Philly when it was pouring rain on the field trip visit day.

Field trip participants were surveyed at the end of the visit. 80% represented first time visitors. An additional 14% represented visitors who had not been to the Barnes in more than two years. Our aim with this program was to engage populations reflective of Philadelphia and we succeeded. Participants reported — African American/Black 39%, Asian/Pacific Rim 4%, Latinx/Hispanic 23%, White/Caucasian 23%, mixed race/other 9%, declined to answer 2%. Additionally, 68% of participants reported their yearly household income was below the Pennsylvania average of 56k.

Field trip participants were invited to give us feedback via discussion over a light lunch and snacks. Overwhelmingly, the trends in this dialog pointed to the VR — and specifically the ability to see what the Barnes is like — as a primary motivator for visitation. While we were not able to record and transcribe every feedback session, we did get a lot of it and you can read it here.

When compared to other high-touch community engagement programs the Barnes has completed within the last year, the per-person cost of this medium-touch project was 45% lower. The higher-touch programs are extremely important because they represent the bedrock of our community engagement efforts. However, one of our biggest learnings was centered around the idea that a medium-touch program can also be effective and it has the ability to scale to larger numbers of people. While a medium-touch does not replace the high-touch, the two types running along side each other have helped us grow deeper relationships with our neighbors. A variety of depth in programs support each other and give us the chance to provide different experiences for those who want to engage in different ways.

After touring with our teaching team, the group is invited have lunch with us. We use this time to gather qualitative feedback, survey demographics, and distribute “Art For All” community cards good for free visits all year. We also bring the VR headsets because participants expressed wanting to revisit that experience and share with their guests, after they’ve seen the galleries.

Lessons Learned

Take what currently works well and expand it. The model for this program is based on our K-12 education plan where Barnes educators travel to public schools to introduce students to the collection ahead of their visit to the Barnes; these students then enter our galleries with a sense of familiarity and comfort. We took an already successful practice and applied to adult learning in community spaces. Because feedback on this current program showed the VR was a primary motivation for those visiting, we are now circling back to test it in K-12 education — both with teacher training and student experiences.

We produced a flyer helping advertise the program, but it’s really hard to explain what VR is and how the program worked. We quickly shifted to setting up in high traffic areas and letting attention do the rest.

The experience worked best as a drop in experience. We started by trying to organize specific times and advertise that to library patrons using flyers. Turns out, it was pretty difficult to explain the program and there were few people signing up for sessions. We quickly shifted our strategy to showing up at times when we knew the library would be busy. We’d set up in high traffic areas and participants would just drop in with the headsets helping draw attention. Once we had one person using a headset, others felt comfortable coming over asking what was going on.

The program worked really well as a drop in experience at high traffic times the library was being used. Shown here, the Barnes VR team is a Lillian Marrero Library in North Philadelphia on a day when Philabundance is also running its weekly food assistance program — combining the two brings helps our program fit into the life of library.

The simpler the VR experience the better. We made a choice to let participants guide their own experience — nothing fancy, just 360 degree photography with hot spots, so people could wander. It was easy to use — “here, just put this on, look at the circles to move around, and tell us what you see.” The participants’ interests — those things that that caught their eye during the VR experience — were used to shape the in-gallery visit.

Create a social experience. The same teaching team was used for both the library visits and the field trips. It was clear that participants knew our team upon arrival at the Barnes — this made for an extremely warm reception. This also helped us get needed program feedback — qualitative and quantitive — at the end of the visit because participants trusted us. Participant feedback indicated that it wasn’t just the Barnes team consistency that was powerful — the ability to tour with their neighbors was also cited as a big win. Removing the VR headset straps helped create an easy path toward conversation because it was easy to take the headsets on and off. Lastly, our field trip invitation included the call to bring as many friends and family as participants would like. This created a needed support structure — the Barnes visit became an outing for entire families and groups of friends.

Taking the straps away leads to a more social experience. We’d most often see people put them on, explore, and then take them off to talk to our teachers. This was our meetup at the South Philly neighborhood library.

Create an onsite experience that mirrors the virtual one. Using the VR, participants could wander and talk with teacher guide. Similarly, during the museum field trip we encouraged participants that they could wander the galleries, join discussions, or do a combination of both — they were in charge of their own experience.

Timing counts. Our visits to libraries happened at high traffic times often when other things were going on — this helped us catch people in the flow a visit. Additionally, we found that Saturday mornings were the best times for the Barnes field trips. Time and time again, participants in feedback sessions told us Saturday morning was a great time. This window was also helpful for us when utilizing public transportation — there’s less traffic on Saturday mornings and busses are usually running on time. As a result, it was easy to coordinate the onsite visit because we could predict the arrival time.

Food helps, but escorted visits are really a big factor. After every field trip, we’d go have pizza and talk with participants to get their feedback. We figured food would help just given the duration — 1 hour for transportation, 2 hour visit, then 1 hour to get home. Feedback indicated that food was helpful, but the escorted visit via SEPTA transit encouraged participants to come along because it was so easy.

Never underestimate the power of an engaged librarian. In our prep work, we visited sites and spoke with librarians — they are the glue of their communities. If the librarian liked the program, we’d bring it to their location and, if they were hesitant, we’d move on. Engaged librarians welcomed our team and helped create a bridge between the Barnes staff and their constituents. Some librarians also came on the field trip to the Barnes and it was so clear to us that this also helped participants feel comfortable on site.

Alberto Pagan (front and center wearing the red shirt), the librarian at the Charles Durham Free Library, was our biggest cheerleader — helping us both at the library and coming on the visit with his community. His group had the largest return rate at 56%.

Bilingual staff make all the difference. The ability of our project coordinator to quickly switch languages on the fly was critical for engagement. When the program team visited Lillian Marrero — a neighborhood library in a primarily Spanish speaking area—we avoided hiring a translator and, instead, we specifically engaged a teacher who knew our collection and could speak Spanish. This helped create a direct connection between project staff and library patrons.

Alicia Mino, Part-time Community, Youth and Family Program Coordinator at the Barnes, guiding Spanish speakers through the VR experience at the Lillian Marrero Library in North Philadelphia.

The same teaching staff is used both offsite and onsite. Alicia, seen here, is touring Spanish speakers through the Barnes during the Lillian Marrero Library field trip.

Seek locations to grow already established roots. When scouting locations for this project we looked at SEPTA routes — how easily we could get the group to the Barnes using public transit — and combined that with areas where we had other community programming taking place. The latter was especially important because we could continue relationships and provide a new experience. More than a few times we’d hear from participants that they were a part of another program we were doing. As an example, our engagement team would be running a VR program at the library and see constituents from Barnes summer camp programming, which took place at nearby recreation centers. In this instance we often engaged the parent in the VR while their children might have experienced the summer camp. Then the whole family would come on the Barnes field trip. We found strategizing in this way helped grow our relationships in neighborhoods very naturally.

We tested this program at neighborhood libraries and, also, a recreation center like this one, Athletic Rec in Lower North Philadelphia. The participant feedback and return rate were very similar at both types of locations. This points to potential successful partner expansion in 2019. Athletic Rec was selected because we had a very supportive rec center leader and the Barnes had previously done summer camp programming at this location continuing our relationship with this community.

Create a path of return. At the end of each visit, we’d hand out materials about various programs including PECO Free First Sunday Family Day and we’d also give out “Art for All Community Cards” which allow free return visits for the next year for the participant and up to three friends or family. Each card is coded with a number which tracks the program’s rate of return, so we can see rate of return and from which library location. Lastly, we’d give out step by step instructions on how to use public transit. This would help the participant get home on their own and return later while having a quick reference.

The “Art for All” Community Pass is tangible card given to participants for free visits over the next year. The cards are all numbered and assigned to specific programs giving us a sense of return rate. We specifically went with the card type that our local YMCAs use, a laminated card which can go in a wallet or be attached to keychains.

Our “Art for All Community Cards” are given out at all medium to high touch community programs. This includes the VR program, summer camps with recreation centers, families in our “PNC Grow Up Great” program, and recipients of our needs-based scholarship program for adult classes. We are currently seeing participants using the cards at an 11.67% return rate across all programs. We think this is an exceptionally high rate of return especially compared to the paper flyers we used to produce, which had a return rate of practically zero. The very tangible feel to these cards combined with giving them out during high value, interactive programs has helped spark participants to come back after their initial visit.

Moving Forward

This pilot project was funded by the Barra Foundation as part of their Catalyst Fund and we’ll be moving it forward throughout 2019 under our Knight Center for Digital Innovation in Audience Engagement. As we do, we’ll be looking to create greater efficiency — the program as it stands is fairly resource light, but we’d like to see it being run by a single teacher and project manager. We’ll be looking at transportation options — SEPTA worked well, but a shuttle that goes to several locations for pickups might work even better and help us cover more ground. Lastly, we are considering many, many more partners — we’d like to continue the program at the libraries while expanding it to other community partners with physical locations throughout the city of Philadelphia.

The Barnes VR in Neighborhood Libraries pilot project was funded by the Barra Foundation as part of their Catalyst Fund. The program will move forward throughout 2019 under the Knight Center for Digital Innovation in Audience Engagement at the Barnes. Read more about the project in this series of Medium posts.

Project Team —

Barnes Foundation: Shelley Bernstein, Consulting Creative Technologist; Steve Brady, Chief Technical Officer; Ana Gamboa, Special Project Coordinator; Ann Moss, Art Team; Jihan Thomas, School Programs Outreach Coordinator; Alicia Mino, Youth and Family Program Coordinator; and Barbara Wong, Director of Community Engagement.

Free Library: Andrew Nurkin, Deputy Director, Enrichment and Civic Engagement; Kalela Williams, Director of Neighborhood Library Enrichment Programming