Instructional Design in the Wild: The Charlottetown Survey on Short-Term Rentals

This is a blog that I wrote for my own Medium but decided to also share here. I saw the City of Charlottetown’s Short-Term Rental Survey and some of the response its received on social media. I decided to take a look at the survey, some of the response, and think about survey design.

Design is everywhere, and instructional design is part of that. Instructional design comes into play for conveying information, assessment and evaluation, and establishing learning relationships. With the response to the City of Charlottetown’s survey on short-term rentals I thought it would be a great time to look at the survey as an assessment. I’m far from an expert in survey design (I’ve only just finished my intro to research methods course), but this can be practice for me as well.

My background in assessment design comes a lot from my bachelor of education experience, particularly the book Classroom Assessment for Student Learning by Chappuis, Stiggins, Chappuis, & Arter. For both selected and open response assessments they start with the same design framework based on 5 keys:

1. Clear purpose
2. Clear targets
3. Sound Design
4. Effective Communication
5. Student (Participant) Involvement

For this post I want to focus on purpose and the design of the survey itself.

Clear Purpose

For clear purpose we want to establish what information we are looking for, who will use the information, how will the information from the assessment be used.

In regard to the survey, we actually had an interview with one of the key stakeholders of the survey. Zilke says: “Essentially the whole purpose of the survey was to basically be open-ended to just garner a lot of feedback from the public”. The input from the survey will also be used to prepare recommendations for city council in a meeting later this summer. This is also communicated in the introduction of the survey. In the introduction we see the targets that should define how the questions are written: whether there is any change to the bylaws in regards to short-term rentals. Whether short-term rentals should be allowed, and what rules should be created.

This is a broad spectrum of targets, but by providing space for input and follow engagements to incorporate that input this can be a fine starting point.

Sound Design

We have established our goals and from that we would want to develop our tool. The city ultimately went with a survey that is available in both online and print formats and that is a good way for that broad open-ended feedback goal. It also means that the data from the survey can be processed quickly which is important as the recommendations are expected to be presented in June.

For an online survey we may also have security concerns. You may want to have respondents supply a valid email to complete a survey depending on the value and stakes. For this survey there is one response allowed per IP address, while this will likely cut down on a person’s ability to complete the survey multiple times by creating fake email accounts, it may cut down on responses from a single address (e.g. participants replying from a public computer in a library, members of the same family, roommates in the same apartment, etc.) There are going to be trade-offs here and the decisions made should be considered for how they may impact results.

Survey Design

With this survey we want to establish an understanding of who are respondents are, whether they believe that the bylaws should be changed, whether short-term rentals should be allowed, and if they are permitted what regulations should be put in place.

For our questions there are three primary types:

•Open-ended questions — questions where the participant provides a response to a question. These are great for when you don’t or can’t provide a reasonable spectrum of answers. This is a great type of question for Zilke’s goal of garnering a lot of feedback from the public, especially if you plan on capturing a lot of ideas, evaluating them, and doing more research.

•Close-ended questions — questions with answers selected from a provided list. This might be to choose between distinct options or having respondents respond on a scale (e.g. the Likert scale). Closed answer questions are good for providing quicker turn around for surveys with lots of participants. By providing the answers you save time from having to code participant responses, but you want to make sure that you adequately cover the responses your participants could have. This type would be great for determining what part of your sample the particular respondent belongs to. Closed-ended questions are also good for selections of options. We have a goal in our survey to determine whether or not bylaws should be changed, that could make for easily designed closed-ended question. Keep in mind that if you keep a question closed-ended, if it hasn’t been designed carefully, it may frustrate or disengage your participants if they feel unrepresented in the options and do not have a way of letting you know what their actual preference is.

•Semi-closed-ended questions — These are questions where you provide a set of answers, but make space for respondents to provide their responses. You’ve probably seen this as “Other (please specify)” in other surveys. Here you can get the benefit closed-ended questions and open-ended questions. This is kind of a safety check if you want to use a closed-ended question, but aren’t sure you’ve covered your bases. If you haven’t properly prepared a closed or semi-closed-ended question, you may be better suited to employ an open-ended question to provide more space and recognition in the event that “Other (please specify)” is a popular response.

In terms of our goals, my suggestion would be a survey with this kind of lay-out:

Introduction (provides the purpose, goals, and background to the respondents)

Closed-ended questions for identification of the sample/stakeholder (e.g. which categories you belong to). Ideally we will have an understanding of who are stakeholders are by the time we are ready to send a survey to print. Having these be closed-ended questions should let use quickly quantify the results based on the groups inside of our sample.

Closed-ended questions about whether short term rentals should be allowed. This goal has big implications, but is honestly represented in a closed-ended response that can quickly provide the council with an idea whether a significant amount of their constituents want to see action on the issue.

Semi-closed-ended and open-ended questions for regulations if the participant believes short-term rentals should be allowed. This survey is a broad focus, but if the goal were to survey the public about their opinion on options that the municipality was considering you may move this to closed-ended and semi-closed-ended questions since that would better reflect how the city would respond to the survey results.

A look at the survey itself

So let’s take a look at the survey as it was run. It’s a cross-sectional survey made up of 2 open-ended questions, 5 closed-ended questions, and 5 semi-closed-ended questions.

City Council has recently directed staff to review the issue of short-term rentals in the City of Charlottetown. Short-term rentals are defined as the rental of a dwelling unit or a portion of a dwelling unit for a period of less than 30 consecutive days. The issue of short-term rentals has a number of different implications for residents, homeowners, renters and the tourism industry. This survey is intended to collect feedback from all the various stakeholders who have an interest in this issue. Depending on the outcome of the survey and future public consultation, City Council may adopt regulation to address the concerns expressed through the public consultation process.

Feedback collected will be used to inform and shape subsequent public consultation methods, regulation, and policy. The survey is anonymous and should take less than 15 minutes to complete. The deadline to complete the survey is Friday, May 31 at 5 p.m. Only one survey submission per IP address will be accepted, duplicates will be removed.

Thank you for participating! If you have any other feedback you’d like to provide, you can email us at planning@charlottetown.ca

Here we have an introduction, it provides a little bit of feedback regarding what a short-term rental is and that the goal is to collect feedback from all stakeholders. One thing that would have also been useful here (either summed or linked would be a description of the current regulation in place for short-term rentals if any are in place. A change of bylaws may mean more or less restriction on short term rentals and understanding what is already in place is important.

Leaving out that information may let misunderstanding impact the results and you’re more so relying on participants to bring their own interpretation and experience to the survey, even though that they experienced may not be permitted by the city.

1. What type of resident are you? (Select all that apply)

Full Time

Seasonal

Vacationer

Commercial

Owner

Renter

Here is a closed-ended question about identifying what kind of stakeholder the participant is. It lines up our suggestion

2. Have you ever advertised your dwelling unit for short-term rental?

Yes

No, but I am considering it

No, I don’t plan on it

Another closed-ended question about identifying our participant.

3. What do you think the benefits would be if the City of Charlottetown changed the rules to allow people to rent out their homes to tourists on a short-term basis? (check all that apply)

Creates additional income for owner(s)

Creates a more vibrant tourism industry for the City

It helps owners afford to live in their homes

Some local jobs are generated

If property zoned and taxed, it contributes to the overall tax-base

Other (please specify)

I think this is a good question if we’re going to have participants weigh the pros and cons of short-term rentals and then follow up with our goal for identifying whether stakeholders want to see changes to the bylaw. One thing that I notice is that it seems to imply that short-term rentals are not allowed under the rules. This would be great information to have in the introduction.

The City has heard there are concerns with allowing short-term rentals throughout Charlottetown (a short-term rental is usually a few days or weeks, but less than one month). If you have concerns, what specifically are they? (check all that apply) If you don’t have concerns, please indicate that in the last box below. Please rank your concerns using the drop down box beside each response (where 1 is your top concern).

It may drive up local housing/rental prices []

It decreases the housing supply for local renters []

Guests contribute to parking congestion []

The character of neighbourhoods is threatened []

Guests don’t know how to dispose of garbage/recycling []

Guests are noisy []

May increase traffic in the neighbourhood []

I don’t have concerns

Other (please explain)

So here we have the con side of the previous question. One thing that is strange here is there is a significance scale that wasn’t attached the benefits side. Maybe that would be something to include in question 3 as well. Two more changes from the benefits question is the addition of a “none of these” style option with “I don’t have concerns”, and the hedging on some of the replies “May increase traffic; may drive up local housing”. Since we’re writing this retrospectively and there have been complaints about perceived bias in the questions so the framing here may have contributed to that. Having these two questions mirror each other more, both in hedging and the significance rating may have quelled that and might have also provided a bit more insight into the public’s opinion about possible benefits.

5. Have you ever been disrupted (i.e. noise, lack of parking, security issues) by short-term rental(s) in your neighbourhood?

Yes

No

Here we have another con style question, this may be identifying a population in the sample, it may also leading into some of the regulations that should be placed on short-term rentals. I would have liked a summary question following the pros and cons lead up that would have given some data about the number of participants who thought that existing laws are fine, need to be tightened, or loosened.

6. Please explain how your quality of life has been negatively impacted?

This is a follow-up to 5, one of our 2 open-ended questions. This is a great space for participants who have answered in the affirmative to provide their responses which the city may not otherwise be able to properly reflect.

7. The current Zoning & Development By-law regulations don’t allow everyone to rent out their home to tourists for short-term rental purposes. Despite this, there are more and more unregistered short-term rentals in the City of Charlottetown, helped by the availability of websites such as Airbnb, TurnKey and VRBO. What do you think the City of Charlottetown should do about this issue?

Allow short term rental(s) in all residential areas, but make rules to protect the integrity of the neighbourhood

Leave tourist homes (short term rentals) where they are (single detached homes) and add more resources to be able to enforce current regulations

Do nothing

Other, please provide suggestions

Here we get a little bit of information about the existing laws. This would have been much served in the introduction in my opinion. The information here seems to be presented unclearly to me “The current Zoning & Development By-law regulations don’t allow everyone to rent out their home to tourists for short-term rental purposes.” Who is allowed? Or how do they become allowed? The answer seems to imply that only single detached homes are allowed to be used for short-term rentals, but again I think more background would help. This is the summary style question I had mentioned following question 5.

That said our goal in our introduction was “As part of the public consultation process, this survey is one opportunity provide input into whether short-term rentals should be allowed throughout the community and if so what rules should be created to ensure the integrity of the neighbourhood is maintained.” Our responses to the question don’t seem to match that goal statement. One is to change bylaws to allow in all residential areas, but create regulations, the second is to not change the rules and enforce them, and the third is to do nothing. We have an other option, but I think we could have designed this question to better align with the stated goal and not have to opened the door to a lot of answer coding. Coming up we have question 10, that may have been better here with a follow up question of whether existing bylaws should be loosened, tightened, or unchanged.

8. Renting homes for short periods of time to tourists on Airbnb has been said to contribute to the decrease in availability of rental housing. How can the City of Charlottetown help prevent short-term rentals from negatively impacting the housing supply (check all that you feel should apply)?

Apply the same rules to short term rentals as bed & breakfast or hotel operators

Only allow short term rentals a specified amount of time each year

Restrict short term rentals to single detached dwellings

This question I think is really weak. First we have a lack of background again. What do the rules for bed & breakfast or hotel operators entail? Additionally, as a broad survey this question likely would have benefitted from being an open-ended or semi-closed-ended question. A closed-ended question like this is probably best served after broad information has already been gathered and the councillors are deliberating between set recommendations.

9. If the City of Charlottetown were to introduce a limit on the amount of time for short-term rentals, would you support?
Year Round

Seasonally Only

Other

This is a semi-closed-ended question looking at one facet of regulation, specifically looking at time. I think it’s fine, maybe tighten up the wording of the question to “If the City of Charlottetown introduced a limit on when properties may be available for short-term rentals, would you prefer short-term rentals be available:

All Year

Summer seasonally (Start Date, End Date)

Other (please specify)”.

10. In your opinion, what type of dwelling should short-term rentals be permitted in (check all that you think should apply)?
Single detached dwelling

Semi-detached and/or duplex

Townhouse Unit

Apartments

Should not be permitted in any type of dwelling unit

Other

I mentioned that I think this question should come up earlier to better break up and balance a question. This question could be used for understanding whether the bylaw should be changed depending on how respondents reply as well. This is also the final question regarding regulation. It should be mentioned that a follow-up regarding principal residence rental or multiple property rentals was petitioned for by a critic of the survey. If the city has been aware of that facet it may have been worth reflecting in the survey or explaining in an introduction. Additionally I think this survey may have better met its goal with providing an additional open-ended question for participants to regulations not tied to time or type of dwelling.

11. Are you a resident of:

Charlottetown

Cornwall

Stratford

Other

This is a very strange question, it should probably be among questions 1 and 2. Also Cornwall and Stratford have their own municipal government and bylaws. Perhaps this question is best moved up and asked whether the respondent is a principal resident of Charlottetown or not. It could also be set like this:

You are a resident of:

Charlottetown

Prince Edward Island, but not Charlottetown

Canada, but not PEI

Other

This may better group your stakeholders as someone who may have properties to rent, who may rent properties, or be impacted by rental properties.

12. Please share any other thoughts, ideas or suggestions you may have regarding short term rentals in Charlottetown.

A final open-ended question. A good practice to have for additional comments. Without question 11 this may have been better understood as opening a door for additional facets for regulation, if necessary.

One last piece with this survey that is recommended for design is piloting the survey with stakeholders to evaluate it. With the publication of that CBC article from someone who was saying critical elements are missing and the responses may be weaker the survey would have likely benefitted from a small test with some contacts of the stakeholders the survey is meant to represent.

Overall I thought this was an interesting piece that came up following a tool that many of us in education may take for granted (I know I do myself). The fallout here is likely representative of the higher stakes nature of the element but can provide a lesson for how to improve our practice. I know that going forward I’ll be thinking about the goals for my surveys, ensuring my questions are aligned with those goals, and getting feedback from stakeholders about questions.