Protocols

Neighbourhoods refers to large areas based on the City of Toronto’s formal division of areas (E.g., Dorset Park and Mount Olive).

Routes are smaller areas within neighbourhoods that we used for scheduling surveying sessions. Routes were developed based on number of houses to visit and the total area.

Letter and flyer are physically printed materials given to residents as a reminder to complete the survey.

Protocol are the procedures and processes for surveying the target population.

Residents refer to individuals who surveyors asked to complete the survey. In this handbook, we use residents unless otherwise specified.

Participants are individuals who agreed to complete the survey.

Replenish is a meeting between the survey coordinator and surveyors to exchange necessary materials and formulate solutions to any survey-related challenges.

Replenish Protocol is an outline developed for each replenish that outlines which surveyor is responsible for which route and what materials are needed from the survey coordinator.

What were our protocols?

Surveyors will encounter many different situations making it necessary to develop protocols that include the information and procedures they need for each unique situation. While we describe the protocols that we developed in this section, we recognise that organisations interested in this topic may need to develop their own protocols tailored to their goals and the communities they will be surveying.

We recommend the following considerations when developing the surveying protocols:

Not only for safety purposes but also logistically, we have found that having surveyors work in pairs makes surveying more enjoyable and efficient. The one exception that can be made to this recommendation is having community researchers survey visit the neighbourhoods they live in alone. Community advisors also play a crucial role in giving context around neighbourhoods that allow us to keep surveyors safe and use our time in the field efficiently.

We have found that surveying before 5:00 pm yields a low response rate becauseof working hours. We also recommend stopping surveying before or shortly after sunset for safety purposes. Keep in mind the sunset time changes depending on the season, which will influence how much time surveyors spend in the field and survey responses. Our weekday shifts were approximately three hours long.

We have found the highest response rate on weekends when we visited households after lunch time. Shifts were approximately six hours long on weekends with a half hour break in between.

We provided a $20 incentive to participants as cash or an online gift card. We did not mail any gift cards or cash because we wanted to track who received our incentives, which required an in-person or online transaction.

For residents who were not available at the door, we left letters and flyers that provided all the information they needed to complete the survey online. We expected an average of two to five responses per three-hour shift, and a roughly equal amount completed online by using the letters or flyers. Click hereto jump to the resources section.

We recommend creating a one-stop-shop handbook for surveyors with all the information and procedures they need to respond to the different situations they may encounter when in the field. Click hereto learn more about the field handbook.

We developed our protocols through discussion with community members of the areas we intended to survey, as well as organizations that regularly survey or canvass in Toronto’s suburban communities. We developed two protocols that we describe below: field surveying and high-traffic protocol.

A. Field Surveying

Field surveying was our most randomised approach. Surveyors used physical and digital materials (e.g., route handouts, letters, flyers, and Google Sheets) to visit each household in their route. Click here to learn more about the resources we used. Surveyors were prepared for four responses:

When residents agreed to complete the survey, surveyors found the resident’s unique code available on the Google Sheets to enter on the survey screen, and then gave it to the resident. Once the survey was complete, surveyors filled out a receipt, gave the resident a copy of the receipt with the cash incentive, and thanked them for their participation. Surveyors also recorded the response on Google Sheets.

At least a quarter of residents refused to do the survey each shift. In about half of these cases, we found that a “no” response meant “not now.” Therefore, surveyors responded to “no” by asking residents if there was another time that might work better for them, or if they preferred to do the survey online. In most cases, residents agreed to meeting at a different time. In some cases, however, a “no” meant a “no”; surveyors thanked these residents, gave the letter or flyer just in case they changed their mind, and left.

Some residents mentioned immediately after opening the door that they are unavailable at the moment and surveyors should come back later. In these cases, we followed the protocol for the “no” response to schedule a time convenient for the resident. In other cases, residents preferred to complete the survey online, and surveyors gave these residents their letter or flyer. We monitored residents’ responses in the survey software; if they still did not complete the survey, surveyors visited residents again to remind them to complete the survey.

For approximately half households, residents did not open the door. Surveyors marked these houses in their route handout and visited them again at the end of the same shift. We found that visiting each house at least twice within the span of three hours doubled the number of opened doors. However, about a quarter of houses still did not open their doors at the second visit. In this case, surveyors left a letter for residents to complete the survey online. Surveyors visited the same route on a second day to repeat the same process (i.e., visit 1 at the beginning and visit 2 at the end of shift). In total, surveyors knocked on each house at least four times spanning two days. If residents still did not open the door at the fourth visit, surveyors left a flyer for them to complete the survey online. Surveyors also noted if the letter they left in their mailbox on the first day was still in their mailbox or had been taken by the resident.

We surveyed during the 2019 election season. We found that in some areas, residents were more skeptical in opening the door perhaps because they confused our surveyors with political canvassers or campaigners. In such cases, surveyors had to change their script to emphasise that they were not a part of any campaign team. University of Toronto apparel such as shirts and lanyards also helped. This example illustrates how certain protocols may need to be modified based on the community expectations or certain events. It is essential for surveyors to be mindful of the various changes that may affect response rate. At the same time, our survey findings showed that the election period did not change resident responses to survey questions.

B. High Traffic Protocol

We used a second surveying approach that we called the high-traffic protocol (HTP). The aim of this approach was to have surveyors stand at strategic locations –such as grocery stores, community hubs, libraries, and apartment buildings –and approach residents as they entered or exited the location. We used this approach to achieve the following two goals:

  1. Increase the overall response rate
  2. Increase the representation of certain groups

We recommend testing HTP in various locations. Pilot testing allowed us to develop guidance that increased the response rate of HTP. We developed the following guidelines:

  • You can start the conversation by saying either “do you want to make a quick $20” or “we are looking to improve how neighbourhoods in Scarborough, and we need your help.”
    • Use the $20 gift card as a “hook” for them to listen to you. Without saying the $20, the number of negative responses will increase.
    • You need to mention that you are from the University of Toronto to legitimise your presence.
    • You should approach residents with a smile and a welcoming demeanour. Be approachable and make eye contact.
    • You should not wear your bags while interacting with residents. Store them in a secure location.
    • Possible script for HTP: “Hi, I am a University of Toronto student. We want to know your thoughts about healthy neighbourhoods/Scarborough, and we will give you $20 to complete a quick survey.”
  • Do not interrupt if someone is talking on the phone. If someone is talking to someone else beside them, then you can interrupt.
  • You can approach residents who are standing around (e.g., waiting for a ride).
  • Since you are in pairs, each person can stand on either side of the entrance. Do not stand beside each another. Also, each surveyor should be talking to different residents.
  • Approach residents who are leaving rather than entering. Residents are more often in a rush when they are going inside, then they are leaving though this may depend on the location so you can modify accordingly to match your context.
  • Approach every fifth individual that leaves the exit. If an individual is in a hurry, it may be best not to approach them because it is likely they will give a negative response.

We selected the HTP locations through discussion with community members and viewing Google Maps. There were some challenges that we encountered when implementing HTP. The major challenge was ensuring that interact with residents from inside our sampled neighbourhoods. We minimised the number of residents from outside sampled neighbourhoods by conducting HTP at locations within the neighbourhoods and focusing on local locations such as grocery stores and libraries rather than large malls. However, it is difficult for surveyors to identify if a resident resides in the area unless they ask the resident. It might be essential for surveyors to ask the resident if they reside in the area. We do not recommend asking specific questions that might arouse suspicion; instead, we recommend conducting HTP at local locations, and asking general questions such as “we are interested in getting responses from those living in this area. Do you live around here?”

HTP is also not a fully randomised protocol compared to field surveying, but there are certain strategies that can make it quasi-randomised by approaching every fifth resident and spending an equal amount of time at different types of locations (i.e., restaurants, community centres, libraries, grocery stores).

C. Replenish

Once per week, we invited surveyors to a meeting we called replenish. Prior to each replenish, the survey coordinator created a replenish protocol that outlined who needs to attend the replenish, what materials they need, and the protocol for transferring materials between surveyors. At the replenish, each surveyor (1) received route materials (handouts, letters and flyers), (2) received cash incentives, (3) received access to the Google Sheets they will need to use in the next week, (4) provided the survey coordinator with completed receipts, and (5) asked any questions about the schedule or challenges in the field.

There were four types of materials we exchanged with surveyors at replenish:

  • The survey coordinator organised mails, letters, and route handouts for each surveyor based on the replenish protocol.
  • If any surveyor was supposed to transfer route materials to another surveyor, then the coordinator ensured that both surveyors were on the same page.
  • At each replenish, we first discussed how much money each surveyor had.
  • Per surveying shift, we expected 0 to 4 responses, which meant surveyors needed up to $80 per shift. This number was multiplied by the number of days the fanny pack will be in the field until the next replenish.
  • When the survey coordinator gave cash to surveyors, both signed a transfer form and recorded in a paper log located inside the money box that stored all the cash. Each moment cash, receipts, and/or fanny pack and materials was transferred between surveyors or survey coordinator, we completed the form to document the exchange. Surveyors signed the form and sent a photo of it to the survey coordinator. This strategy promoted accountability of all surveying materials when in the field.
  • When surveyors ran out of cash, they gave gift cards until the next replenish.
  • At each replenish, the survey coordinator received all receipts from receipt books.
  • The survey coordinator also ensured that each receipt was an actual receipt and not a duplicate. The coordinator removed duplicates by recording the household ID associated with each response on completed receipts.
  • Each tablet was associated with a number from 1 to 4 that corresponded to one of four Google Drive Gmail addresses.
  • At each replenish, surveyors ensured that their tablet has access to the Google Sheets.

D. Field Reflections

Our surveyors visited neighbourhoods with a considerable diversity of ethnicities, races, and backgrounds. It was essential for us to document and/or discuss their experiences interacting with residents. We developed a Community Voices Reflexive Log with the following prompts to be answered after each shift:

  1. Who answered the door?
  2. Who accepted/declined the survey?
  3. Describe your interactions with resident(s)
  4. Describe the characteristics of resident(s)
  5. Describe your impressions of the field handouts and surveying protocols
  6. Describe the neighbourhood you visited today
  7. Discuss how your social location influenced your interaction with residents

The survey coordinator conducted a thematic analysis of all entries in the reflexive log in the first survey phase and found the following:

These findings informed how we modified our protocols for the second survey phase. In addition, we used these findings as practical examples our experiences in the field, including what worked and what did not, and why.

  • Residents in apartments were more aggressive and skeptical of the survey compared to residents in houses. Residents in apartments were skeptical when the $20 cash incentive was offered to them.
  • Older residents were difficult to reach. They had a harder time using the tablet and answering some questions; surveyors walked some residents through the entire survey. Older residents are also unlikely to have an email address for the gift card option, so we recommend surveyors having cash on them for this population.
  • University of Toronto name tags and lanyards, as well as surveyors’ status as students was a successful strategy for engaging residents in conversations about their neighbourhood.
  • The $20 was less of an incentive, and more of a hook to prevent residents from immediately closing their door.
  • Some residents seemed quite interested in the goals of the survey study. It was important to appreciate their enthusiasm in our work.
  • Have surveyors work in pairs, survey after 5:00 pm on weekdays until sunset and after 2:00 pm on weekends until sunset to optimise survey responses.
  • Provide monetary incentives to resident and process them in-person or online.
  • Develop a field handbook that includes all information and procedures surveyors may need to respond to different circumstances.
  • Develop protocols to match the needs of the project. We developed two protocols: field surveying (door-to-door surveying in neighbourhoods) and high traffic protocol (meet-and-complete surveying at high traffic locations). Host weekly or biweekly “replenish” meetings that provide surveyors with the materials they need to survey. These meetings will also provide surveyors with an opportunity to voice any challenges.
  • Incorporate a way for surveyors to document their reflections and experiences in the field. These reflections will help in improving the surveying process
Previous Section
Next Section