Surveys and Data Collection
Our research team develops customized surveys and stays up to date on industry-recognized survey methods, such as computer-assisted telephone interviewing (CATI) and web-based techniques, to obtain quantitative data. We also use audio computer-assisted surveys, in-person field follow-up surveys, meticulously designed focus groups, and carefully adapted interviews. A single method of evaluation may not adequately capture the breadth of a project’s work or the impact that it is having, so for many projects, multimethod approaches are essential. There are many online surveys out there but it can be hard for someone to know just which ones they should go for.
For examples of relevant projects, see our project archive or contact DIR.
Instrument Development
The instruments that we design range from large-scale CATI surveys to focus group protocols, in-depth questionnaires, and inventory forms to support data-extraction activities. We perform cognitive interviewing and testing to identify and eliminate potential problems with questionnaire design, awkward wording or unfamiliar terminology, confusing response categories, and sequencing. This testing assures validity, reliability, and bias reduction.
Survey Methods
The collection of data by survey-including in-person, mail, and computer-assisted telephone interviewing (CATI)-is an integral part of DIR’s research. Depending on the needs of the project, we use one method or a combination of the following data collection methods:
In our Houston-based state-of-the-art CATI center, we conduct both large-scale complex and short-term quick-turnaround surveys. The 40 work-station center is open seven days a week and is staffed with experienced English-speaking and multilingual interviewers. Our supervisors conduct audio monitoring to assure quality and accuracy during data collection. All interviewers are trained, tested, and approved in basic interviewing skills, confidentiality, interviewer bias, refusal conversion techniques, and quality assurance monitoring.
Web surveys are highly efficient and effective ways to interview certain groups and ensure comprehensive and accurate data. Our web survey software provides real-time online data collection and reporting. Using mixed-mode strategies, we often combine the convenience and cost-effectiveness of web surveys with follow-up by telephone, fax, and e-mail to ensure high response rates.
Self-administered surveys include mailed surveys, teacher assessments and ratings, and organizational needs assessments.
DIR has applied its national field resources to in-person interviews, such as intercept surveys, field surveys with cell phones to connect with our CATI Center, paper and pencil (PAPI), and computer-assisted personal interviews (CAPI). Our multilingual field interviewers have been extensively trained in tracking and locating, gaining cooperation, refusal conversion, and maintaining confidentiality. DIR conducted four rounds of interviews for the Subsidized Transitional Employment Demonstration (STED), and achieved response rates exceeding 75 percent.
To conduct literacy and language assessments in school settings, we use standardized assessment tools in both English and Spanish, such as the Woodcock Johnson III, the TOWRE-2, and the Gates-MacGinitie group reading assessment. Most recently, for the Evaluation of the Multi-Tiered System of Support for Behavior (MTSS-B), we used both the School-Wide Evaluation Tool (SET) and Individual Student Systems Evaluation Tool (ISSET)
Systematic classroom observation is widely used so that researchers can study the process of education in a natural setting. For large-scale education evaluations, DIR conducts classroom observations to provide more details and precise evidence for school-improvement strategies. We use interrater reliability to ensure that each observation produces accurate representations of the variables being measured. Our staff includes educators, diagnosticians, and others experienced in related disciplines. For example, DIR used the ASSIST protocol to conduct 1,800 classroom observations as part of the Impact Evaluation of the Multi-Tiered Systems of Support for Behavior (MTSS-B).
DIR designs focus group protocols that gather rich, detailed responses, and we use probes to gain further information and clarify confusing issues. We recruit, organize, and conduct focus groups and use the resulting qualitative data to develop a deeper understanding of the perceptions, opinions, beliefs, and attitudes in communities of concern to our clients. We produce custom reports that range from summaries to cross-cutting, thorough assessments. We conduct focus groups with multilingual and minority youths, business professionals, educators, and panel experts. In 2016, DIR engaged the Governing Board members of the Regional Education Laboratory (REL) Southwest in a focus group discussion regarding the most important areas of educational need in the regions.
We use in-depth interviews to identify the perceptions and experiences of individuals expressed in their own words and report those perceptions to our clients. DIR develops probing and flexible interview guides that elicit candid and reflective responses. We conduct in-depth interviews with educators, federal government employees, program managers, community-based organizations, employers, and labor representatives.