Services

Study Design & Methodology

ISR staff works with clients to design data collection efforts for qualitative and quantitative studies that involve human subjects research. We have experience in questionnaire development, semi-structured interview protocols for focus groups or interviews, cognitive testing of instruments for reading level and overall comprehension, and pilot testing for preexisting instruments. We also have experience with bilingual instruments, translation and back-translation. We work on cross-sectional and longitudinal data collection efforts as well as program evaluations.

Technical Assistance with Survey Planning

More than ever before, key decision makers are in need of empirical guidance for improving the operations of their organizations. Such guidance, however, cannot be based on a plethora of raw data. Measurable impact requires actionable intelligence, vital to which is reliable information that has to be distilled from representative data. As depicted in the following schematic, scientific sample design and diligent survey administration; effective analysis of survey data; and coherent interpretation of the results are critical steps towards the ultimate goal of enabling the production of a measurable impact.

Another fundamental tenet of our strategy is recognition of the fact that rigorous survey research programs are designed and implemented to minimize the so-called Total Survey Error, as depicted in the following diagram. Under this comprehensive framework each component of error should receive proper attention, since with imbalanced focus on any particular source of error, other error components will gain opportunities to grow and create weak links in the survey process. Effectively addressing all these issues requires academic knowledge, hands-on experience and transparent execution—a package the ISR team offers.

Survey Instrument Review and Revision

As a scientific survey research unit, ISR is accustomed to reviewing prior versions of questionnaires with a fresh view on survey methodology to accurately prioritize core components, identify question revisions and recommend items for deletion. ISR uses questionnaire flow charts to determine question source citations, outcome measures and policy insight. Such charts will allow for clear identification of duplicate information obtained from multiple items, reference historical data from source citations and create future questions to obtain new information on recent policy changes. New trending topics can be exposed through importing the previous survey questionnaires into the flow chart to ensure that important measures are not missed.

Cognitive testing ensures that the language of the questionnaire and other materials such as advance letters and survey instructions are appropriate for the target population. Because a well-designed questionnaire format is important, we strive to design an instrument that is easily navigated by the interviewer and, if self-administered, visually appealing to the respondent. We are highly experienced in experimental methods to test wording, response format, context and mode, as well as in reliability and validity assessments and interaction coding (Source: Fahimi 2012).

Sampling Design & Recruitment

We help clients design the most efficient and cost-effective sample selection strategies, including probability-based designs such as dual-frame telephone and address-based sampling plans, as well as nonprobability-based designs such as convenience, snowball and intercept studies.

In addition, ISR owns and operates BeHeardPhilly, a multimode panel that includes 9,500+ Philadelphians who have opted in and agreed to be surveyed. Panel members have been recruited via probability and multiple non-probability methods. Panel members opt their preferred mode of response which includes phone, web and SMS, and they opt the number of times they are willing to be surveyed in a given year. We continue to grow the panel by targeting recruitment methods for hard to reach populations.

ISR uses a consultant sampling statistician for complex sampling designs, post-stratification weighting and computation of adjusted standard errors.

Sampling Design & Analysis

The foundation of any scientifically conducted survey must be a sound sampling strategy. ISR devotes considerable attention to sampling methods, balancing statistical considerations with practical limitations dictated by field costs.

The Institute has also developed sampling design strategies for telephone surveys and for studying special populations. In addition, ISR has experience in computing sampling errors and in devising weighting and imputation schemes to facilitate analysis of data. ISR is working with consultant Mansour Fahimi from Marketing Systems Group (MSG) on sampling design, weighting for complex designs and analysis.

Survey Weighting

ISR recommends that survey data be weighted to reflect variations in probabilities of selection as well as differential nonresponse and other factors which cause the sample and population distributions to differ. Although the national household sample and most telephone surveys are designed to produce self-weighting samples of households, variations on this self-weighting design are inevitable. The first source of variation in sampling probabilities arises from the special needs of the survey where certain subpopulations are oversampled, the oversampling criterion usually being a set of demographic variables like age, sex, race and residential location. The second source of variation is the usual requirement that only one person in a sample household be interviewed, although several eligible respondents may live there. Because the number of eligible respondents in a household varies, the actual probability of selection of an individual is inversely proportional to the number of eligible respondents in the household. Because all ISR samples are selected by probability methods, the probability of selection can be computed for every household and for every individual. Thus, sampling weights, which are inversely proportional to the overall probability of selection, can be assigned to each survey respondent.

Data Collection

ISR is a full-service data collection shop and has as in-house 35-seat call center where we conduct computer-assisted telephone interviewing (CATI) using CASES. We also have experience with computer-assisted in-person interviewing (CAPI), and audio computer-assisted self-interviewing (ACASI). We have conducted mail, web, face-to-face field and SMS surveys. We specialize in multimode surveys and offer push-to-web options for all surveys. We can recruit subjects and run focus groups for any study and have experience in mock-jury research. In addition, we specialize in respondent tracking and tracing for longitudinal studies. ISR also has interactive voice recognition (IVR) capabilities.

Computer-assisted Telephone Interviewing (CATI)

ISR occupies 12,000 feet of newly renovated space in the Entertainment and Community Education Center (ECEC building located on Temple University’s Main Campus). Our call center has 35 production stations, expandable to 45, for telephone interviewing, web support, inbound call handling and tracing operations. It is located in a controlled-access environment, ensuring the strict security of all data collection, and operates seven days a week.

The institute uses state-of-the-art technology to facilitate data collection and management. The institute’s call center is networked with high speed connections; allowing interviewing stations to be monitored remotely. ISR has been using Computer-assisted Survey Execution System (CASES) software for CATI and direct data entry since 1986, and for CAPI since 1992. Currently, ISR uses the latest versions both 5.5 and 6 beta of CASES that support both CATI/CAPI data collection modes developed by the Survey Research Center at the University of California, Berkeley. The CASES software package supports varying levels of survey complexity. The institute continues to make advancements in optimizing cost efficiency and reducing survey error, such as the modeling and construction of algorithms for call scheduling in random-digit-dialed landline telephone surveys.

Computer-assisted In-person Interviewing (CAPI)

ISR fielded one of the first national CAPI surveys in July 1992. Over 300 CAPI-trained interviewers conducted more than 15,000 CAPI interviews for the National Survey of Families and Households (NSFH). ISR continues to utilize its CAPI capabilities and experience to serve the needs of studies requiring face-to-face interviewing. The institute has trained a national staff of more than two thousand interviewers, resident in their respective communities throughout the United States, who have been trained by, and work for, ISR on a continual basis. During each study, every interviewer’s work is reviewed and, at the conclusion of the study, the interviewer is rated for field performance and quality of data collected. Ratings become part of the interviewer’s employee record; only those interviewers whose work meets ISR’s standards are retained. This results in continuous upgrading of the field staff.

ISR maintains an inventory of mobile devices used to administer questionnaires of varying complexity allowing the interviewer to concentrate on other details like the proper use of visual aids and rapport with the respondent.

Self-administered Questionnaires

Web Surveys

ISR utilizes both CASES and Qualtrics for web surveys. Using both programs maximizes cost efficiencies while maintaining quality interviewer-respondent interaction. CASES Web QA extends the capabilities of its core CATI/CAPI package with the ability to administer Version 5 CASES instruments as web surveys over the internet through a browser interface. With this architecture, data can be collected for a single CASES study simultaneously using traditional CASES CATI/CAPI procedures (the Windows client), and via the web. In other words, a single survey instrument can be authored in CASES 5 for execution in both a CATI call center and over the web.

The Qualtrics online survey tool is a web-based, self-authoring tool. It enables anyone on ISR’s research team or the client to build, send and analyze noncomplex surveys. The result of this client option minimizes web survey development costs, freeing the client to focus on research rather than web development.

Data Entry, Coding and Validation

Data verification will be “heads-down” direct data entry from the returned survey, followed by the reentry or verification of the data. Using the “heads-down” approach, data entry and verification from the paper questionnaire will be done in two separate and specialized steps.

Initial entry for verification
Verification

Any discrepancies between the inputs of the two sessions are displayed and resolved during the verification session. In verify mode, the answers entered in the initial entry session are compared with the answers entered in the verify session for that case. If there is a difference between the two inputs, the verifier is provided with the option to resolve the discrepancy through an interactive review process.

Respondent Tracking and Tracing

The Institute for Survey Research has over 40 years of experience tracing respondents for longitudinal studies and list sample. ISR customizes tracing strategies on a per project basis. Using the demographic profile, data collected during survey administration, and supplemental information obtained from known data sources, ISR tracks each respondent. Tracing customization includes age-appropriate online searches such as social network sites for younger respondents and property records information for older respondents. ISR can implement tracing measures into any budget, at any research phase and method of data collection. ISR also offers tracking services outside of data collection.

Because the age of the tracing information directly affects its cost and accuracy, ISR preemptively tracks respondents by periodically updating contact information. This is the initial cost-effective approach used prior to more aggressive strategies.

Telephone Tracing
ISR telephone tracing begins in data collection. CASES records call history of respondents and their contacts. The system records interviewer notes, call records and address information. Once a case enters the tracing queue, tracing coordinators review the information and contact all available respondent phone numbers, friends, relatives and neighbors. Tracing coordinators also use reverse directories to obtain telephone information for previous respondent address information.

Advance Tracing
Tracing coordinators use nationwide databases to locate subjects who have died or relocated. ISR uses the National Death Index and prison database searches to verify availability. ISR also uses comprehensive tracing strategies to confirm newly obtained information from multiple sources.

Database Searching
ISR uses free and subscription access to multiple nationwide databases to obtain current contact information. Those databases include but are not limited to the U.S. Postal Service National Change of Address database, credit bureaus, state Department of Motor Vehicle records and other directory searches. ISR also uses comprehensive tracing strategies to confirm newly obtained information from multiple sources.

Field Tracing
Field interviewers are trained to establish rapport with respondents, relatives, neighbors, associates and everyone encountered in the field. Field interviewers are trained to take detailed notes of their surroundings and pursue all leads. Field coordinators confirm leads and give detailed instructions for the appropriate actions.

Data Processing

ISR has data entry, coding and validation capabilities. We use in-house, high-quality digital recording and video equipment and use a third-party transcription service for interview and focus group audio files.

Analysis & Reporting

ISR can provide quantitative and qualitative data analyses, statistical modeling and reports. In-house we use SPSS, SAS and NVivo, and can refer consultation with Stata and R. Report graphics are generated with Tableau and dashboard technology Domo.

Information Technology

With a full-time systems engineer, ISR can provide database design and management, application design and development, medical and legal data extraction. Our data management plan can be found here. (insert link to our data management plan). In addition, ISR has developed a proprietary multi-mode panel management system that can be purchased via a SaaS license. Please contact us for more information.

Education & Outreach

Staff at ISR can provide customized research methods and data collection training workshops or guest lectures.

Affiliations

Collaborators

MDRC- Building Knowledge to Improve Social Justice
Marketing Systems Group
Textizen

Institute for Survey Research