Non-experimental & Survey
Research Design
Problems
Large amounts of raw data to deal with
Few computerized programs analyzing qualitative verbal information
Replication is seldom possible and in most cases doen’t make much sense(Tayor, C. et al, 2005)
Leave the analysis task to the researcher’s ingenuity
Problems inherent from qualitative data
Naturalistic Observations
Involving observing research subjects in natural environments
Without making any attempt to control or manipulate variables
Observing apes or children, shoppers
Research should not make any changes in the situation to affect the behaviors
Some Advantages & Disadvantages
Extremely high external validity in the real world
Only able to speculate about the causes – hardly to prove
To be in the circumstances or to go further Unobtrusive Observations
Preventing the difficulty of disturbing the subject’s behavior by the mere acts
Building a blind to observe a particular species of birds
Hiding behind the mirror not to disturb the subjects
Make the researchers’ presence be common for the subjects
Abandoning direct observations
Naturalistic & Unobtrusive Observations
Content Analysis
Being used mainly to study & describe the cultural functions
A study of social interactions/expressions between people &
groups(Berg, 1998).
As investigation or undercover, observing and recording what happen in the group
Issues about the conducting process(Berg, 1998)
Observing as a participant or non-participant
To decide to observe or to act?
Gaining Access to a Field Setting
First task when conducting ethnographic research
Some barriers/obstacles such as membership, restriction
Gaining Entry into the Group
Commonly with Gaining Access Issues – “Case & Approach”
Becoming Invisible - Overt entry or Covert entry?
Making Observations & Recording Data
Noting, Scraping, Voice Recording
Analyzing Ethnographic Data
Identifying themes/hypotheses & coding systematic patterns
Ethnography
Conceptual Feature
Being used to analyze a written/spoken record
Aim for the occurrence of specific categories, events, items, or behaviors
Archival Sources are commonly used to content analyze
Holsti(1969) defined three characteristics of content analysis
Should be objective(or explicit)
Should be systematic
Should have generality
Categories should reflect the purposes of the research & be exhaustive, exclusive, independent, and derived from one classification system.
Consider who will do the analysis to be unbiased
The validity depends on the material analyzed
= Researcher should obtain the relevant materials
Content Analysis & Its steps
Performing Content Analysis
Clear Operational Definitions of Terms
Sufficiently defined to allow precise categorization
Reviewing related research using content analysis to add, delete, or expand categories to fit research needs.
Reading the materials to be analyzed before developing categories
Decision about a unit of analysis after developing
Recording unit & Context Unit
Content Analysis & Its steps
Conceptual Feature
To avoid the possibility of researcher’s subjective conclusion not to reflect the strength of the relationships in the literature review
A set of statistical procedures to combine or compare results from different studies
Researcher can find and analyze existing research to make
statistical decisions about the strength of the observed effects of independent variables and the reliability of results across studies Steps
Identifying Relevant Variables
Questions must be sufficiently focused to allow for a reasonable meta-analysis
Should narrow the scope of the analysis and decide what variables as the review
Locating Relevant Research To Review
File Drawer Phenomenon with a biased sample (Type I Error)
Doing the Meta-analysis
Comparing & Combining studies to determine the variable’s effect
Meta-Analysis & Its steps
Meta-Analysis & Its steps
Sample of Factors to Include When Meta-Analyzing Literature Full reference citation
Names and addresses of authors Sex of experimenter
Sex of subjects used in each experiment
Characteristics of subject sample(Such as how obtained, number)
Task required of subjects and other details about the dependent variable Design of the study(including any unusual features)
Control groups and procedures included to reduce confoundings
Results from statistical tests that bear directly on the issue being considered in the meta- analysis(effect sizes, values of inferential statistics, p values)
Sample of Factors to Include When Meta-Analyzing Literature
Source: Adapted from Rosenthal, 1984
Meta-Analysis & Its steps
TECHNIQUE COMMENTS
Comparing Studies Used to determine if two studies produce significantly different results Significance testing Record p values from research and convert them to exact p values(such as a
finding reported at p <.05 may actually be p=.036). Used when information is not available to allow for evaluation of effect sized
Effect size estimation Record values of inferential statistics (F, t, for example) along with associated degrees of freedom. Estimate effect sizes from these statistics. Preferred over significance testing.
Combined Studies Used when you want to determine the potency of a variable across studies.
Significance testing Can be used after comparing studies to arrive at an overall estimate of the probability of obtaining the two p values under the null hypothesis (there is no causal relationship between the analyzed variables).
Effect size estimation Can be usedafter comparing studies to evaluate the average impact across studies of an independent variable o n the dependent variable
Meta-analytic Techniques for Comparing and Combining Two Studies
Source: Adapted from Rosenthal, 1984
Some applications
Questions from a local political party during election time
Questions on warranty registration cards
Questions to subscribe
Polling about some issues
Using field survey to evaluate specific attitudes or to predict behavior
Important thing to design the questionnaires
The first step is to define the topic of the study clearly
To design assessing the characteristics of the participants(demographics)
To define predictor variables, demographics and non- demographics can be included
To design assessing the behavior of interest(criterion variable)
Survey Research
Several Item Formats
Open-Ended Items
Let the participant respond in one’s own words
May fail to provide the needed information
May run the risk of misclassifying the answers
Restricted Items(Closed-Ended Items)
Limited number of specific responses in a logical order
Easier but less rich to summarize and analyze than O.E.I.
May not be correct and fit to the participant
Partially Open-Ended Items
Using “other” category
Survey Research
Several Item Formats
Rating Scales
A variation on the restricted question
Commonly do not exceed 10 points and not go below 5 points
Decision of the scale label is important
Three types of anchored points
Participant to interpret the meaning of the rest
All are labeled for the participant to know exactly what each point means
The ends and the midpoint are anchored
Likert scale
Provides a series of statements to which participants can indicate degrees of agreement or disagreement
Survey Research
What is important to construct questionnaires?
The items must be organized into a coherent, visually pleasing format
Demographic items should not be presented first not to lead participants to be bored.
First question to let the participant be motivated to continue
Questionnaires should have continuity; related items to be together
The order in which questions are included on a questionnaire is to affect the responses of participants – General vs. Specific
Sensitive information asking items – from less objectionable ones to more objectionable ones
Design a navigational path directing respondents to read all the information on a page
Create effective visual navigational guides to help respondents stay on the navigational path
Develop alternate navigational guides to help with
situations where the normal navigational guide will be interrupted
Survey Research
Survey Research
SUGGESTION EXAMPLE
Use simple rather than complex words Use “work” rather than “employment”
Make the stem of a question as short and easy to understand as possible, but use complete sentences
“Would you like to study in America?”
Avoid vague questions in favor of more
precise ones Use “How many years have you lived in your current house?” rather than “Years in your house”
Avoid asking for too much information.
Respondents may not have an answer readily available
Use a list of ordered alternatives rather than an open- ended question when asking how often the respondent does something
Avoid “check all that apply” questions Instead of “check all that apply,” list each item separately and have respondent indicate liking/disliking for each Avoid questions that ask for more than
one thing Instead of asking “Would you like to study and then live in America?” ask “Would you like to study in America?” and
“Would you link to live in America?” separately Soften the impact of potentially
sensitive questions Instead of asking “Have you ever stolen anything?” ask
“Have you ever taken anything without paying for it?”
Suggestions for Writing Good Survey Items
Source: After Dillman, 2000
