Examining Best Practices for Sampling and Weighting Dual ...

Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr. David Redlawsk Rutgers, The State University of New Jersey PANJAAPOR 2015 Spring Event We know we need to include cell phones in telephone RDD samples and weight accordingly but we dont agree on how PANJAAPOR 2015 Spring Event Considerations Sampling What % of the sample

should be cell phones? Landlines? Weighting Base weight? Frame overlap adjustment? PANJAAPOR 2015 Spring Event Calibration adjustments? Trimming? Low variance! Data and Methods Data Rutgers-Eagleton Poll Methods Demographics: Chi-square Sampling analysis: weight C & LL samples

to make up varying proportions of overall sample Weighting: 17 weighting & trimming combinations PANJAAPOR 2015 Spring Event Weight Calculations Base weight: sample selection probability multiplied by household selection probability Frame adjustment: Compositing, Effective Sample Size PANJAAPOR 2015 Spring Event From Kennedy (2012) Weight Calculations Calibration

Basic Sample Balancing one manual iteration of each demographic variable Raking multiple iterations of each demographic variable; SPSSINC RAKE Extension Trimming Hard cap: .2 and 5 Percentile cap: 2nd and 98th percentiles PANJAAPOR 2015 Spring Event Weighting Combinations Taking a cue from Kennedy (2012): COMBINATION 0 A B C D E

F G H I J K L M N O P BASE WEIGHT? No Yes Yes Yes Yes Yes Yes Yes

Yes No No No No No No No No PANJAAPOR 2015 Spring Event FRAME OVERLAP N/A 1/2 Compositing 1/2 Compositing 1/2 Compositing 1/2 Compositing Effective Sample Size Effective Sample Size

Effective Sample Size Effective Sample Size 1/2 Compositing 1/2 Compositing 1/2 Compositing 1/2 Compositing Effective Sample Size Effective Sample Size Effective Sample Size Effective Sample Size CALIBRATION TRIMMING METHOD Basic Sample Balancing Hard Cap, .3 & 3 Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap

Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing Percentile Cap Raking Hard Cap Raking Percentile Cap Basic Sample Balancing Hard Cap Basic Sample Balancing

Percentile Cap Raking Hard Cap Raking Percentile Cap *0 = Rutgers-Eagleton Poll practices at the time of research Hypotheses I. Cell-only respondents will follow patterns in the literature of demographic distinctiveness and geographic mobility II. Dual users from the cell frame will differ demographically from dual users from the landline frame III. A higher % of cell phones than used in the Rutgers-Eagleton Poll will be required in the sample to match NHIS PANJAAPOR 2015 Spring

Eventestimates Hypotheses IV. Half compositing will be less efficient (higher DEFF) than effective sample size V. No difference in efficiency between hard cap and percentile cap; cant know a priori which will increase variance more PANJAAPOR 2015 Spring Event Demographics Statistically significant differences at = .001 C Only vs. Dual Dual from cell Gender Age Race

Party ID Educatio n Income PANJAAPOR 2015 Spring Event vs. LL only vs. Dual from LL

* * = .01 Demographics Differences in gender composition Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event Geographic Mobility Match between FIPS code (county) from sample provider and from respondent Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event

Simulated Sample Composition CELLLANDLIN CELL- MOSTL DUAL ELANDLIN ONLY Y USE MOSTLY E-ONLY NHIS (NJ) 19.4% 30% C 70% LL 10.4% 40% C 60% LL 13.9% 50% C

50% LL 17.4% 60% C 40% LL2015 Spring 20.8% PANJAAPOR 70% C Event 25.7% 31.1% 15.2% 6.9% 33.7%

26.6% 23.3% 6.0% 34.1% 25.8% 21.1% 5.1% 34.4% 24.9% 19.0% 4.3%

34.8% 24.1% Rutgers-Eagleton Poll 16.9% 3.4% Design Effects Rutgers-Eagleton Poll PANJAAPOR 2015 Spring Event Margins of Error Rutgers-Eagleton Poll

*Calculated using Langer Research Associates MoE Machine PANJAAPOR 2015 Spring Event Average deviation from NHIS estimates Rutgers-Eagleton Poll NHIS (NJ) CELLCELLONLY MOSTLY 19.4% 25.7% PANJAAPOR 2015 Spring Event DUAL LANDLINE LANDLINE USE

-MOSTLY -ONLY 31.1% 15.2% 6.9% * indicates that the weighting combination came within 1% of the cell-only estimate Discussion Support for Hypotheses I, II, and III: Significant demographic differences among cell only, dual users, and landline only Estimated geographic mobility much greater for Rs from cell sample vs. Rs from landline sample Significant demographic differences

between dual users from cell and landline samples More PANJAAPOR 2015cell Springphone interviews (~50%) are Event Discussion No support for Hypothesis IV Compositing was not less efficient than Effective Sample Size Little support for Hypothesis V Percentage cap was more efficient than the hard cap PANJAAPOR 2015 Spring Event Weighting

Recommendations Ultimately, no obvious best practices Criteria for choosing weighting and trimming combinations: Includes a frame adjustment (avoid biased estimators) Inclusion of base weight Brings percentage of cell-only respondents within 1% of NHIS estimates Percentile cap more efficient PANJAAPOR 2015 Spring Event Weighting Recommendations Under these criteria, good weighting and trimming combinations are: B: Base Weight, Compositing, Basic

Sample Balancing, Percentile Cap D: Base Weight, Compositing, Raking, Percentile Cap F: Base Weight, Effective Sample Size, Basic Sample Balancing, Percentile Cap PANJAAPOR 2015 Spring Event Future Research Replicate using other data Strengthen the theoretical basis of the criteria used to recommend weighting combinations Experiment with more aggressive percentile caps (e.g. 5% and 95%) PANJAAPOR 2015 Spring Event Thank You [email protected]

PANJAAPOR 2015 Spring Event

Recently Viewed Presentations

  • External Examiners induction session For newly appointed External

    External Examiners induction session For newly appointed External

    HE Student Handbook (generic information in Moodle) Statement of resources. Module Specification. Module delivery details. Module Guides. Academic Regulations (January 2018) Mitigation process. Academic Misconduct. Academic Appeals. Recognition of Prior Learning (RPL) Alternative Assessment Plans
  • 幻灯片 1 - Ranger

    幻灯片 1 - Ranger

    5. Linear LPS Method. As the streaks share the same right-end, the minimum values are increasing if the streaks are listed in the increasing order of left-ends. Figure below: l-v plotfor the above example. Real-world datasets, sequence length from thousands...
  • Welcome to the RWA Bristols First and Finest

    Welcome to the RWA Bristols First and Finest

    The RWA Art Wall and Art Trolley. ... Mixed Media on canvas. ... Radio (call-in discussions) Easton Community Centre (artist-led . workshops…) Bluecoat Gallery, Liverpool (performances, talks . workshops) Impact and Legacy. Enhanced awareness and appreciation of Jamaican art.
  • Sample Presentation - De La Salle-College of Saint Benilde

    Sample Presentation - De La Salle-College of Saint Benilde

    Always check your Benilde Mail for Pre-Enrollment Confirmation Email. Reasons for not receiving the Email after clicking the Email My Pre-enrollment Record button: - No/incorrect Benilde Email - Failure during sending of Pre-enrollment Record *You may contact ITD for assistance.
  • Redox Reactions Oxidation-reduction reactions, also called redox reactions,

    Redox Reactions Oxidation-reduction reactions, also called redox reactions,

    Instead of drawing a cell diagram such as or chemists have devised a shorthand way of completely describing a cell called line notation. This notation scheme places the constituents of the cathode on the right and the anode components on...
  • Advanced Microeconomics

    Advanced Microeconomics

    Neoclassical economics evolved from the Classical school defined by Ricardo, Smith, and Marx. Classical school defined by: focused on the economy: productivity, growth, economies of scale, gains from trade, land rent. understood a gap between value in use and value...
  • Presentation COPPERCHASE - International Civil Aviation ...

    Presentation COPPERCHASE - International Civil Aviation ...

    Copperchase FPL2012 Copperchase contracts are signed for FPL2012 support irrespective of 'last minute' changes (within reason) All Copperchase Updated systems will operate either side of the change-over.
  • Expanding the National DPP Lifestyle Change Program through

    Expanding the National DPP Lifestyle Change Program through

    (Chida et al., 2009) Increases the ability to forgive. (Saunders et al., 2013) Improved mental health and lower depression scores. (Levin J., 2001) Lower C-reactive protein levels or inflammation. (Ferraro & Kim, 2014) Increased medication adherence. (Lewin et al., 2005)...