Diarrhea - [email protected]

Cognitive Load Theory Applications to Procedural Training Justin L. Sewell MD, MPH, FACP Assistant Professor of Medicine, UCSF Student, UCSF/University of Utrecht Doctoral Program in Health Professions Education My path BS, MPH, MD at U Arizona IM Residency, GI Fellowship at UCSF Joined UCSF faculty July 2011 TSP 2012-13 selected education and educational research as my niche January 2015 UCSF/U Utrecht Doctoral Program in Health Professions Education

Experiences during training Challenges learning to perform endoscopic procedures Complexity Cognitive and psychomotor aspects Too much information! Feeling overwhelmed and unable to perform Primary source of identity as a gastroenterologist Whole task learning frustration when not successful No curriculum felt like I was thrown in led to a number of questions How do GI fellows learn to perform endoscopic procedures? How should GI fellows learn to perform endoscopic procedures? Why dont we have a curriculum? What are best practices for teaching endoscopy? Why hasnt anyone studied this in detail?

Focus of my research Endoscopy training of gastroenterology fellows Cognitive load theory as a lens Rationale (theoretical, my own experience) Adapt for use in other areas of health professions training Cognitive Load Theory Sweller J. Cogn Sci 1988. Young JQ. Med Teach 2014. Young JQ. Med Teach 2014. Cognitive Load Theory Most studied in classroom setting Has intuitive application to medical education,

particularly in workplace learning Tenets of CLT can be used to understand (diagnose) AND lead to action (treat) Includes measurable constructs that should be understandable to medical community Measuring cognitive load Single item self-rating of mental effort (Paas) Very, very high to very, very low mental effort Multi-item self-rating instruments (Leppink) Measure different types of CL Response time to secondary task Physiological measurements (HR variability, electrical skin conductivity) Functional MRI? Sweller J. Cogn Sci 1988. Young JQ. Med Teach 2014.

Leppink J. Behav Res Methods 2013. Leppink J. Learn Instruc 2014, Planned studies 1. Systematic review CLT in clinical workplace learning 2. Instrument to measure CL during colonoscopy training 3. Identify trainee, setting, patient, and attending factors associated with CL during colonoscopy training 4. CLT commentary with JQY, Perspect Med Educ 5. Correlations between CL and performance longitudinal 6. Qualitative study teaching and learning struggles among fellows and faculty implications for curricular development 7. Experimental study teaching styles, CL and performance Cognitive Load Inventory for Colonoscopy (CLIC) Goal: develop and collect validity evidence for an instrument to measure CL during colonoscopy training

CLIC Development Literature review Discussed with faculty and fellow colleagues Initial item development Expert review (3 CLT experts, 3 colonoscopy experts) Pilot testing (16 fellows) and cognitive interviews (8 fellows) Final instrument (19 items 8 IL, 6 EL, 5 GL) Artino AR. Med Teach 2014. Intrinsic load Please rate your agreement with the following statements regarding the colonoscopy youve just completed: 1. Physically manipulating and controlling the colonoscope was difficult. 2.

Using ancillary instruments (i.e., biopsy forceps, polypectomy snare, injector needle, clips, cautery device) was difficult. 3. Identifying normal anatomy and/or landmarks was difficult. 4. The endoscopic findings were complex or difficult to characterize. 5. It was difficult to manage identified pathology (i.e., polyps that were difficult to remove, bleeding that was difficult to control). 6. It was difficult to keep track of, or remember, all the endoscopic findings. 7. Managing the patients level of comfort was difficult. 8. Overall, this colonoscopy was difficult and/or complex. Extraneous load Please rate your agreement with the following statements regarding your experience during the colonoscopy youve just completed: 9. My supervisors instructions were unclear. 10.

My supervisor used language that was confusing or unfamiliar. 11. The manner in which my supervisor provided instructions or teaching was ineffective for my learning. 12. I felt distracted by other people present in the endoscopy room. 13. I felt distracted by the environment (i.e., my pager going off, environmental noise, the layout of the room). 14. I felt distracted by things on my mind unrelated to this colonoscopy. Germane load/Working memory resources allocated to managing intrinsic load Please rate your agreement with the following statements regarding your level of mental effort during the colonoscopy youve just completed: 15. I invested substantial mental effort learning how to control or manipulate the colonoscope and/or other endoscopic equipment. 16. I invested substantial mental effort identifying or understanding colonic anatomy. 17. I invested substantial mental effort understanding, remembering, and/or managing the endoscopic findings. 18.

I invested substantial mental effort learning how to manage patient comfort level. 19. Overall, I invested substantial mental effort learning during this colonoscopy. Methods Subjects: 1,061 GI fellows throughout US 477 responses (45%) Exploratory study using EFA (N=116) Confirmatory study using CFA (N=361) Exploratory factor analysis Aggregates a given set of observed/measured items to a smaller set of unobserved factors based on the bivariate correlation structure

Confirmatory factor analysis Normed 2 = 2 /df should be <5 Comparative fit index (CFI): explains proportion of information explained by the model by comparing how model improves on baseline (null) model, range 0-1 should be >0.9 Root mean square error of approximation (RMSEA): estimates the amount of error of approximation (badness of fit) should be < 0.1 (or <0.06) Standardized root mean square error of approximation (SRMR): measures overall difference between observed and predicted item correlations should be <0.06 Age, mean (SD) Female, No. (%) Year in training, No. (%) 1st year 2nd year 3rd year 4th year Prior colonoscopies, No. (%)

<50 51-100 101-150 151-200 201-250 251-300 >300 Geographic region, No. (%) Northeast South Midwest West Number of total fellows in program, No. (%) <6 fellows 7-12 fellows >13 fellows Exploratory Confirmatory study (N=116) study (N=361)

Total (N=477) ACGME fellows 2013-14 (N=1,458)1 32.2 (3.2) 44 (37.9) 32.5 (2.8) 122 (34.1) 32.4 (2.9) 166 (35.0) 32.3 (NR); P=0.51 503 (34%); P=0.903 25 (21.6)

53 (45.7) 34 (29.3) 4 (3.5) 85 (23.7) 126 (35.1) 140 (39.0) 8 (2.2) 110 (23.2) 179 (37.7) 174 (36.6) 12 (2.5) 486 (33.3); P<0.001 482 (33.1); P=0.07 490 (33.6); P=0.25 NR 11 (9.5)

14 (12.1) 7 (6.0) 12 (11.2) 16 (13.8) 12 (10.3) 43 (37.1) 14 (3.9) 36 (10.0) 34 (9.5) 36 (10.0) 39 (10.9) 40 (11.1) 160 (44.6) 25 (5.3) 50 (10.5) 41 (8.6) 49 (10.3) 55 (11.6)

52 (11.0) 203 (42.7) 42 (36.2) 30 (25.9) 24 (20.7) 20 (17.2) 134 (37.3) 83 (23.1) 76 (21.2) 66 (18.4) 176 (37.1) 113 (23.8) 100 (21.1) 86 (18.1) 30 (25.9) 61 (52.6)

25 (21.6) 88 (24.6) 170 (47.5) 100 (27.9) 118 (24.9) 231 (48.7) 125 (26.4) CLIC EFA 3 factor solution fitting IL/EL/GL model

of CLT Item loadings were high with little crossloading >0.2 63% of variance explained Study #1 CLIC CFA 1-factor model 2-factor model 3-factor model 3-factor model (w/o E4 and E6) 2 , df, P-value1

Normed 2 2 = 3077.03, df=152, P<0.001 Normed 2 = 20.24 2 =2090.95, df=151, P<0.001 Normed 2 = 13.85 2 = 824.02, df=149, P<0.001 Normed 2 = 5.53 2 =388.99, df=87, P<0.001 Normed 2 = 4.47 CFI (>0.9) 0.45 RMSEA (<0.1) 0.23 SRMR (<0.06)

0.16 0.63 0.19 0.20 0.87 0.11 0.06 0.91 0.10 0.04

Normed 2 (<5) Comparative fit index (>0.9) Root mean square error of approximation (<0.1) Standardized root mean square residual (<0.06) Study #1 CLIC CL and year in training 6 P<0.0001 for all CL types 5 4 IL EL GL 3 2 1 0 1st year

2nd year 3rd year 4th year Evidence for validity Content evidence Adaptation from other instruments Developed additional items Starting with more items than expected to retain Pilot testing and cognitive interviews Response process Pilot testing REDCap

Internal structure Factor analysis Internal consistency External validity Sampling Relationships to other variables Study #2 Correlates with CL How do characteristics of fellows, settings, procedures, and supervisors affect CL during colonoscopy training? Goal is to identify factors that could be modifiable using curricular design Study #2 Correlates with CL Fellow characteristics: 13

Year, # prior colonoscopies, age, gender, fellow type, program size, geographic region, specific program, on call at time, on call night before, paged during, hours of sleep past week, fatigue level Procedure characteristics: 8 Date, time, setting, emergent, indication, queue order, # people in room, ancillary maneuvers Patient characteristics: 5 Bowel prep, tolerance, sedation, age, gender Supervisor characteristics: 4 Level, confidence, engagement, took control Characteristics (directionality if appropriate) Year in training (increasing) Number of prior colonoscopies (increasing) Hours of sleep per night (decreasing) Level of fatigue (increasing)

Colonoscopy not performed in endoscopy suite Queue order (increasing) Number of people in room (increasing) On call at the time of, or night before, colonoscopy Paged during colonoscopy Fair or poor bowel prep Poor patient tolerance Sedation by anesthesia Patient age <18 or >80 years old Female patient Number of ancillary maneuvers performed Urgent or emergent indication for colonoscopy Junior supervisor (versus senior) Supervisor engagement (increasing) Supervisor confidence (increasing) Supervisor took over colonoscopy Type of characteristic Fellow

Fellow Fellow Fellow Setting Setting Setting Setting Setting Procedure Procedure Procedure Procedure Procedure Procedure Procedure Supervisor Supervisor Supervisor Supervisor

Hypothesized effect on cognitive load Intrinsic load Lower Lower None None None None None Lower None Higher Higher None Higher Higher Higher Higher None None

None Higher Extraneous load Lower Lower Higher Higher Higher None Higher Higher Higher None None Lower Higher None None Higher

None Lower None None Germane load Lower Lower Lower Lower None Lower None Lower Lower None Lower Higher None None

Higher Higher None Higher Higher Lower Study #2 Correlates with CL Plan is for individual regression analyses for each variable Then build multivariable regression models Separate out effects (variance explained) of prior experience and fellow, setting, procedure, and supervisor characteristics Possibly build separate models for year in training Study #3 Systematic review Studies using CLT for clinical workplace learning in the health professions Identify best practices for teaching and

curricular design Address challenges and debates related to CLT BEME Search plan Relatively small number of titles so will keep search broad Databases: PubMed, ERIC, PsycInfo, CINAHL, Scopus Future plans Longitudinal study: how does CL measured at time point A affect performance at time point B? Qualitative study: teaching and learning struggles related to CL among fellows and faculty, implications for curricular design Experimental study: use simulation to study tenets of CLT and/or use simulation-based curriculum to improve CL early in training

Future directions for CLT Measurement challenges (lack of correlation among measurement types) Subjective rating scales versus objective measures Is germane load really germane? Longitudinal studies correlating CL and performance

Recently Viewed Presentations

  • Financial and Managerial Accounting Wild, Shaw, and Chiappetta

    Financial and Managerial Accounting Wild, Shaw, and Chiappetta

    Sales Returns and Allowances On June 12, Barton Co. sold merchandise costing $4,000 for $7,500 on account. The credit terms were 2/10, n/30. P2 4-* Sales Returns and Allowances On June 14, merchandise with a sales price of $800 and...
  • Criminal Defenses - St. Francis Preparatory School

    Criminal Defenses - St. Francis Preparatory School

    Criminal Defenses How do I get out of this? ... Self-Defense There is a long-standing belief that a person should be allowed to defend themselves from an attack It goes further to say that a person should not have to...
  • Rapid Prototyping - MIT

    Rapid Prototyping - MIT

    Rapid Prototyping & Alternative Manufacturing Techniques or how to get it done differently than we told you yesterday Outline Welding Waterjet EDM Stereolithography & 3-D printing Casting Forming Injection Molding Thermoforming Outline Welding Waterjet EDM Stereolithography & 3-D printing Casting...
  • Solving Inequalities with Addition and Subtraction

    Solving Inequalities with Addition and Subtraction

    solve inequalities using addition and subtraction. SOL: A.1, A.3 Designed by Skip Tyler, Varina High School Edited by Eddie Judd, Crestwood Middle School 1) Graph the solution set of x < 3. When you have < or >, use an...
  • Tires, Tubes &amp; Services

    Tires, Tubes & Services

    1. States will contact contractors directly to begin the Participating Addendum process. 2. States may have submitted Intents to Participate during solicitation, this will provide the information for contractors to contact states interested in signing a Participating Addendum. 3.
  • IE by Candidate Classification: Jansen&amp;Abney, Cohen et al

    IE by Candidate Classification: Jansen&Abney, Cohen et al

    Good results in (Kushmeric and Frietag, 2000) BWI algorithm BWI algorithm BWI example rules Cohen et al Improving A Page Classifier with Anchor Extraction and Link Analysis William W. Cohen NIPS 2002 Background: "co-training" (Mitchell&Blum, '98) Suppose examples are of...
  • Hamlet by William Shakespeare - English 12 C. Smith

    Hamlet by William Shakespeare - English 12 C. Smith

    Hamlet's view of women is decidedly dark. In fact, the few times that Hamlet's pretend madness seems to veer into actual madness occur when he gets furious at women. Gertrude's marriage to Claudius has convinced Hamlet that women are untrustworthy,...
  • When was most of the maths you learn

    When was most of the maths you learn

    The Koch Snowflake But are fractals actually useful? But are fractals actually useful? Yes! e.g. in film Image:Animated fractal mountain.gif - Wikipedia, the free encyclopedia But are fractals actually useful? and other egs: fractal antenna lungs When was most of...