Pediatrics Milestones Assessment Collaborative

Pediatrics Milestones Assessment Collaborative (PMAC) American Board of Pediatrics Association of Pediatric Program Directors National Board of Medical Examiners The Story of PMAC: A workplace-based assessment system for the real world Patricia J. Hicks, MD, MHPE Alan Schwartz, PhD Disclosures PMAC research and development is supported by the National Board of Medical Examiners and the American Board of Pediatrics Acknowledgements Site Lead Investigators: Baylor College of Medicine: Teri Turner, MD, MPH, MEd. Boston University: Daniel J. Schumacher, MD MEd. Childrens Hospital of Philadelphia: Jeanine Ronan, MD; Rebecca Tenney-Soeiro; MD, Anna Weiss, MD. Childrens National Medical Center: Aisha Barber Davis, MD. Cincinnati Childrens Hospital: Javier Gonzalez del Rey, MD, Med; Joe Knadler, MD; Sue Poynter, MD, MEd. Dartmouth-Hitchcock Medical Center: Kimberly Gifford, MD; Samantha House, DO. Emory University: Susie Buchter, MD. Inova Childrens Hospital: Meredith Carter, MD; Kathleen Donnelly, MD. Monmouth Medical Center: Renuka Verma, MD. Nationwide Childrens Hospital: John Mahan, MD; Rebecca Matthews, MD. Phoenix Childrens Hospital: Vasudha Bhavaraju, MD; Ryan Bode, MD; Grace Caputo, MD, MPH; Kelly Kelleher, MD. Seattle Childrens Hospital: Heather McPhillips, MD, MPH; Sahar Rooholamini, MD. State University of New York at Stony Brook: Robyn Blair, MD. University of California Davis: Su-Ting Li, MD, MPH. University of California Los Angeles: Alan Chin, MD; Kate Perkins, MD, PhD. University of Chicago Corner Childrens Hospital: Nicola Orlov, MD, MPH. University of Florida: Nicole Paradise Black, MD; Lily Chang, MD; Cynthia Powell, MEd. University of Illinois at Chicago: Michelle Barnes, MD; Jen McDonnell, MD; Amanda Osta, MD; Emri Tas, MD. University of Louisville: Kimberly Boland, MD; Olivia Mittel, MD; Sara Multerer, MD. University of Michigan: Hilary Haftel, MD, MHPE; Kristen Wright, MD. University of South Alabama: Sophia Goslings, MD; Franklin Trimm, MD. University of Tennessee: Sandra Arnold, MD; Mark Bugnitz, MD. University of Texas at Austin Dell Medical School: Lynn Thoreson, DO. University of Utah: Jaime Bruse; Bruce Herman, MD. University of Vermont: Ann Guillot, MD; Karen Leonard, MD. University of Virginia: Linda Waggoner-Fountain, MD, MEd. Winthrop-University Hospital: Ulka Kothari, MD; Jill Leavens-Maurer, MD; Robert Lee, MD. Wake Forest University: Allison McBride, MD; Nick Potisek, MD; Wright State University: Ann Burke, MD. American Board of Pediatrics Carol Carraccio, MD, MA; David G. Nichols, MD, MBA. National Board of Medical Examiners Martin Bruegel; Leah Burnett; Brian Clauser, EdD; Melissa J. Margolis, PhD; Peter Katsufrakis, MD, MBA; Tom Rebbecchi, MD; Marcia Winward, MPhil. Association of Pediatric Program Directors Laura Degnon, CAE; Beth King, MPP; Alan Schwartz, PhD. Workplace-based Assessment in the Real World Use Cases Requirements Challenges

Key Groups Shaping Assessment in Graduate Medical Education Pediatrics Milestones Working Group APPD NBME ABP ACGME Assessment for Reporting Eval uatio n Form Ev alu a t io Printe d on Apr MI LE Eva Eval Date : 19, - MedHu b n F or m 2014 ST ON luato r: uati

E- B on of: AS ED F EL L OW E VA LU AT I ON This achi is a new eves som mileston e, b Upd Evalu ut note -based ated atio Asse n Form all, of evaluati from the For ssment The Ped - MedHu com on. To ach feed pon b pnin bac of the 21 iatric ie ve ents a.we k Mile Ev of [email protected] on eval compete ston a parta level, alu e uati yale

the a ticul .edu on formncies Pro je Printe io n ct. d on ar leve fello , pleais divi ded Acad Apr F or l thenw must se ema amo emi19, m choo satisfy c 2014 il Dire ng diffePediatric se PA TIE MI the ALL the rent s 2014 LEctor butto NT rotation ST ON of P . 14(2 n betwcompon C ARE edia Ple ase s S ents tric E- B een Eva Sub and eval): S1 -97. on leve complet AS luato spec ls. Pleaa cons uato ED r: e all rs. Fialty

s e incl isten t que Eval EL L Res stio uati iden ude basis. 1. Prov ns OW on of: If com Et Edu Date VAcatio men he/she ensu ides tran LU AT : n, Y ts. res seam sfer Iale ONUniversi less of care trans This ty, Pnin ition that achi is a new aW s* eves eiss mile : som ston e, e but Upd -bas Eval Demo ated not ed

uatio nstra frequ Asse tes from all, of evaluati n Form ent e ssm the For rro rs andent The Ped variab com on. To - MedHu ility ifeedbac of the Usesiatri pon ach b pnin n the c Mile a stand k 21 com transf a.we Ev alu ents of aie ve a leve te mplat er [email protected] on eval ard ston particul l, the info rmati of e Proa yale uati e pete .edu have onbut mancie Printe tio n je ct. on s diso Adap ar leve fello error formy, w divi n ts Acad s Apr F mus a plea l

of stand emi then omiss templ ded or se ion ate19, 2014 amo ard c m choo t satisfy ema with ng diffePediatric err ors il Dire se m inima ALL PA TIE Almo s MI LEand the butto of o missictor rent l the st alwa 2014 NT ST comm on of Ped rotationys effe n betwcompon . 14(2 C ARE unica and poten ON iatri Ple ase ctivel S ): Etes een N/A ents tial c accu s and y S1 -97. Eva on issue - B AS leve

complet commSubrately luato the trans s fo spec eval ls. Pleaa cons ED r: inform u nic ates ialty uators. Alway e a ll feree r F atio s e incl isten t s que Eval Res effect co mpleELnLeven stio ns uati iden ively ude basis. 1. Prov OW in on of: x t Eduand If com ada pting situatio E VA accur Date men he/she ensu ides tran ately catio ns, LU comm a : n, Yale ts. res templ AT unica seam sfer ate a standa rd tes Univ

nd e inform I ON less of care op en nsuri ersi a tion comm trans ng This ty, regar ition that Pnin during unication dless achi is a new a We s* 2. Mak of hand eve settin iss: off s som mileston g or thera es infor comp e, peu e med lexity but Upd -bas optim tic , ated Demonstra not ed adapt al clini decisiondiagnos Asse frequfrom ing tes all, of evaluati ent e The

a cal s tic stand t he rro rs For ssm judg that resu ent of com on. To Ped feedvariab te mplat ard men ility the and iatri pon ach Uses bac lt pnin t* in ie demo e and k onin the21 com c Mile a stand ents a.we transf a nstra te mplat of a ve [email protected] er ston leve ard ofeval Do es part info rmati ee uati petencieEvalu yale pro fessio not icul artesl, the fello on formhave s b utPro ma yje ct. deve on .edu atio Adap nal leve respo

error issdivi lop n w Acad ts organ ,omiss Form nsibi an mus a plea l then stand ded templ ize emi lity asses se ofema - Me ion ate amo cwith ensur choo t satisfy smen d Pedforard dHu Reco il Direerrng se the ALL t of bin g miatri gnize PA TIE orsdiffe inima diagn Almo the of omis s poten rent cs l 2014 osis

diagn ctoreffect st alwa butto the com NT Ev alu and mana o oses hando tia l of ive sionrotation comm . 14(2 ys n CA RE geme r a difficu pon but edia and S): effe ctivel betw a t iopotenthetialversPunica has nt plan lty d Ple aseDeve Printe s ents accu on testric een N/A y d on S1Sub and on issue leve un ifying eveloping nthe F team eval rately comlop sa Apr syn thesi s speccomm ls. Pleaa cons

plete uato 97. orferee a trans well diagn 19, 3. Dev info rmati ialtyu nicat Alway and m fo r 2014 all que esrs. s e incl isten t s ma nage osis differe zed, focus effect man elop s co mple o nRes eveniden ntial ed stion Deve ively ude basis. men age 1. Prov MI LE s in t Edu x and t and mana diagnosis lops If men and carr plan com ada pting situatio a unify ST ON catioaccurately men he/she t plan ensu ides diagn

ns, ies geme comm as tran osis, ing n, Yale ts. res templ out N/A s* nt tanda thera E - BA unica focus seam plan sfer ate rd and tes Univ inform less of careand pies, a n Evaedluato op en e nsuri ersi SE D a tion e ffecti comm trans r: effici ty, regar F EL during unicationn g ition that ent diagn Pnin Eva ve dless s * work aW 2. o stic luati L OWhand

of Deve settin eiss on of: lop s Makes mana -up and g or : EVoffAL thera and geme Date comp infor out tailore carri peu med nt p : moptim U AT adapt lexity , es anage tic d to la n Demonstra diag the plans ment al clini decision I ON ing a tes indivfrequ Begin nostic ba sed direct cal s s to patienThis idualent er ror stand ives on judg that t ard variab form is a ility s and

from resu te mplat witho men an d ulate achi Uses other lt new ca ut a t *rry out in e a nd mana eves a s djust transf in the mile demo geme Deve templ s ta ndard ing er of som patien ston nstra for Do eslops e -bas ate but t/fam uncom nt plans pro fessio tes https and Upd infor matio e, but ily have prefer out m not devecarri : //yal ated not ed error may n Ad apts

nal patien plicated in anag respo ence lop es Asse all, of eval e.me organ emen from ts s nsibi an s a uati s tanda omiss templ but r arely dhu lity for assesplans ize d t Efficie the ion of on. For ssment The Ped ate with b.co rdensur consi smenin ntly com Reco uncom m/u/a der s in g devepnin feedbac of the carrie iatri pon To achi erro rs min gnize patien plicat of the Almo /eva patiendiagnosis

s out ents effect eve of omis imal s lops a.we k on 21 com cN/A ted Mile t/fam st alwa and luat ts, and manpotentia plansdiagn of a leve mana sion hando ive ily occa o r a acomm ions_ prefe agem l [email protected] pete stone ys effec in moses but geme difficu particul siona l, the vers and form rence in corpo lty ost situa hasent nt tively Effici .edu on f ncies Project. potential unica accu N/A s Deve tes f on

s_pr ratingplanlly incorpd evelo orm, is ra tely tions, issue ar the team ellowcomm leve lopently int.m pati ent/fa patie un ifying orati ping s syndevel the emi s for mus plea divi ded Acad l then h?ev nt/fam a ng thesi ops a well 3. Dev mily inform unica t te s se ema amo trans Alway and ily diagn alua prefe cferee choo differe carriezed, and ation satis s ma nage prefeosis effect rence man elop stionI D=2 il Dire ng diffePediatric seeven fy ALL re nces comp ntials focused PAT ively

s men Deve age theinbutto the mana dout lex s rent and 381 tIE N and ctor lops situa men and carr plan gemeiagnosis accur adapt T C plansmana a unify of Pedi rotat 2014 tion s, n betwcom t plan diagn . ing a ately nt ies[4/ 19/2014 ARE , evengeme comm ponents osis, ing templN/A14(2 out Plea atric ions s* S ): S1 thera een and unica focus comp plan for nt ate a stand 3: 06:0 se com on

pies, Sub ed tes licate nd eval ard -97. info rmati leve opespec 4 PM] an e plete and ls. Pleaa cons d or n comm en surin rare effici ffecti ialty uatogrs. on situa regar isten all ent diagn ve s e inclu tions, during unica Res dless t incorp que ti on iden stio ns work 2. Mak o stic Deve hand 1. Prov de bas is. If oratin t Edu setting of off mana -up and com eslop s a nd g thera out ides

cati or geme men h e/she infor ensu comp carri tailore peu on, Y tran nt pla lexity ts. res es optim ticm anagemed diag d to , ale seam sfer n adapt the al plans bdeciment Univ in g nos Begin indiv less of care a sedsions ersi a directclinical patien idual s to stand trans ty, on ives that tic t judg ition that Pnin from templ ard witho men resultan d ca form ulate s

a Wei other ate ut adjus * t * mana in rry out s and demo ss: geme Deve nstra nt plans Do es lops tes uncom profe Demo innot d eveloout manaand carrie ssion nstra patien plicated respo al s frequ p an tes organ geme ts but nsibi ent e ized nt lity for asses plans Efficie rrors rarel consi ensur ntly variab in Reco y smen and der s ing deve ility

t of uncomplica Uses gn izes carrie s patien diagn lo ps in the patien effe ctive a stand transf ted dia gnos poten out man t/fam mana osis o the ts, N/A plans templ erand hando occa ily ra tial ard prefe o es b agem ate geme siona difficu vers rence ut h in m ost inform entf but form incorp nt plan lly lty de as a tion Effic have on s orati Deve s_pr the Adap velop incorp situations, iently error ma y

team ng unifyi int.m patien patie ing ts a orati synth lops a devel s of ng diagn nt/fam stand h?ev t/f amily temp a we ll - ops omiss ng 3. Dev esize and late w ard and ion aluat prefe osis ily prefe d, focus carrie man differe ith m ionI rence man elo ps errors agem re nces s out Deve ntial ed inima D=2 s Almos age ent of diagn manageme l 381 and lops

men and carri plan t alw and comm o missi o sis m anage plans [4/ 19/2 a unify ays on t plan nt diagn es effec unica ment , even osis, ing poten 014 out N/A and accu s* tively tes focus plan comp tial forth er apies 3: 06:04 rately issue ed licate comm , the s an and d or effici for PM] rare unica trans effec infor Alway situa feree tes

tive ent matio s effect diagn n even incorp tions, comp work ively o stic Deve oratin lex -up a in an d lops g ma nage accur nd ad apting situations, and ately out ment tailore comm a stand ma nage carries templ pla n d to unica ate ard the pla ns men and tes inform in divid t open Begin base ensu direct patien ation

comm do s to ual ring ives regar n t form from during un icati on dless and ulate 2. witho car ry of hand Makut adjus oth ers mana settin out off geme g or ting Deve thera patieesnt/fam comp lops peu in formfor uncom nt plans lexity https optim prefer tic deci ily ed diag out manaand carrie in , : //yal plicat adapt al clini ence s ed ing nospatients e.me sion

geme s a tic but r arely cal dhu s nt stand plans Effici b.co judg that resu ard consi ently in uncom te mplat m/u/a men ders deve carrie plica t * patienlt in lops e and /eva patien s ted demo t/fam and luat N/A ts, plans out mana nstra ily Does occa ions_ prefe i n mo geme tes siona pro fessi form rence in corpo

nt not st situa Effici s deve s_pr on al rating lly in ently respo int.m p an patien corporating tio ns, organ palotient/ devel nsibi h?ev ized t/fam ops fa mil lity for asse and aluat y ily prefe ssme prefe ensur carrie Reco rence ionID nt o ing ren ces s out gnize diagn f th s =23 mana effect s po e osis diagn 81[4 geme

tenti mana or a oses hando ive / 19/2 al plans nt geme difficu but vers , even 014 lty has on Deve deve comp the 3: 06:04 nt plan team unifyi lop lops licate fo r synth a ng diagn ing a d or PM] 3. Dev esize rar well e situa an d d, focus differe m anage osis tio ns, man elop s ntial incorp eoratin ment d Deve age diag and lops men and carr

plan nosis g mana a unify t plan diagn ies geme osis, ing out N/A s* nt thera plan focu pies, sed an effec and effici ent diagn tive work ostic Deve lops mana - up and geme out m and carri tailore nt p es anage d to lan ment the plans indiv Begin ba sed direct patien idu al s to on ives t form from

witho and ulate carry ut adjus others mana out geme ting Deve patien for nt plans lops t/fam uncom https ily out manaand car in prefer : //yal ries patien plica ted en ces e.me ge ment ts but dhu plans Efficie r arely b.co consi ntly in uncom m/u/a de rs deve ca rr pa tient/ plica ies lops /eva patien te d and luat ts,

N/A famil plans out man occa ions_ prefe y agem in m siona rence form in corpo ent ost lly Effici s s_pr rating incorp situations, ently int.m patien patien orati de velop t/fam h?ev t/fam ng s and ily aluat prefe ily prefe carrie ionI rence rence s ou D=2 s mana t s 381 geme [4/1 plans nt 9/20

, even 14 3: comp for 06:04 licate d or PM] rare situa incorp tions, oratin g https : //yal e.me dhu b.co m/u/a /eva luat ions_ 21 Required Milestones 21 ting patien for t/fam ily ence s prefer Assessments data from rotationspecific evaluations N/A CCC Required Milestones

Reported to ACGME Decisions ? Purposeful Workplace-based Assessment Demonstrates overall utility expected from the community Feasible and acceptable Provides educational value Demonstrates reliability Validity evidence supports use in decision-making Reasonable cost van der Vleuten, C. The Assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Sciences Education (1996) 1, 41-67. van der Vleuten, C., et al. A Model for Programmatic Assessment Fit for Purpose. Medical Teacher (2012). 34(3), 205-214. APPD LEARN Mission Meaningful educational research that advances the training of future Pediatricians via collaboration in research by program directors to improve the health and well-being of children. Network-based Assessment R&D Laboratory is the authentic clinical setting where the resident is training Multiple sites for suitable sample sizes, community input, and to accommodate variance among programs Need longitudinal data for predictive validity Network infrastructure provides coordination of IRB, data collection, scholarship APPD LEARN Operations 145+ residency programs Learners uniquely identified by a persistent encrypted identifier allows for linking over time and across studies Dataverse data repository for permanent stewardship and sharing APPD LEARN Member Sites Pediatrics Milestones Assessment Pilot NBME Center for Innovation, ABP and APPD LEARN Assessment Instrument

Development: - setting - methods - items - selection of observer roles What content informs individuals about this readiness? Examples from real world learners Readiness to: Serve as an intern on a Pediatric inpatient unit Pediatrics Milestones Assessment Pilot Data Collection Learners Programs Data collection period (months) MSFs SCOs 239 18 12 1086 713 Pilot Learner Survey - Results Thinking about the feedback you received at the end of your rotation, please indicate your level of agreement with the following statements: It was useful for constructing future goals or identifying a developmental path. It helped me to understand how those with whom I worked perceived my performance. 85%

93% Pediatrics Milestones Assessment Collaborative Vision To conceptualize and build a coherent system of competency-based assessment for physicians across the continuum Mission To uphold the public trust by providing evidence of competent performance through longitudinal assessment that: verifies continued competence beyond standardized testing of medical knowledge drives continuous performance improvement facilitates lifelong learning, and Recognizes and guides both personal and professional growth PMAC Goals To develop an assessment system that demonstrates high utility1 To engage the community in every aspect of the development and evaluation of the project Identify areas where further research is needed to provide the strength of evidence needed for decisions van der Vleuten, C., et al. A model for programmatic assessment fit for purpose. Med Teach (2012). 34(3), 205-214. 1 PMAC Assessment - Built on Key Advancement Decisions Decision Description Ready to serve as an intern Ready to care for patients without the presence of an onsite supervisor Ready to supervise others in the care of patients, without the presence of an onsite supervisor Competent in identifying areas of new learning for self, creating activities to address gaps Competent in identifying areas for improvement in systems associated with practice Ready to practice without direct supervision Time range anticipated for achievement 02/26/2020 MS-III to MSIV

PGY-1 PGY-2 PGY-3 Module 1 - Outcomes Learners Programs Data collection period (months) Total number of instruments MSFs SCOs Phase 1 Phase 2 165 8 4 1373 873 500 289 11 7 2181 1355 826 Module 2 - Outcomes Learners Programs Data collection period (months) Total number of instruments MSFs SCOs Pilot Phase 2 239 18 12 2338 1086 713 248

10 5 2181 1206 870 PMAC Outcomes Acceptability to community High participation; some express desire to drop local instruments Ease of use Feedback facilitated by content-expert-informed reports Reliability high (0.82 with 6 instruments completed) Reported results assist CCC members and supervisors in decision-making 22 Key Elements in the Development of a Workplace-based Assessment System Technology Rigor Community engagement Support and collaboration of key organizations Technology Front-end features Mobile Branching; conditional Item format types Scoring Mapping of items/elements Weighting by duration of observation and role Reporting One link per site for all instruments and modules F, R F, R F, R, N, O MSF SCO-Rounds

H&P Item #1 Recommendations Text comments Item #2 Theme-related comments Global item Global text comments F= Faculty R= Resident N= Nurse O= Other Monthly End-of-Rotation Report Provides competency-level, inference-level, and global results and feedback Score data and feedback on areas for improvement (when relevant) are given Includes all observational assessment results that met the reporting threshold Minimum criterion: 3 MSF (2 from physicians) Complete report: 3 MSF and 2 SCO of a single type Aggregate Report Provides competency-level, inference-level, and global results and feedback for all rotations that received a rotation-level report Instruments that were completed but did not meet the reporting threshold also are provided to the sites Includes additional graphical data Standardized PMAC competency-level performance compared to all other learners within a site Translation of PMAC scores to ACGME Reporting Milestones levels Psychometric Rigor Assessment evidence needs to support the intended use of the assessment Reliability Validity Extrapolation Interpretation Chain of Inference Generalization

Scoring Interpretation Extrapolation Kane MT. Validation. In: Brennan RL, ed. Educational Measurement. 4th Edition ed. Westport, CT: Praeger Publishers; 2006. Chain of Inference e Gen Scoring n atio raliz Interpretation Extrapolation Kane MT. Validation. In: Brennan RL, ed. Educational Measurement. 4th Edition ed. Westport, CT: Praeger Publishers; 2006. Community Engagement Selection of important decisions Identification of assessment content that informs chosen decisions Determination of which interprofessional role most naturally sees behaviors to be assessed Instrument type chosen Review of preliminary items / instruments APPD LEARN Sites Engaged in PMAC Data Collection 2011-present Longitudinal Support from Key Organizations Align with vision/mission of organization Financial support needed is substantial Collaboration with determination and tolerance Scholarship External advisory committee

Next Steps Next Module Changes in front-end design Expanding into other specialties Expanding into other methodologies Instrument Completion Status: Programs Live LinkData to PMAC Database Real-time Review Instrument Selection Instruments offered based on role-privileges Learner Selection Only assigned learners pictured Slider Bar Format Reduces rater tension with assigning numeric score Global Item Use of scale; correlation with other items Recommendations Immediate Feedback and Easy Entry back into System Easy re-entry to complete another instrument Scroll to view submitted instrument PDF to save +/print +/- send to learner

Recently Viewed Presentations