Measuring Graduate Student Attributes: Generic Skills and ...

Keys to Successful Student Outcomes Assessments Gavin T L Brown [email protected] Keynote Presentation at the Second Annual University Teaching Forum @ King Saud University, Riyadh February 11, 2013 Presentation Objectives Participants will appreciate and understand: that the validity of assessments depends on their alignment with curricular goals, targets, and outcomes. the value of the Structure of Observed Learning Outcomes Taxonomy as a tool for ensuring coverage of surface and deep cognitive processes in the design of assessment tasks and curricular goals the importance of quality characteristics of assessment task as an essential component of validity the role that student beliefs concerning assessment and motivations for learning play in achieving learning outcomes the importance of quality, over style, in teaching practices the importance of feedback from assessment practices during instruction towards improved learning outcomes

Alignment degree to which different components that are essential to produce intended outcomes are in agreement, supportive, or consistent. best assessments are those that align with instructional aims and practices, the curricular content or material being taught, and with the purposes and motivations students and instructors bring Alignment: Assessment within teaching, learning, & curriculum Cultural, societal, & policy contexts Decide Content Design Measure Gather Information Teach

Evaluate Quality Interpretation Action Individual &Decision shared beliefs, intentions, purposes Plan Alignment to Curricular Goals If you want assessment to be aligned to curricular goals, the assessment must be within the curriculum Which is better? Assessment event, task, item Domain of interest Assessment event, task, item Domain of interest Where are we going? Curriculum Map Big ideas for each subject, e.g.,

Mathematics is about the relationships of number, chance, shape, & space Reading is about the relationships of technique, effect, purpose, and audience in the art and meaning of language This is what we are really teachingnot just a shopping list of facts & details as necessary as those are Within your disciplines you need a curriculum overview to guide assessment & teaching Curriculum Map: Assessment Because assessment is an interaction of purposes, formats, and quality requirements, I need to teach you skills & knowledge related to these factors. Assessment Validity Quality Reliability

Format Could you outline your own subject? Test-like, Formal, teachercentric Interactive, informal, studentcentric Improvement (Formative) Purpose Evaluative (Summative) Cognitive Targets: Taxonomies of Thinking A set of broad cognitive categories that describe the intellectual processes required to answer a question in a scale of increasing difficulty or complexity

Bloom Knowledge, Comprehension, Analysis, Application, Synthesis, Evaluation Structure of Observed Learning Outcomes (SOLO) Unistructural, Multistructural, Relational, Extended Abstract Why SOLO? Bloom inadequate Comprehension (Inference) easier than analysis which is harder than application? Young children can do analysis, synthesis, evaluation with easy material need not be delayed! Types not hierarchy of cognitive processing Recently revised to switch Evaluate & Synthesise The Structure of Observed Learning Outcomes (SOLO) Taxonomy A taxonomy developed by analysing the structure of student responses to assessment tasks according to grades assigned

by JB Biggs & K Collis, 1982 SURFACE (increase in quantity) Unistructural, (usually Fail) Multistructural, (normally a C) DEEP (change of quality) Relational, (a B or better) Extended Abstract (usually an A) Elements of SOLO

Given Facts, Ideas, Information Black Dots Irrelevant, not Given Xs Relevant, not Given Os Respondent Triangle Response Rs SOLO Surface: Unistructural & Multistructural Unistructural Use 1 fact or idea Multistructural Use list (2 or more) of facts or ideas but not related to each other SOLO Deep: Relational & Extended

Abstract Relational How facts or ideas are related to each other Extended Abstract The general underlying principle, rule for set of data, ideas, relationships that gives meaning to all SOLO in Higher Education A grade/excellence Sophisticated synthesis of material connecting the given material to abstractions or themes that integrate and give new meanings B grade/good Integrated analysis of given material utilising patterns and themes within expected material C grade/satisfactory A listing of relevant, given material without attention to patterns or themes

Students perceptions of effective teaching The concept of the caring teacher was particularly important at School A; clear explanation was more highly valued by students at School C; and School C student did not place as much importance on teacher humour. These variations may reflect the ethos of the school another factor Surface Questions Unistructural What kind of teacher did School A students like? _________________________________________ Multistructural What two characteristics did School C students emphasise? a) ___________________________________ b) ___________________________________ Deep: Relational What might explain the differences

between schools? a) The schools had different ethical approaches b) The teachers were of differing socioeconomic backgrounds c) The teachers at one school were more caring d) The schools had students from differing socioeconomic backgrounds Deep: Extended Abstract What do students look for in a teacher? a) Friendliness, caring, and humour b) An adult-figure not found at home c) A person from a similar background d) Whatever causes them to learn SummarySOLO as a cognitive goal SOLO is a true hierarchic taxonomy

increasing quantity & quality of thought SOLO is powerful in creating variety in the difficulty of curriculum & cognitive challenge SOLO level depends on assumed Givens the prior knowledge & tools available to students Both Surface & Deep needed, not one better than other Assessment Givens Pros & Cons for ALL methodsno perfect method of collecting information All assessment takes work in either: Design & creation Marking/scoring Manageability matters Interpretability matters Planning matters Quality issues: Selected Responses Learners choose a correct answer Positives

Highly manageable Easy to score Generates multiple samples of domain Raises mean achievement Negatives Difficult to write quality questions Tendency to surface level Dependent on reading Guessing factor Recognition of answer Quality issues: SHORT Constructed Responses

Learners write brief answers Positives Students create answer to reveal knowledge Easy to score Easy to write Negatives Scoring requires key May be difficult to restrict range of possible correct answers May be confounded with handwriting, spelling Quality issues: LONG Constructed Response Learners write long answers (essays, assignments) Positives Easy to write Forces extended intellectual engagement with domain Negatives

Very hard to score consistently & reliably Confounded with communication skills Can be prepared in advance Single sample of domain because of time factor Quality issues You cant create a perfect assessment but you can ensure that it meets quality requirements There is always error and alternative explanations Align the task(s) to the learning goals as closely as possible Check, check, check.dont trust your intuition Validity Chain Item Construction Chain as Metaphor1 All aspects are linked weakness at any one point

calls into question all inferences & decisions No one link more important than any other Links identify key aspects that must be evaluated Validation Evidence 1 Source: Crooks, T. J., Kane, M. T., & Cohen, A. S. (1996). Threats to the valid use of assessments. Assessment in Education, 3(3), 265-285. Items Domain Assessment Design Administration Scoring Performance Score Aggregation Generalisability

Merit Evaluation Action Evaluation Consequences Beliefs Beliefs and attitudes filter, frame, and guide how participants understand, react to, and behave within contexts Beliefs are products of individual experience and socialisation processes so that they are shared For successful life within a society, beliefs tend to be ecologically rational Beliefs matter because INTERPRETATION of POLICY SIGNAL Teacher/student Beliefs, Conceptions, Experiences, Values, Priorities POLICY SIGNAL Curriculum Document

INTENTION PRACTICE Classroom Activities IMPLEMENTATION Teachers (esp.) are the doers of national and local policy efforts 4 Purposes of Assessment Framework for understanding all beliefs about assessment as developed by Brown 2002, 2004 TENSION Accountability Improvement Ecological Rationality: Cultures & contexts value different things Country Teaching Level N

Improvement M (SD) Brown (2011) New Primary 573 4.10 (.69) Zealand Secondary 404 4.02 (.66) Fletcher et al. 2012 New University 877 4.44 Zealand Brown, Kennedy, Fok, Chan, & Yu (2009) Hong Kong* Primary & 288 4.76 (.38) Secondary Brown, Lake, & Matters (2011)

Queensland Primary 784 4.00 (.67) Secondary 614 3.84 (.66) Brown & Michaelides (2011) Cyprus Primary & 249 4.36 (.75) Secondary Student Accountability M (SD) School Accountability Irrelevance M (SD) M (SD) 3.54 (.95)

2.70 (1.01) 2.90 (.68) 3.93 (.86) 2.68 (1.06) 2.90 (.65) 4.13 2.84 2.92 4.70 (.45) 4.09 (.62) 2.39 (.42) 3.64 (.76) 3.93 (.75)

2.74 (1.12) 2.69 (1.10) 2.89 (.76) 2.87 (.69) 3.82 (.81) 2.81 (.83) Ecological Rationality: Cultures & contexts value different things Correlation of Improvement and School/Student Accountability by country and education level NZ Improvement School Accountability Student Accountability Qld

Primary Secondary Prosp.Tchr University 0.47 0.47 0.55 0.23 0.23 0.49 0.76 HK Spain Primary Secondary P+S

Pre-Service 0.43 0.45 0.65 0.91 0.08 0.12 0.91 0.60 Red=weak to no similarity; italic=moderate; bold=strong Students Conceptions of Assessment Students may not see assessment in the same

way as teachers because of their different roles and responsibilities Assessment makes Students Accountable. Assessment is Irrelevant, Bad, or Unfair Assessment Improves Achievement and Learning Teachers use it to guide their teaching I use it to guide my learning Assessment is emotionally Beneficial Self-regulation powerful way to understand student beliefs about assessment Student self-regulating beliefs about assessment students did not enjoy tests but believed teachers gave them to improve teaching & used them to improve their own learning Student self-regulation around tests Also Fletcher et al. found that university faculty believed in assessment for improvement much more than university students

Unless students see tests being used for improvement, students might not self-regulate SO.. Do YOU and others in your department actually use tests to give feedback or just ranking/total? Belief in duty: Cultural impact Pakeha NZ students [Western individualism influence] Asian students in NZ [Confucian filial piety influence] Feedback How do you know if You are doing your work well Your work is high quality, meeting expectations? Feedback

Your own evaluation Your own physiological responses What those around you say or do What effect your work has on the environment around you No. of No. of No. of meta-analyses Studies effects effect -size Cues 3 89

129 1.10 Feedback 74 4157 5755 0.95 Reinforcement 1 19 19 0.94

Video/ audio feedback 1 91 715 0.64 Medium CAI feedback 4 161 129 0.52 Goals & feedback 8 640

121 0.46 Student evaluation feedback Corrective feedback 3 25 100 1149 61 1040 0.42 0.37 Small Summaryof eff ect-sizesrelatingtofeedback eff

ects Delayed vs. immediate 5 178 83 0.34 Reward 3 223 508 0.31 Immediate vs. delayed 8

398 167 0.24 Punishment 1 89 210 0.20 Praise 11 388 4410

0.14 Programmed Instruction 1 40 23 -0.04 The power of feedback Hattie & Timperley (2007) Large Variable Model of Feedback Hattie & Timperley (2007) Purpose Reduce Discrepancy

CurrentGoal Student Increased effort; Change goals Teacher Appropriate goals Effective feedback 3 Questions Where going? Where Now? What Next? 4 Levels Task; Process; Self-Regulation; Self Giving effective Feedback Positive Effect Task what you need to do to improve on this task Process

what you need to do to improve on the whole process Self-Regulation what you need to do to monitor and control yourself Negative Effect Self Praise does not work!!! Good work, well done, etc. do not close the gap Likewise, blame & shame does not inform! Teacher Conceptions of Feedback: A survey study with NZ teachers Factor Feedback around accuracy with grading was clearly rejected in favour of focus on student learning and processes of learning adoption of assessment for learning that rejects grades, scores, marks, and emphasis

on correction teachers have clearly indicated a preference for growth over student wellbeing in their conceptions of feedback M se SD III Growth in Student Learning 4.90 .03 .60 IX Process Focus 4.45 .04 .86 VII Interactive

4.18 .03 .80 V Timely 3.87 .04 1.00 I Student Led 3.72 .03 .79 II Student Well-Being 3.43 .04 .82 VIII Teacher Only 3.40 .05 1.08 VI Accuracy & Grades

3.06 .04 X Expected by Community 2.88 .05 1.23 IV Ineffective Grades 2.45 .03 .84 .79 Question prompts to generate feedback Feedback level Question prompts Task-level Does this answer meet the success criteria?

Is this answer correct/incorrect? How can the student elaborate on the answer? What did the learner do well? Where did the learner go wrong? What is the correct answer? What other information is needed to meet the criteria?

What is wrong and why? What strategies did the learner use? What is the explanation for the correct answer? What other questions can the learner ask about the task? What are the relationships with other parts of the task? What other information is provided in the hand-out?

What is the learners understanding of the Process-level concepts/knowledge related to the task? Question prompts to generate feedback Feedback level Question prompts Self-regulation How can the learner monitor his/her own work done? level How can the learner carry out self-checking?

How can the learner evaluate the information provided? How can the learner reflect on his/her own learning? What did the learner do to ...? What happened when the learner ...? How can the learner account for...? What justification can be given for ...?

What further doubts does the learner have regarding this task? How does this compare to...? What do all these information have in common? What learning goals have been achieved? How have the learners ideas changed? Key ideas of feedback It is not the presence or absence of feedback that makes a difference, but its nature and quality. Feedback can have a positive or negative influence on learning. Feedback needs to be congruent with our

views of learning if they are to be valid. Feedback is an integral part of learning. Effective feedback involves the active engagement of learners in their own learning. Receiving and using feedback Three conditions for effective and useful feedback (Sadler, 2008): The learner needs the feedback The learner receives the feedback and has time to use it The learner is willing and is able to use the feedback Sadler, D.R. (2008). Beyond feedback: Developing student capability in complex appraisal. Assessment and Evaluation in Higher Education, 35(5), 535-50. SRL+Efficacy+FeedbackPerformance The importance of self-reliance, rather than dependence on others

Final Message Assessment must align to teaching goals including cognitive and curriculum content Assessment can use multiple methods provided they meet quality standards The educational (not administrative) use of assessment focuses on educational benefits Feedback to instructors and learners is needed to improve processes and outcomes Keys to Successful Student Outcomes Assessments Gavin T L Brown [email protected] Keynote Presentation at the Second Annual University Teaching Forum @ King Saud University, Riyadh February 11, 2013

Recently Viewed Presentations

  • CUTANEOUS LEISHMANIASIS Anam Saghir 12-10007 Cutaneous Leishmaniasis  Infectious

    CUTANEOUS LEISHMANIASIS Anam Saghir 12-10007 Cutaneous Leishmaniasis Infectious

    Epidemiology and Importance in Pakistan 1.5 million Afghanistan, Iran, Iraq, Algeria, Saudi Arabia, Peru, and Pakistan. endemic in 88 countries,72 developing countries,350 million (WHO) Hindukush and Karakoram sub mountain range (Chitral, Dir and Gilgit) Himalayan sub mountain range (Mansehra, Abbottabad,...
  • Websense Confidential Template 2012 4:3

    Websense Confidential Template 2012 4:3

    The Big Picture. ThreatVision is a POC Hotfix for v7.7.3 WSG/A. Hotfix not available to customers. See internal KB . article. Requires SE setup. Applies to V10000 G2R2 and V10000 G3 only
  • ACADEMIC ADVISING - University of Memphis

    ACADEMIC ADVISING - University of Memphis

    Secured Transactions. Conflicts of Law. Constitutional Law (Federal and TN) Contracts. Criminal Law & Procedure. Family Law. Evidence. Professional Responsibility. Property (real and personal) Remedies. Torts. Decedents' Estates
  • Hybrid Orbitals - Science

    Hybrid Orbitals - Science

    MO and VB result in the same. The paired electrons spread out to form the final electron clouds. Difference. VB Theory: electrons of molecules move on the atomic orbitals.
  • Recent Trends in Fingerprint Evidence 2012 Texas Forensic

    Recent Trends in Fingerprint Evidence 2012 Texas Forensic

    The examiner considers information such as substrate, development method, various levels of ridge detail, and pressure distortions. A separate analysis then occurs with the exemplar print. Comparison is the side-by-side observation of the friction ridge detail in the two prints...
  • God called. Jonah ran. God chased. Jonah obeyed

    God called. Jonah ran. God chased. Jonah obeyed

    1 Jonah prayed to the LORD his God from inside the fish: 2 I called to the LORD in my distress, and He answered me. I cried out for help in the belly of Sheol; You heard my voice.
  • Photography Composition - Weebly

    Photography Composition - Weebly

    Photography The Elements of Art and Principles of Design Principles of Design Emphasis / Focal Point Balance Unity Contrast Movement/ Rhythm Pattern/Repetition Economy BALANCE Balance is a sense of stability in the body of work. Balance can be created by...
  • Chapter #3

    Chapter #3

    Provide options that are plausible with each other to attract the students to choose distracters where only those with high intellectual levels can get the best option. 2. Provide options that are grammatically consistent. If the stem is singular, the...