Measuring Graduate Student Attributes: Generic Skills and ...
Keys to Successful Student Outcomes Assessments Gavin T L Brown [email protected] Keynote Presentation at the Second Annual University Teaching Forum @ King Saud University, Riyadh February 11, 2013 Presentation Objectives Participants will appreciate and understand: that the validity of assessments depends on their alignment with curricular goals, targets, and outcomes. the value of the Structure of Observed Learning Outcomes Taxonomy as a tool for ensuring coverage of surface and deep cognitive processes in the design of assessment tasks and curricular goals the importance of quality characteristics of assessment task as an essential component of validity the role that student beliefs concerning assessment and motivations for learning play in achieving learning outcomes the importance of quality, over style, in teaching practices the importance of feedback from assessment practices during instruction towards improved learning outcomes
Alignment degree to which different components that are essential to produce intended outcomes are in agreement, supportive, or consistent. best assessments are those that align with instructional aims and practices, the curricular content or material being taught, and with the purposes and motivations students and instructors bring Alignment: Assessment within teaching, learning, & curriculum Cultural, societal, & policy contexts Decide Content Design Measure Gather Information Teach
Evaluate Quality Interpretation Action Individual &Decision shared beliefs, intentions, purposes Plan Alignment to Curricular Goals If you want assessment to be aligned to curricular goals, the assessment must be within the curriculum Which is better? Assessment event, task, item Domain of interest Assessment event, task, item Domain of interest Where are we going? Curriculum Map Big ideas for each subject, e.g.,
Mathematics is about the relationships of number, chance, shape, & space Reading is about the relationships of technique, effect, purpose, and audience in the art and meaning of language This is what we are really teachingnot just a shopping list of facts & details as necessary as those are Within your disciplines you need a curriculum overview to guide assessment & teaching Curriculum Map: Assessment Because assessment is an interaction of purposes, formats, and quality requirements, I need to teach you skills & knowledge related to these factors. Assessment Validity Quality Reliability
Format Could you outline your own subject? Test-like, Formal, teachercentric Interactive, informal, studentcentric Improvement (Formative) Purpose Evaluative (Summative) Cognitive Targets: Taxonomies of Thinking A set of broad cognitive categories that describe the intellectual processes required to answer a question in a scale of increasing difficulty or complexity
Bloom Knowledge, Comprehension, Analysis, Application, Synthesis, Evaluation Structure of Observed Learning Outcomes (SOLO) Unistructural, Multistructural, Relational, Extended Abstract Why SOLO? Bloom inadequate Comprehension (Inference) easier than analysis which is harder than application? Young children can do analysis, synthesis, evaluation with easy material need not be delayed! Types not hierarchy of cognitive processing Recently revised to switch Evaluate & Synthesise The Structure of Observed Learning Outcomes (SOLO) Taxonomy A taxonomy developed by analysing the structure of student responses to assessment tasks according to grades assigned
by JB Biggs & K Collis, 1982 SURFACE (increase in quantity) Unistructural, (usually Fail) Multistructural, (normally a C) DEEP (change of quality) Relational, (a B or better) Extended Abstract (usually an A) Elements of SOLO
Given Facts, Ideas, Information Black Dots Irrelevant, not Given Xs Relevant, not Given Os Respondent Triangle Response Rs SOLO Surface: Unistructural & Multistructural Unistructural Use 1 fact or idea Multistructural Use list (2 or more) of facts or ideas but not related to each other SOLO Deep: Relational & Extended
Abstract Relational How facts or ideas are related to each other Extended Abstract The general underlying principle, rule for set of data, ideas, relationships that gives meaning to all SOLO in Higher Education A grade/excellence Sophisticated synthesis of material connecting the given material to abstractions or themes that integrate and give new meanings B grade/good Integrated analysis of given material utilising patterns and themes within expected material C grade/satisfactory A listing of relevant, given material without attention to patterns or themes
Students perceptions of effective teaching The concept of the caring teacher was particularly important at School A; clear explanation was more highly valued by students at School C; and School C student did not place as much importance on teacher humour. These variations may reflect the ethos of the school another factor Surface Questions Unistructural What kind of teacher did School A students like? _________________________________________ Multistructural What two characteristics did School C students emphasise? a) ___________________________________ b) ___________________________________ Deep: Relational What might explain the differences
between schools? a) The schools had different ethical approaches b) The teachers were of differing socioeconomic backgrounds c) The teachers at one school were more caring d) The schools had students from differing socioeconomic backgrounds Deep: Extended Abstract What do students look for in a teacher? a) Friendliness, caring, and humour b) An adult-figure not found at home c) A person from a similar background d) Whatever causes them to learn SummarySOLO as a cognitive goal SOLO is a true hierarchic taxonomy
increasing quantity & quality of thought SOLO is powerful in creating variety in the difficulty of curriculum & cognitive challenge SOLO level depends on assumed Givens the prior knowledge & tools available to students Both Surface & Deep needed, not one better than other Assessment Givens Pros & Cons for ALL methodsno perfect method of collecting information All assessment takes work in either: Design & creation Marking/scoring Manageability matters Interpretability matters Planning matters Quality issues: Selected Responses Learners choose a correct answer Positives
Highly manageable Easy to score Generates multiple samples of domain Raises mean achievement Negatives Difficult to write quality questions Tendency to surface level Dependent on reading Guessing factor Recognition of answer Quality issues: SHORT Constructed Responses
Learners write brief answers Positives Students create answer to reveal knowledge Easy to score Easy to write Negatives Scoring requires key May be difficult to restrict range of possible correct answers May be confounded with handwriting, spelling Quality issues: LONG Constructed Response Learners write long answers (essays, assignments) Positives Easy to write Forces extended intellectual engagement with domain Negatives
Very hard to score consistently & reliably Confounded with communication skills Can be prepared in advance Single sample of domain because of time factor Quality issues You cant create a perfect assessment but you can ensure that it meets quality requirements There is always error and alternative explanations Align the task(s) to the learning goals as closely as possible Check, check, check.dont trust your intuition Validity Chain Item Construction Chain as Metaphor1 All aspects are linked weakness at any one point
calls into question all inferences & decisions No one link more important than any other Links identify key aspects that must be evaluated Validation Evidence 1 Source: Crooks, T. J., Kane, M. T., & Cohen, A. S. (1996). Threats to the valid use of assessments. Assessment in Education, 3(3), 265-285. Items Domain Assessment Design Administration Scoring Performance Score Aggregation Generalisability
Merit Evaluation Action Evaluation Consequences Beliefs Beliefs and attitudes filter, frame, and guide how participants understand, react to, and behave within contexts Beliefs are products of individual experience and socialisation processes so that they are shared For successful life within a society, beliefs tend to be ecologically rational Beliefs matter because INTERPRETATION of POLICY SIGNAL Teacher/student Beliefs, Conceptions, Experiences, Values, Priorities POLICY SIGNAL Curriculum Document
INTENTION PRACTICE Classroom Activities IMPLEMENTATION Teachers (esp.) are the doers of national and local policy efforts 4 Purposes of Assessment Framework for understanding all beliefs about assessment as developed by Brown 2002, 2004 TENSION Accountability Improvement Ecological Rationality: Cultures & contexts value different things Country Teaching Level N
Improvement M (SD) Brown (2011) New Primary 573 4.10 (.69) Zealand Secondary 404 4.02 (.66) Fletcher et al. 2012 New University 877 4.44 Zealand Brown, Kennedy, Fok, Chan, & Yu (2009) Hong Kong* Primary & 288 4.76 (.38) Secondary Brown, Lake, & Matters (2011)
Queensland Primary 784 4.00 (.67) Secondary 614 3.84 (.66) Brown & Michaelides (2011) Cyprus Primary & 249 4.36 (.75) Secondary Student Accountability M (SD) School Accountability Irrelevance M (SD) M (SD) 3.54 (.95)
2.74 (1.12) 2.69 (1.10) 2.89 (.76) 2.87 (.69) 3.82 (.81) 2.81 (.83) Ecological Rationality: Cultures & contexts value different things Correlation of Improvement and School/Student Accountability by country and education level NZ Improvement School Accountability Student Accountability Qld
Primary Secondary Prosp.Tchr University 0.47 0.47 0.55 0.23 0.23 0.49 0.76 HK Spain Primary Secondary P+S
Pre-Service 0.43 0.45 0.65 0.91 0.08 0.12 0.91 0.60 Red=weak to no similarity; italic=moderate; bold=strong Students Conceptions of Assessment Students may not see assessment in the same
way as teachers because of their different roles and responsibilities Assessment makes Students Accountable. Assessment is Irrelevant, Bad, or Unfair Assessment Improves Achievement and Learning Teachers use it to guide their teaching I use it to guide my learning Assessment is emotionally Beneficial Self-regulation powerful way to understand student beliefs about assessment Student self-regulating beliefs about assessment students did not enjoy tests but believed teachers gave them to improve teaching & used them to improve their own learning Student self-regulation around tests Also Fletcher et al. found that university faculty believed in assessment for improvement much more than university students
Unless students see tests being used for improvement, students might not self-regulate SO.. Do YOU and others in your department actually use tests to give feedback or just ranking/total? Belief in duty: Cultural impact Pakeha NZ students [Western individualism influence] Asian students in NZ [Confucian filial piety influence] Feedback How do you know if You are doing your work well Your work is high quality, meeting expectations? Feedback
Your own evaluation Your own physiological responses What those around you say or do What effect your work has on the environment around you No. of No. of No. of meta-analyses Studies effects effect -size Cues 3 89
0.14 Programmed Instruction 1 40 23 -0.04 The power of feedback Hattie & Timperley (2007) Large Variable Model of Feedback Hattie & Timperley (2007) Purpose Reduce Discrepancy
CurrentGoal Student Increased effort; Change goals Teacher Appropriate goals Effective feedback 3 Questions Where going? Where Now? What Next? 4 Levels Task; Process; Self-Regulation; Self Giving effective Feedback Positive Effect Task what you need to do to improve on this task Process
what you need to do to improve on the whole process Self-Regulation what you need to do to monitor and control yourself Negative Effect Self Praise does not work!!! Good work, well done, etc. do not close the gap Likewise, blame & shame does not inform! Teacher Conceptions of Feedback: A survey study with NZ teachers Factor Feedback around accuracy with grading was clearly rejected in favour of focus on student learning and processes of learning adoption of assessment for learning that rejects grades, scores, marks, and emphasis
on correction teachers have clearly indicated a preference for growth over student wellbeing in their conceptions of feedback M se SD III Growth in Student Learning 4.90 .03 .60 IX Process Focus 4.45 .04 .86 VII Interactive
4.18 .03 .80 V Timely 3.87 .04 1.00 I Student Led 3.72 .03 .79 II Student Well-Being 3.43 .04 .82 VIII Teacher Only 3.40 .05 1.08 VI Accuracy & Grades
3.06 .04 X Expected by Community 2.88 .05 1.23 IV Ineffective Grades 2.45 .03 .84 .79 Question prompts to generate feedback Feedback level Question prompts Task-level Does this answer meet the success criteria?
Is this answer correct/incorrect? How can the student elaborate on the answer? What did the learner do well? Where did the learner go wrong? What is the correct answer? What other information is needed to meet the criteria?
What is wrong and why? What strategies did the learner use? What is the explanation for the correct answer? What other questions can the learner ask about the task? What are the relationships with other parts of the task? What other information is provided in the hand-out?
What is the learners understanding of the Process-level concepts/knowledge related to the task? Question prompts to generate feedback Feedback level Question prompts Self-regulation How can the learner monitor his/her own work done? level How can the learner carry out self-checking?
How can the learner evaluate the information provided? How can the learner reflect on his/her own learning? What did the learner do to ...? What happened when the learner ...? How can the learner account for...? What justification can be given for ...?
What further doubts does the learner have regarding this task? How does this compare to...? What do all these information have in common? What learning goals have been achieved? How have the learners ideas changed? Key ideas of feedback It is not the presence or absence of feedback that makes a difference, but its nature and quality. Feedback can have a positive or negative influence on learning. Feedback needs to be congruent with our
views of learning if they are to be valid. Feedback is an integral part of learning. Effective feedback involves the active engagement of learners in their own learning. Receiving and using feedback Three conditions for effective and useful feedback (Sadler, 2008): The learner needs the feedback The learner receives the feedback and has time to use it The learner is willing and is able to use the feedback Sadler, D.R. (2008). Beyond feedback: Developing student capability in complex appraisal. Assessment and Evaluation in Higher Education, 35(5), 535-50. SRL+Efficacy+FeedbackPerformance The importance of self-reliance, rather than dependence on others
Final Message Assessment must align to teaching goals including cognitive and curriculum content Assessment can use multiple methods provided they meet quality standards The educational (not administrative) use of assessment focuses on educational benefits Feedback to instructors and learners is needed to improve processes and outcomes Keys to Successful Student Outcomes Assessments Gavin T L Brown [email protected] Keynote Presentation at the Second Annual University Teaching Forum @ King Saud University, Riyadh February 11, 2013
Epidemiology and Importance in Pakistan 1.5 million Afghanistan, Iran, Iraq, Algeria, Saudi Arabia, Peru, and Pakistan. endemic in 88 countries,72 developing countries,350 million (WHO) Hindukush and Karakoram sub mountain range (Chitral, Dir and Gilgit) Himalayan sub mountain range (Mansehra, Abbottabad,...
Secured Transactions. Conflicts of Law. Constitutional Law (Federal and TN) Contracts. Criminal Law & Procedure. Family Law. Evidence. Professional Responsibility. Property (real and personal) Remedies. Torts. Decedents' Estates
The examiner considers information such as substrate, development method, various levels of ridge detail, and pressure distortions. A separate analysis then occurs with the exemplar print. Comparison is the side-by-side observation of the friction ridge detail in the two prints...
Photography The Elements of Art and Principles of Design Principles of Design Emphasis / Focal Point Balance Unity Contrast Movement/ Rhythm Pattern/Repetition Economy BALANCE Balance is a sense of stability in the body of work. Balance can be created by...
Provide options that are plausible with each other to attract the students to choose distracters where only those with high intellectual levels can get the best option. 2. Provide options that are grammatically consistent. If the stem is singular, the...
Ready to download the document? Go ahead and hit continue!