
Identifying Inter-rater Consistency in Design and Creativity Tasks
Consistency in inter-rater evaluations for designs is a key aspect to building confidence within academic settings. Through this study we understand further about the current and new analytical and methods applied by raters and to promote consistency in rater ratings. This study is performed a proposed sample which consists of creativity-based design projects from varying fields and situational prompts from architecture, engineering, and product design. Using statistics and clustering-based models we aim at identifying rater characteristics and unique traits which lead to enhanced inter-rater consistencies. The study enunciates commonly used statistical testing measures as Interclass Correlation Coefficient (ICCs) and Cronbach’s alpha which have been the industry standard for inter-rater reliability testing. However, with the shortcomings such as the inability to consider rater profiles and insights into design illustrations which pose difficult in being captured. To address this landscape of rater-reliability and consistency we apply data analytical techniques such as Clustering and Inter-rater combinatorics which promote assessing the inter-rater reliabilities and help ascertain confidence in the rating processes employed. The results of this study lend insights into design rating processes for creative tasks with applications to potential real-world reliable rating metrics in all spheres. The inferences imparted through this study help enhance the validity and fairness of design rating processes and promote consistency in the evaluation of creative ideations.
Advisor - Dr Scarlett Rae Miller School of Engineering Design The Pennsylvania State University, State College, Pennsylvania, 16802 Email: shm13@psu.edu
Files
Metadata
Work Title | Identifying Inter-rater Consistency in Design and Creativity Tasks |
---|---|
Access | |
Creators |
|
Keyword |
|
License | In Copyright (Rights Reserved) |
Work Type | Conference Proceeding |
Publisher |
|
Publication Date | 2023 |
Deposited | May 04, 2023 |
Versions
Analytics
Collections
This resource is currently not in any collection.