Identifying Inter-rater Consistency in Design and Creativity Tasks

Consistency in inter-rater evaluations for designs is a key aspect to building confidence within academic settings. Through this study we understand further about the current and new analytical and methods applied by raters and to promote consistency in rater ratings. This study is performed a proposed sample which consists of creativity-based design projects from varying fields and situational prompts from architecture, engineering, and product design. Using statistics and clustering-based models we aim at identifying rater characteristics and unique traits which lead to enhanced inter-rater consistencies. The study enunciates commonly used statistical testing measures as Interclass Correlation Coefficient (ICCs) and Cronbach’s alpha which have been the industry standard for inter-rater reliability testing. However, with the shortcomings such as the inability to consider rater profiles and insights into design illustrations which pose difficult in being captured. To address this landscape of rater-reliability and consistency we apply data analytical techniques such as Clustering and Inter-rater combinatorics which promote assessing the inter-rater reliabilities and help ascertain confidence in the rating processes employed. The results of this study lend insights into design rating processes for creative tasks with applications to potential real-world reliable rating metrics in all spheres. The inferences imparted through this study help enhance the validity and fairness of design rating processes and promote consistency in the evaluation of creative ideations.

Advisor - Dr Scarlett Rae Miller School of Engineering Design The Pennsylvania State University, State College, Pennsylvania, 16802 Email: shm13@psu.edu

Files

Metadata

Work Title Identifying Inter-rater Consistency in Design and Creativity Tasks
Access
Open Access
Creators
  1. Aditya Atul Gambhir
  2. Dr. Scarlett Rae Miller
Keyword
  1. Creativity
  2. Consistency
  3. Clustering
  4. Data Analytics
  5. Inter-rater reliability
  6. Cronbach’s alpha
License In Copyright (Rights Reserved)
Work Type Conference Proceeding
Publisher
  1. Proceedings of the ASME 2022 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
Publication Date 2023
Deposited May 04, 2023

Versions

Analytics

Collections

This resource is currently not in any collection.

Work History

Version 1
published

  • Created
  • Updated
  • Added Creator Aditya Atul Gambhir
  • Added Creator Scarlett Miller
  • Added Paper - Aditya A. Gambhir.pdf
  • Updated License Show Changes
    License
    • https://rightsstatements.org/page/InC/1.0/
  • Renamed Creator Dr. Scarlett Rae Miller Show Changes
    • Scarlett Miller
    • Dr. Scarlett Rae Miller
  • Published
  • Updated Work Title, Publisher Show Changes
    Work Title
    • IDENTIFYING INTER-RATER CONSISTENCY IN DESIGN AND CREATIVITY TASKS
    • Identifying Inter-rater Consistency in Design and Creativity Tasks
    Publisher
    • Proceedings of the ASME 2022 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
  • Updated