Human or AI? Comparing Design Thinking Assessments by Teaching Assistants and Bots
Sumbul Khan
Singapore University of Technology and Design
Candidates are nominated by the technical program committee chair based on recommendations of technical program committee members and reviewers. The general chair together with the technical program committee chair are responsible for nominating a best paper award committee and sending a final shortlist to this committee for selection. The number of the papers in the shortlist shall represent the top 1% of all the submissions in the year (or 3 papers, whichever is smaller) for one category.
After the candidate list has been established, the general chair and technical program committee chair propose a best paper award committee. Members of the committee cannot have a paper in the shortlist. The committee has at least 3 members and is approved by the organizing committee. The committee should be broad enough in expertise to represent the IEEE TALE areas and should as much as possible represent a mix of academic lineage and demographics. Committee members should have a record of serving on the IEEE TALE or related conferences by the IEEE Education Society. One of the members should take up the role as the chairperson for making the final judgment after considering the reviews by the members.
The committee receives the shortlisted papers with paper metadata including but not limited to anonymized reviews and best paper nomination scores, before the first day of the conference welcoming for the initial review of these papers.
The main task of the committee is to select the best papers from the shortlist to receive the award. All these papers should be eligible candidates to receive the award. The reviewing process typically involves the following steps: (1) Gathering preliminary brief review comments from the award committee members prior to the conference; (2) Attending the oral presentation and asking probing questions for clarification; (3) Giving independent overall rating on each candidate; then 3) Selecting candidates for the best paper award. The final selection should be sent by the chair of the award committee to the technical program committee chair and the general chair for confirmation.
The number of best paper awards given in a year should not be significantly higher than a usual practice. For reference, usually the top 5 papers were shortlisted to be considered for the award in the past TALE conferences. The committee may use their discretion and professional judgment to decide if the awardees could represent the best work worth to be promoted by the conference.
The award committee members may consider the merits of the following criteria:
Contribution (15%): Have the authors explicitly and clearly articulated how their research will contribute to current work?
Theoretical and Methodological Orientation (15%): Have the authors explicitly and clearly described how they performed their research? To what extent does their approach align with currently accepted theories and methodologies?
Findings and Conclusions (15%): To what extent are research findings compelling and different from what has been previously published? To what extent have the research findings contributed to existing knowledge and/or literature? Would the findings be worthwhile to be extended and published in any journal? Are implications for future research considered? If appropriate, are implications for practice considered?
Organization, Illustrations and Oral Presentation Clarity (55%): To what extent does the paper and demonstrate effectively organized, clear, and concise presentation appropriate for the readership of the Proceedings? To what extent do tables and figures meaningfully add to the narrative? Is the author as the presenter able to present the paper with clarity to demonstrate a full knowledge of the work? Can the presenter answer the questions from the audience clearly?
Any New Contribution or existing one with new methodology / execution
The score in each category should be given by the committee members after attending the oral presentation, and the overall score should be used in determining the best paper awards regardless of the peer review scores. In other words, the paper with a higher peer review score should not be more advantageous than the one with lower peer review score. The category 4 is given the highest weight, carrying the common practice in the conference that the best paper is mainly reviewed based on the oral presentation clarity.
Each category should hold a sub-score based on the 5-Likert scale, as follows:
1 = Poor
2 = Fair
3 = Good
4 = Excellent
5 = Outstanding
After each sub-score is collected, the following formula should be used to compute the overall score:
Total Score = (Subscore_1 + Subscore_2 + Subscore_3) x 15% + Subscore_4 x 55%
All the total scores from each committee member will be averaged as the resulting final score. If the resulting final score is at least 3 or above, it should be eligible for the best paper award. The committee should meet and discuss with each case, such that the decision can be made beyond the quantitative results.
Sumbul Khan
Singapore University of Technology and Design
Yue Chen
Queen Mary University of London
Pak Ming,
The University of Hong Kong
Hoi-Shun Lui,
The University of Queensland
Muyang Niu,
Macao Polytechnic University
Si Chen,
West Chester University
Mahesha Samaratunga,
Queen Mary University of London
Satori Hachisuka, Akiko Nakazawa, Hanako Itsubo, Naomi Iwazawa, Toru Fujimoto, and Yuhei Yamauchi
Aarsh Desai, N.V.J.K Kartik, Priyesh Gupta, Vinayak, Ashwin T S, Manjunath K. Vanahalli, and Ramkumar Rajendran
Satori Hachisuka, Kayoko Kurita, and Shinchi Warisawa,
The University of Tokyo, Japan
Eric Luk, Sunny Poon, Morris Jong, and James Au,
The Chinese University of Hong Kong, Hong Kong SAR
Chin-Ching Tsai,
National Taiwan Normal University, Taiwan
Vincent Tam,
The University of Hong Kong, Hong Kong SAR
Sophia Zhang, Truman Pham, and Herbert Thomas,
AcademyEx, New Zealand
Lamees Elhiny, Xinfeng Ye, Sathiamoorthy Manoharan, and Ulrich Speidel
Manoj Thulasidas, Kyong Jin Shim, and Jonathan Teo,
Singapore Management University
Chao Shu, Yue Chen, and Kok Keong Chai,
Queen Mary University of London
Oscar Canovas, Felix J. Garcia Clemente, and Federico Pardo,
University of Murica
X. Zhang et al.
P. -H. Chang, S. -F. Cheng, C. -W. Cheng, J. -J. Tang and C. -B. Tzeng
H. Cecotti, M. Callaghan, B. Foucher and S. Joslain
D. Gaurav et al.
M. K. Jonson Carlon, J. M. Gayed and J. S. Cross
S. Zhang, G. K. W. Wong and G. Pan
Katriona O’Sullivan
Maynooth University, Ireland
Amanda Jolliffe, Richard Ryan, and Kevin Marshall
Microsoft Ireland, Ireland
Abdul Haris Setiawan and Ryo Takaoka
Yamaguchi University, Japan
Lilis Trianingsih
Universitas Sebelas Maret, Indonesia
Takuro Owatari, Atsushi Shimada, Tsubasa Minematsu, Maiya Hori, and Rin-ichiro Taniguchi,
Kyushu University, Japan
Hua Chai, Matthew Priestley, Xiuhui Tang, and Jayashri Ravishankar,
University of New South Wales, Australia
Tim Giese, Martin Wende, Serdar Bulut, and Reiner Anderl,
Technische Universität Darmstadt, Germany
Tokio Takahashi and Masato Uchida,
The University of Hong Kong
Rex Perez Bringula, Ian Clement Fosgate, Neil Peter Garcia, and Josf Luinico Yorobe,
University of the East, Philippines
Rex Perez Bringula, Ian Clement Fosgate, Neil Peter Garcia, and Josf Luinico Yorobe,
Waseda University, Japan
Masaru Honjo and Akio Yoneyama,
KDDI Research, Inc., Japan
SVKM’s NMIMS Mukesh Patel School of Technology Management and Engineering, India
Jarmo Seppälä,
Aalto University, Finland
Shinjiro Matayoshi, Daisei Tanaka, and Sena Yun
Kyoto University of Advanced Science, Japan
M. F. Ercan and J. Caplin
R. Horst and R. Dörner
D. Y. W. Liu et al
T. Murray and L. Vigentini
S. Jiang and G. K. W. Wong
Kai Pan Mark and Lilian Lee Ping Vrijmoed
Clifford De Raffaele, Serengul Smith and Orhan Gemikonakli
Kornraphop Kawintiranon, Peerapon Vateekul, Atiwong Suchato and Proadpran Punyabukkana
Yen-Yin Wang, Yu-Chun Cheng, Chieh-Ju Kuo, I-Chang Tsai, Min-Tsuei Chen, Chin-Yu Chou and Yung-Hsuan Chen
Thomas Staubitz, Ralf Teusner, Nishanth Prakash and Christoph Meinel
Dave Towey, T.Y. Chen, Fei-Ching Kuo, Huai Liu and Zhi Quan Zhou
K. Jayakodi, M. Bandara and I. Perera
G. K. W. Wong, H. Y. Cheung, E. C. C. Ching and J. M. H. Huen
L. Jin, W. Huang and Z. Wen
Shahid Alam and LillAnne Jackson
Clinton J. Woodward, James Montgomery, Rajesh Vasa, and Andrew Cain
Chi-Un Lei, Hai-Ning Liang, and Ka Lok Man