Visualization for Academic Hiring
Academic hiring committee members usually have diﬃculties when they evaluate the application materials: Going through the huge amount of unstructured information, comparing applicants, and discussing their ranking during the committee meeting. The objective of this project is to see whether a visualization tool can alleviate these problems, improving not only academic recruitment outcomes but also user experience during the process. I did this project in INRIA, Aviz team, as my master thesis.
First I studied the limitations of current evaluation approaches in academic hiring. Bibliometrics is one of the evaluation approaches and it provides a certain perspective on research achievements. However, many research pointed out the limitations of bibliometrics, such as oversimpliﬁcation (Belter, 2015; Bornmann&Daniel, 2008; Zhu et al.,2015; Radicchi et al., 2017; Lane, 2010), and inconsistency among disciplines (Belter, 2015).
Text-based assessment methods are widely used for evaluating candidates too, which includes reviewing CVs, letters of recommendation, and interviews. Of course, recruiters can gain many insights from such rich information, but obviously, these approaches require time and energy. Besides, they can lead to the wrong decision because these approaches highly depend on the evaluator’s ability (Chen et al., 2011; Rojas, 2013; Moore, 2012).
Evaluating candidates is difficult. There is no perfect solution. A reasonable approach now is to evaluate applications thoroughly and discuss extensively, but searching for information takes evaluators much time and is likely error-prone. Therefore, using visualizations as part of the process could help.
Indeed, there were some visualization approaches relevant in this context (Latif and Beck, 2018; Filipov et al., 2019). But, none of them were designed for academic hiring specifically, while providing a straightforward and easy way to review publication records and career experience.
User Requirement Analysis
Academic hiring process
First I got the knowledge about the specific academic hiring process. There are slight difference among research institutions, but it is similar in general.
Problems in this process
According to the interview results, I listed eleven problems that occur in the academic hiring process, and divided them into three categories. Some problems have been mentioned a lot, for example, hard to evaluate the usefulness of published work, and it is hard to compare applicants from different domains.
1/ REVIEW application packages
- Too much text to read and search
- No standard form for every piece of information
- Handwritten documents are hard to read
2/ EVALUATE scientific outcomes
- Hard to evaluate the usefulness and applicability of their work
- Domain-dependent publication standards
- Hard to predict the potential fairly
- Knowledge barrier between different domains
3/ COMPARE different candidates
- High memory load
- High physical load
- Hard to compare applicants from different domains
- Easy to become subjective
To understand how researchers use visualizations to illustrate their careers and scientiﬁc output, I emailed researchers in the Aviz alumni. I also used web search to ﬁnd relevant design works, in order to get more inspirations. I collected more than twenty examples. I found most of them used a timeline to represent the chronological information of education, publication, and career.
Finally, I gathered some requirements for my work:
Give enough candidate’s information, including scientiﬁc outcomes, mobility, so that evaluators can have a rough but relatively accurate image of the candidates.
Give a global picture of all candidates, so that evaluators can have a baseline.
Allow comparing different candidates visually, so that evaluators can recognize their strengths and weaknesses easily.
Users need a straightforward representation, so a timeline could be an initial design.
Focused on hiring for junior researchers, a visualization tool should be designed to increase the quality of the evaluations, as well as the efficiency and conﬁdence of the user during the hiring process
After that, I chose which data to use in my design. Some of the data I used comes from the applicant’s CV, including education background, research experience, publications, and personal information. Citation counts were fromGoogle ScholarAnd the ratings of journals and conferences were from expert suggestions andCORE Rank.
But this design can only tell the overall publication results, but not tell the details about each publication.
Considering the limited time, I focused on two important dimensions: (1) readability and usability of the visualization (2) inﬂuence of the visualization on discussions in an academic hiring committee.
Evaluation Study I
I first conducted a survey on the information accuracy and usability of the visualization with six junior researchers who provided the data for my final design.
Because they would care more about the data and find detailed accuracy and usability issues easier. I received much feedback then updated my data and my design.
Evaluation Study II
I used the improved visualization in study II. In study II, I set up a simulated hiring committee meeting with three researcher participants.
In this study, they were asked to review the application packages of the junior researchers from study I. There were two phases (One is for the control condition, one is for the experimental condition), the variable of the study is whether using a visualization tool. They used only text CVs to evaluate the candidates and gave ranking in the first phase, but in the second phase, they were allowed to use the visualization tool.
I found there are four reasons why they used the visualization on discussions.
They used it to verify the previous ranking
Brought interesting points into discussion
They used it to aid in understanding text information
They used it as a medium to communicate
In the post-survey of study 2, the participants were positive for using the visualization in evaluating, discussing, and comparing. In the interview after the study two, they said the biggest advantage of the tool is reducing the reviewing time.
Discussion about the design
Finding its usefulness is not the end. I finally concluded some potential benefits and drawbacks of using visualization in academic hiring:
The data used for the design is only a small part of what evaluators consider in academic hiring. Functions such as reordering timelines or aligning them in different ways could also be useful in this context. As for future work, a better visualization tool need to be designed.
In evaluation study: (1) The participants were mostly in the HCI field, which is different from real academic hiring. (2) I used the same participants for both the control condition and the experimental condition, which might influence the results. Future work could invite more researcher participants to better simulate a hiring committee and get more reliable results. (3) This research only started to explore potential benefits of visualizations, more studies should be conducted to evaluate my initial research objective: improve the quality of the outcomes and user experience of hiring.
Finally · Relevant Links
I would like to thank my thesis advisor Pierre Dragicevic and Petra Isenberg. The door to their oﬃce was always open whenever I ran into a trouble spot or had a question about my research or design. They consistently steered me in the right the direction whenever they thought I needed it.