Visualization for Academic Hiring

为学术招聘进行数据可视化工具设计

2019

INRIA, Aviz

Academic hiring committee members usually have difficulties when they evaluate the application materials: Going through the huge amount of unstructured information, comparing applicants, and discussing their ranking during the committee meeting. The objective of this project is to see whether a visualization tool can alleviate these problems, improving not only academic recruitment outcomes but also user experience during the process. I did this project in INRIA, Aviz team, as my master thesis.

01 研究和设计方法
Methodology
02 文献研究&相关产品
Literature Research

First I studied the limitations of current evaluation approaches in academic hiring. Bibliometrics is one of the evaluation approaches and it provides a certain perspective on research achievements. However, many research pointed out the limitations of bibliometrics, such as oversimplification (Belter, 2015; Bornmann&Daniel, 2008; Zhu et al.,2015; Radicchi et al., 2017; Lane, 2010), and inconsistency among disciplines (Belter, 2015).

Text-based assessment methods are widely used for evaluating candidates too, which includes reviewing CVs, letters of recommendation, and interviews. Of course, recruiters can gain many insights from such rich information, but obviously, these approaches require time and energy. Besides, they can lead to the wrong decision because these approaches highly depend on the evaluator’s ability (Chen et al., 2011; Rojas, 2013; Moore, 2012).

Evaluating candidates is difficult. There is no perfect solution. A reasonable approach now is to evaluate applications thoroughly and discuss extensively, but searching for information takes evaluators much time and is likely error-prone. Therefore, using visualizations as part of the process could help.

Indeed, there were some visualization approaches relevant in this context (Latif and Beck, 2018; Filipov et al., 2019). But, none of them were designed for academic hiring specifically, while providing a straightforward and easy way to review publication records and career experience.

03 用户需求调研
User Requirement Analysis

In order to do research through design, I needed to understand a specific academic hiring process and the problems recruiters might encounter in this context. Therefore, I interviewed four researchers with past experience in academic hiring.

Academic hiring process

First I got the knowledge about the specific academic hiring process. There are slight difference among research institutions, but it is similar in general.

Problems in this process

According to the interview results, I listed eleven problems that occur in the academic hiring process, and divided them into three categories. Some problems have been mentioned a lot, for example, hard to evaluate the usefulness of published work, and it is hard to compare applicants from different domains.

1/ REVIEW application packages

  • Too much text to read and search
  • No standard form for every piece of information
  • Handwritten documents are hard to read

2/ EVALUATE scientific outcomes

  • Hard to evaluate the usefulness and applicability of their work
  • Domain-dependent publication standards
  • Hard to predict the potential fairly
  • Knowledge barrier between different domains

3/ COMPARE different candidates

  • High memory load
  • High physical load
  • Hard to compare applicants from different domains
  • Easy to become subjective

Email Survey

To understand how researchers use visualizations to illustrate their careers and scientific output, I emailed researchers in the Aviz alumni. I also used web search to find relevant design works, in order to get more inspirations. I collected more than twenty examples. I found most of them used a timeline to represent the chronological information of education, publication, and career.

Finally, I gathered some requirements for my work:

Give enough candidate’s information, including scientific outcomes, mobility, so that evaluators can have a rough but relatively accurate image of the candidates.

Give a global picture of all candidates, so that evaluators can have a baseline.

Allow comparing different candidates visually, so that evaluators can recognize their strengths and weaknesses easily.

Users need a straightforward representation, so a timeline could be an initial design.

04 可视化设计
Visualization Design

Focused on hiring for junior researchers, a visualization tool should be designed to increase the quality of the evaluations, as well as the efficiency and confidence of the user during the hiring process

After that, I chose which data to use in my design. Some of the data I used comes from the applicant’s CV, including education background, research experience, publications, and personal information. Citation counts were fromGoogle ScholarAnd the ratings of journals and conferences were from expert suggestions andCORE Rank.

Combing with design and heuristic evaluation, I delivered more than ten design concepts

Ad-hoc Design

I began with an ad-hoc design and then experimented with iterations. The ad-hoc design comes from one of my supervisor Petra Isenberg, and was used in the hiring committee she attended.
In this design, each line is a candidate. The timeline shows the time spent in Ph.D. and postdoc. The numbers of publication in journal (J) and conference (C) are shown. Finally, there are the grades from evaluators.

Reflection

This design was considered useful by committee members but it only shows quantitative data and ignores the quality of the scientific outputs.

First Iteration

Here is the information visualization for one candidate. Age distribution among applicants is shown. I tried to add more publication details, including publications and citation counts for each year on the timeline.

Reflection

But this design can only tell the overall publication results, but not tell the details about each publication.

Second Iteration

In order to show richer data, I made multiple timelines to represent only one candidate. I used blocked color because it can help the user to gain a resulting imagery, which will help them to better remember the profile. The citation count of each publication is shown. In this way, users can recognize the citation distribution very clear by the density of line.

Reflection

But this design can only tell the overall publication results, but not tell the details about each publication.

Third Iteration

I made seven variations from one researcher’s data, in order to facilitate the discussion about the choice of visual encodings. I made a more compact layout for each candidate’s timeline.

Reflection

Although these designs can explain quality of publications better, I think for young researchers, their independence should be considered: an applicant who always publishes with their advisor is quite different from an applicant with large collaboration networks. Therefore, these differences need to be better revealed.

Fourth Iteration

I used the real data of six junior researchers (Two of them are anonymized because of their concerns of confidentiality ). Each line is one of them. Beside basic visual encoding, I refined the author order as well, using clearer glyphs. I also added basic interaction: when the mouse hovers on a circle, it will show the details. I implemented the visualization using javascript.Click to use the Live Demo

Try different colors to verify the visual recognition of the design.

05 设计评估
Evaluation

Considering the limited time, I focused on two important dimensions: (1) readability and usability of the visualization (2) influence of the visualization on discussions in an academic hiring committee.

Evaluation Study I

I first conducted a survey on the information accuracy and usability of the visualization with six junior researchers who provided the data for my final design.

Because they would care more about the data and find detailed accuracy and usability issues easier. I received much feedback then updated my data and my design.

Evaluation Study II

I used the improved visualization in study II. In study II, I set up a simulated hiring committee meeting with three researcher participants.
In this study, they were asked to review the application packages of the junior researchers from study I. There were two phases (One is for the control condition, one is for the experimental condition), the variable of the study is whether using a visualization tool. They used only text CVs to evaluate the candidates and gave ranking in the first phase, but in the second phase, they were allowed to use the visualization tool.

Evaluation Results

I found there are four reasons why they used the visualization on discussions.

They used it to verify the previous ranking

Someone looked at the visualization and said: In our ranking, the first one looks good, we made the right decision, and the second one….

Brought interesting points into discussion

Someone reviewed the visualization, then pointed at one element and said: Look here…

They used it to aid in understanding text information

Someone was confused about some publications, then he found the visual representations of these publications.

They used it as a medium to communicate

Someone talked by looking at and pointing at the visualization (printed version).

In the post-survey of study 2, the participants were positive for using the visualization in evaluating, discussing, and comparing. In the interview after the study two, they said the biggest advantage of the tool is reducing the reviewing time.

06 对设计的讨论
Discussion about the design

Design Value

Finding its usefulness is not the end. I finally concluded some potential benefits and drawbacks of using visualization in academic hiring:

  • null

    It can reduce the time spent in reviewing an applicant’s materials, and it can help the evaluator get a basic knowledge about the research background of all applicants quickly

    From interview after evaluation study II

  • null

    It can serve as a medium to help committee members focus on the information that was discussed

    From observation in evaluation study II

  • null

    It may reduce emphasis on qualitative data

    From interview in user requirement study

  • null

    It can aid in the acquisition and understanding of textual information on CVs.

    From observation in evaluation study II

  • null

    Visually ranking candidates can help committees to gain quick reflections

    From observation in evaluation study II

  • null

    It may cause evaluators to the misreading of publishing trends. For example, the quality of an applicant’s publications seems to be declining in the visualization because newer publications receive fewer citations due to time accumulation

    From interview in user requirement study

Limitation&Future work

The data used for the design is only a small part of what evaluators consider in academic hiring. Functions such as reordering timelines or aligning them in different ways could also be useful in this context. As for future work, a better visualization tool need to be designed.

In evaluation study: (1) The participants were mostly in the HCI field, which is different from real academic hiring. (2) I used the same participants for both the control condition and the experimental condition, which might influence the results. Future work could invite more researcher participants to better simulate a hiring committee and get more reliable results. (3) This research only started to explore potential benefits of visualizations, more studies should be conducted to evaluate my initial research objective: improve the quality of the outcomes and user experience of hiring.

Finally · Relevant Links

Acknowledgments

I would like to thank my thesis advisor Pierre Dragicevic and Petra Isenberg. The door to their office was always open whenever I ran into a trouble spot or had a question about my research or design. They consistently steered me in the right the direction whenever they thought I needed it.