Who Shortlists? Evidence on Gender Disparities in Hiring Outcomes

Authors: Almasa Sarabi (Amsterdam Business School); Nico Lehmann (Erasmus School of Economics)
Interviewers: Irna Ishrat (Aligarh Muslim University); Alara Cansu Yaman (University of Goettingen)

Article link: https://doi.org/10.1177/0001839224128394

It is an impressive study that uses four different data sources and employs both quantitative and qualitative methods. What was the initial puzzle driving this study? Was the motivation rooted in real-time phenomena, such as organizational practice (observing shifting responsibility from hiring managers to HR), or from gaps in academic literature about decision-maker roles in hiring?

First of all, thank you for the great questions! The puzzle really came from what we observed in practice. We had the opportunity to collaborate with a multinational company we call Alpha, which was in the process of redesigning its hiring process across all its worldwide locations. We were aware of upcoming changes to the hiring process, had conversations with members of Alpha’s global HR team, and were excited to follow up on these changes as they unfolded. The story might read easy and clear now but, in the beginning, it was anything but that. We were just completely drawn in by the setting and decided to go with it.

“Beyond documenting bias or structural barriers, what can organizations do structurally to change gender disparities in hiring outcomes?”

And then, having that perspective and insight allowed us to connect a real-world organizational change with a broader theoretical question about how decision-making structures influence gender disparities in hiring. There is a large body of research showing that employers’ decisions can result in gender disparities at the point of hire for various reasons. What we know much less about, however, is which organizational designs or interventions might actually help reduce those disparities. In other words, beyond documenting bias or structural barriers, what can organizations do structurally to change gender disparities in hiring outcomes? We find that transferring the shortlisting from hiring managers to HR departments increases the number of women hired into the organization.

ASQ pieces are known for methodological rigour. Your study used a staggered difference-in-differences design across seven waves and triangulated quantitative data with surveys and interviews. What feedback did reviewers give regarding the method section? What aspects did you strengthen post-review to enhance robustness and causal identification?

We received extremely constructive feedback from both our Editor Chris Rider and three fantastic reviewers across several rounds. Their input really helped us sharpen both the design of our study and how we communicated the tests.

One important area of feedback focused on strengthening the identification strategy. Based on suggestions we received throughout the review rounds, we refined some of our existing analyses and added tests, for example, the event study and the stacked difference-in-differences specifications. These helped demonstrate more clearly how the change in hiring process unfolded over time.

Another major theme of feedback was about communication and structure. Our reviewers pushed us to be clearer about the logic of the research design—why particular specifications were necessary and what each test contributed—and the identifying assumptions of these tests. They also encouraged us to structure the manuscript more clearly as a post-hoc analysis paper.

“The ASQ review process was incredibly stimulating and developmental. We had read about it in this very blog series”

The ASQ review process was incredibly stimulating and developmental. We had read about it in this very blog series, but we were so grateful to our Editor Chris Rider for his guidance, patience, and just pure investment in the process! The process helped us not only improve the empirical strategy but also tell the methodological story in a way that readers can hopefully follow more easily to form their own judgement.

How did you identify the focal organization (Alpha)? What has helped you in identifying the focal organization? Do you have any recommendations for PhD students or early-career researchers on identifying the best research settings for their field studies and accessing them to conduct their research?

Finding the right research setting is often one of the hardest parts of conducting field-based research. In our case, there was certainly an element of luck involved. One of us became aware of the company’s initiative to redesign its hiring process and realized that this could provide a unique opportunity to study an organizational intervention as it was unfolding. At the same time, these opportunities rarely appear completely by chance. They often emerge from relationships (and trust) that we as researchers build over time. Activities such as executive teaching, inviting practitioners as guest lecturers, or engaging in outreach with companies can create connections that eventually develop into research collaborations.

“Activities such as executive teaching, inviting practitioners as guest lecturers, or engaging in outreach with companies can create connections that eventually develop into research collaborations.”

One lesson we took from this experience is that promising research settings often reveal themselves in small, easy-to-miss ways—a new initiative, a conversation, or a practitioner describing a challenge. These moments rarely come with full clarity. Even if a collaboration does not immediately turn into a full project, staying engaged with practitioners and understanding their challenges can still generate valuable insights—and sometimes, those early conversations grow into meaningful research.

Another lesson we took from this experience is that access to organizational interventions or settings is a necessary but often not a sufficient condition to generate new knowledge. It is equally important is to find (empirical) ways to better understand the dynamics underlying an intervention and contemplate about the reasons of the intervention effects. Organizational interventions often do not allow for multiple treatment arms, making it difficult to cleanly identify underlying mechanisms. One way to circumvent this challenge is to adopt a mixed-method approach: quantitative tests can be used to establish the average treatment effect of an intervention, while qualitative tests can be used to shed light on the mechanisms driving the effect. In our case, we were able to conduct semi-structured interviews and use survey findings to make sense of the underlying mechanisms as best we could. It helped that one of us had prior experience in qualitative research methods. Therefore, a simple but perhaps effective recommendation for PhD students and early career researchers could be that being open to different research methods (e.g., in the PhD education) and valuing methodological diversity within co-author teams can potentially pay off.

As organizations adopt AI-driven screening, what are the potential implications of your research for future studies on AI or other emerging technologies used in hiring processes, such as AI taking on responsibilities like short-listing?

Although our study does not examine AI directly, the findings do raise interesting questions about how AI might be used in hiring processes. We speculate about that in our Harvard Business Review article, but this is really mere speculation. One implication is that systems, whether human or algorithmic, that evaluate applicants using clear and pre-defined job position criteria may reduce the reliance on stereotypes when making early screening decisions. We know from prior research that focusing on objective indicators such as years of experience or educational background can create more standardized evaluations. AI might also have the potential to lower opportunity costs for investing time and effort in evaluating candidates when pre-screening many applicants.

“One implication is that systems, whether human or algorithmic, that evaluate applicants using clear and pre-defined job position criteria may reduce the reliance on stereotypes when making early screening decisions.”

However, simply transferring shortlisting responsibilities to AI will likely not eliminate bias. Many AI systems used in hiring have already been criticized for reproducing biases embedded in historical data. If the training data reflect past disparities, the algorithm may inadvertently reinforce those patterns. Another important issue is how organizations interact with AI tools. Without proper incentives and oversight, decision-makers may either rely too heavily on algorithmic recommendations or organizations may underinvest in training and monitoring these systems.

For these reasons, we see the most promising approach as treating AI as a decision-support tool rather than a replacement for human judgment. Well-trained HR professionals and hiring managers can use AI-generated insights while still applying contextual understanding and nuance that algorithms currently lack.

Interviewer Bios:
Irna Ishrat is a doctoral student in Management in the Department of Business Administration, Aligarh Muslim University, India. Her research critically explores the phenomenon of home-based work, focusing on the intersection of informality, gender, and social hierarchies in the Global South. Methodologically, she uses qualitative methods to produce theoretically grounded and socially relevant scholarship.

Alara Cansu Yaman is a doctoral candidate at the University of Goettingen, Germany. Her research broadly focuses on gender equity and equality in the workplace, and women’s career success. She draws on both quantitative and qualitative methods to conduct cross-cultural studies to examine the role of organizational, cultural, and institutional contexts in women’s career experiences.