Artificial Intelligence and Job Interviews: Is the AI Tech Ready?

IRI Podcast episode on AI Interviews

AI-driven technology has become indispensable for companies worldwide. According to Predictive Hire, 55% of hiring managers have invested in AI recruitment, and most Fortune 500 companies are using AI technology to filter resumes – but are we ready for AI Interviews (or, more importantly, are AI Interviews ready for us)? Today, we’re joined by Hilke Schellmann, an Emmy Award-Winning Journalist and a Journalism Professor at New York University. In 2021, she addressed this very question in a four-part investigative report into automated hiring practices for MIT Technology Review’s podcast In Machines We Trust. Here, she explains:

  • How AI job interviews are typically conducted;
  • How AI hiring could benefit employers and candidates alike;
  • What Ms. Schellmann's research discovered; and .
  • The ethical need to confirm the fairness and functionality of these tools.

If you prefer to read along while you listen, we've done all the hard work for you! We listened back to this episode and took notes below, and access is free! 


“I was interviewed by a robot.”

  • Ms. Schellmann first became interested in AI in 2017, when she asked a Lyft driver about his day, and he replied “I had a weird day. I was interviewed by a robot.” 
  • Months later, she’d attend a conference on Artificial Intelligence, where a former EEOC employee warned that the practice of AI job interviews needed to be looked into.
  • Ms. Schellmann was invited to take part in a podcast called In Machines We Trust as a result of an article she’d been writing for MIT Technology Review about the auditing of AI-based algorithms used in hiring.
    • In Machines We Trust is a four part series that analyzes automated hiring practices, looking at artificial intelligence tools and more.
    • In the hiring funnel, there are several different ways to incorporate artificial intelligence, such as building a candidate pool, through job platforms, applicant tracking, AI personality games, and AI interviews.
    • Artificial intelligence is also used in background checks, social media analysis, and more.
    • Artificial intelligence is now found throughout the hiring process.
  • Currently, AI interviews are being primarily used on recent college graduates and for those seeking hourly positions, and many people are not in favor of it. 
    • They feel it is impersonal, they cannot ask questions, and it is a one way process.


Get all the notes, links, tips, tricks and most important content from this episode - for free!

By signing up you agree to our terms

Artificial Intelligence Job Interviews

  • Traditionally, for job interviews candidates sit with an HR manager, chat, and answer questions, either in person or on Zoom. With AI job interviews, there is no manager to interact with. The candidate simply receives pre-recorded questions on their computer monitor or phone , and they record themselves answering those prompts. Those answers are then assessed by artificial intelligence.
    • The AI makes assessments based on any number of factors, including word choice, language fluency, voice intonation, even facial expressions, and those scores are then sent to hiring departments.
  • There are many positives with the AI job interview process, for both the employers and the job candidates:
    • In AI interviews, this structured interview process means every candidate is asked the same questions, eliminating some degree of potential bias from the process;
    • AI interviews save labor costs for the HR department because it is faster, and HR professionals can either watch the recordings or have artificial intelligence score the interviews.
    • From a scheduling standpoint, it’s more convenient for candidates because it allows them to record their responses at a time when it is most convenient for them.
    • Some candidates appreciate that AI interviews are conducted more as a presentation of themselves, rather than as a conversation, because it takes away a lot of the anxiety around talking to another person. 

Research into the Claims

  • Companies claim the AI interview process is not only efficient, but that it can find the strongest candidates, and now, researchers are trying to assess how that works and how true this claim is.
  • Ms. Schellmann and her team focused their research on companies that use AI hiring processes that have a certain market share. They looked at:
    • How the AI job interviews worked at these companies;
    • How the AI interview processes are marketed; and
    • Whether or not AI can, in fact, find the best candidates.
  • After nearly five years of research on this topics, Ms. Schellmann is currently in the process of writing a book on this subject, taking an even deeper dive into AI testing, and her book is expected to hit shelves in early 2024.
  • Ms. Schellmann took part in some of the testing herself, to see what other candidates go through when going through artificial intelligence.

Takeaways from AI Testing

  • Ms. Schellmann and her team discovered through their testing that the artificial intelligence systems they looked at had flaws.
    • One system was supposed to be able to detect the candidate’s proficiency as an English speaker. As a test, Ms. Schellmann, who is fluent in German, intentionally answered all the interview questions in German. The AI dubbed her fluent in English – even though she’s not spoken a work of English during the interview!
    • Another tool tested promised to assess the candidates’ personality, based on the Big 5 Personaility Traits, but delivered different results every time she took it, indicating that the tool was neither reliable or stable in its assessment.
  • These results are concerning because these systems are already being used, companies are displaying their use, with confidence.
    • These systems are being used on real people, for high stakes hiring decisions.

The Ethics of AI Interviews

  • There are many questions surrounding the use of artificial intelligence during the hiring process.
    • Experts say there should be an audit done on the tool to test its fairness and functionality.
    • Fairness
      • In some of the AI games, some people with disabilities were not able to complete the challenges because of their motor abilities, and some of the games had to relevancy to their competency to complete the job. This poses a fairness question for those people, especially because they do not know if they need to ask for an accommodation if they do not know what the questions may be.
      • Fairness also comes into play when thinking about if women and men play AI games in the same way, if people have an accent or a speech impairment, and that needs to be taken into account. The priority should be thinking about if artificial intelligence hiring practices will work for them.
    • Functionality
      • U.S. law dictates that these AI hiring tools don’t need to work, they just need to be fair for everyone, but while fairness is important, it  doesn’t necessarily yield results - on paper, hiring with a random number generator would be considered “fair:, but would not necessarily yield the best people for the job.
      • Companies, and job applicants, want the AI interview process to work, and have it be fair to candidates and their skills.Not getting hired because of flawed technology would be very frustrating for potential candidates.
  • The EOC recently came out with a statement about the use of artificial intelligence in hiring, and the rules currently in place still apply, but those rules are not necessarily picked up on the scope of video interviews and remote hiring practices.
    • Previously, it was easier to demonstrate a case of unfair hiring practices, but now, it is more difficult to prove if AI is being used in the hiring process, how candidates are classified, and if candidates are being wronged. This is more challenging for candidates to prove biases or unfairness in court.
    • The EOC statement provided technical advice to employers, saying they acknowledge AI hiring practices, and reminding them that hiring and employment laws still apply.
  • Companies need to make sure that the technology works before using it in their hiring practices.
    • While smaller companies probably don’t have data scientists, it is the companies responsibility to test the AI technology, rather than relying on a larger company or a third party vendor.

The AI Employer Experience

  • AI is a technology that feels like it would help employers, and it can be more efficient for them.
    • Technology can eliminate human biases about experience, education, etc. however, it is important to make sure the technology works.
    • Using AI can save companies time and money, and provide a very structured based process.
  • AI can be an effective interviewer through structure, but it is still unclear if it can accurately identify strong job candidates.
    • Researchers and companies need to continue testing the efficacy of AI, and if it is predictive of employer success. Most HR departments are not set up for this testing, which would take many years and is very cost intensive. 
    • Most companies do not have enough data to test their employees' success regularly, nevermind the efficacy of AI technology, and it is challenging to determine employee success depending on the career. 
  • Using AI technology in the hiring process can bring value to an organization if they are used in the right way.
    • The technology might not only be used in an AI job interview, but also looking at peoples resume, at what drives candidates, skills, and an overall holistic approach to thinking about how successful candidates could be on the job.
    • Companies turn to AI to be more cost effective, but some candidates really do not like virtual or AI job interviews, and that can be a turnoff for them working at the company.
  • Companies need to be more transparent and do more testing regarding their use of artificial intelligence. 
    • Companies need to look at what goes into these algorithms, the training data, and understand the results of the algorithm.
    • Vigorous testing and vigorous transparency are the best ways to see if an algorithm works before being really put into practice company wide. 
    • There are new laws, such as in NYC, about remote hiring practices. The GDPR is also looking into storing hiring data, access, and what companies can do with the results. 

Hilke Schellmann Background

  • M.S., Investigative Reporting, Broadcast, Multimedia from Columbia University - Graduate School of Journalism
  • Magister (Combined B.A./M.A.), Cultural Theory, American Studies and Political Science from Humboldt-Universität zu Berlin
  • Graduate School of Arts & Sciences, American Studies, Performance Studies from New York University
  • Ms. Schellmann began her career working on documentaries, including work as an Assistant Editor for the American Undercover Series for HBO, and serving as Coordinating producer for a National Geographic special about the Berlin Wall
  • She served as the Staff Reporter/Producer for Story House Productions
  • Her freelance work includes work as a Radio Producer for WNYC, as a Videojournalist for The New York Times, as a Producer for German Public Television, ARD and ZDF, and as a Showrunner/Executive Producer and Reporter for The Wall Street Journal
  • She worked as a Multimedia Reporter/Producer for The Wall Street Journal
  • She was a Producer at Vice Media, LLC
  • She served at the Director of Video Journalism/Adjunct Professor for Columbia University - Graduate School of Journalism
  • Ms. Schellmann has won numerous awards and recognition for her work as Reporter/Producer with H2H Films
  • Additionally, she currently serves as Assistant Professor of Journalism with New York University



Subscribe & Review The ProjectHR Podcast!

Thanks for tuning into this week’s episode of ProjectHR. If the information in our weekly conversations and interviews have helped you in your business journey, please head over to wherever you get your podcasts and subscribe to the show. We'd also love it if you left us a five-star review! Your reviews and feedback will not only help us continue to deliver great, helpful content, but it will also help us reach even more amazing professionals just like you!

Print Friendly, PDF & Email