How to Run an "AI Open Laptop Interview"

The job market is rapidly evolving in the age of AI. As candidates increasingly turn to AI tools to help them through the hiring process, employers are working to adapt—reimagining how they evaluate skills, potential, and authenticity.

Many people are trying to figure out how to prevent applicants from using AI to ‘cheat’ their way through the hiring process. Hiring managers want to know if applicants know how to do the job on their own. 

This is a problem. It’s divorced from today’s reality. 

Once a person is hired for that role, aren’t they going to use AI? Everyone seems to have the mandate to ‘use AI’ coming down from the C-suite.

How do we fix the hiring process to adjust to the AI reality?

Rather than treating AI as a threat, forward-thinking organizations are exploring ways to ensure that their hiring practices remain both fair and future-ready. I have one way you can do that: The “AI Open Laptop Interview.” Instead of trying to block applicants from using AI, ask them to show you how they would use it. 

Yes, you need to change your interview style. 

Yes, it’s going to involve watching them awkwardly type in real time. 

Yes, you’re going to need new questions. (I got you covered there.)

No, it’s not impossible. Let’s get into it.

Set the Stage

Let applicants know they’ll be asked to share their screen and turn their camera on. Tell them they’ll be given a challenge, and they’ll be expected to use AI tools to work on it in real time. This is the equivalent of a live coding interview for engineers. 

Many AI tools have a free tier or limited use without paying. Don’t judge them on whether or not they can afford the top-of-the-line version. Your company is going to give access to whomever is hired for the role, right?

(Right?)  

Assess for AI Proficiency in 4 Stages

There are 4 stages to this interview:

1. Tool Selection and Building an Initial Approach
2. Refinement and Evolution 
3. Application and Testing
4. Integration and Scaling

Start with a legitimate problem set or challenge which they’ll face in the role. Then, head into stage one of the interview.

Stage 1: Tool Selection and Initial Approach 

When you’ve posed the hypothetical problem or challenge, ask the applicant to explain their thought process behind how they’ll approach the problem. Some starter questions may be:

• “Given this problem, tell me which tools you’re going to use and why you think they’ll be the most effective.” 
• “Tell me what you’re thinking.” 
• “Walk me through your process for crafting the initial prompt for this task.”
• “What inputs/data/context do you need to get started?”
• “Why did you choose that model?” 

Be prepared to give them more information about things like your ideal customer profile, sales qualification process, product value proposition, unique selling points, tech stack, or custom data requirements. AI needs context to give better responses, and your applicants need context, too. 

Stage 2: Refinement and Evolution

Once you’ve gotten a better understanding of the applicant’s approach to selecting tools and scoping the problem, start asking questions about how they’ll refine and evolve their approach. Some starter questions may be: 

• “How would you refine your method to improve the output?”
• "How can you integrate the output from one AI tool into another to achieve a more comprehensive solution?" 
• If you had more time than we do during this interview, what would you do next?

Watch to see if they leverage multiple tools. How adept are they at prompt engineering? Do they evaluate the responses from AI and refine their prompts? Do they use advanced prompting techniques like few-shot learning or chain-of-thought? 

Stage 3: Application and Testing 

After getting a better understanding of their more refined approach to using AI, you can move on to the long-term vision. Ask questions about how they would apply what they’ve shown and described, and how they would test efficacy. Some starter questions may be:

• “Where should you consider including a human-in-the-loop check for safety and quality controls?” 
• “How would you implement that?”
• “How do you typically evaluate AI outputs for quality and accuracy?
• "How would you ensure the data privacy and security of this AI application within our current infrastructure?"

Do they understand where AI can make mistakes, and how to catch the problem before it's too late? Look for their thinking around recovering from AI failures or outages. Do they think about reinforcement mechanisms to continually improve outcomes?

Stage 4: Integration and Scaling

Here’s the golden goose for use of AI in business. Will it integrate? Will it scale? Some starter questions to assess this may be:

• “How would you make this work with our existing tech stack?”
• “How would you roll out a solution like this to a team?“
• “What metrics would you track to monitor the performance of this solution once it’s deployed? 
• “If this was successful, what would you need to scale it by 10X?”

Ask them how they think about scale and automation. Do they understand how to go beyond copy/paste from AI to integration with existing workstreams or applications? If applicable to the role, how do they think about the change management of rolling out a solution like this?

Evaluation Criteria

I recommend scoring candidates (0-10) across four key dimensions:

1. Tool Proficiency: Familiarity with relevant AI tools and understanding of their appropriate applications
2. Prompt Engineering: Ability to craft effective prompts and iteratively improve them
3. Creative Application: Novel approaches to solving the problem beyond obvious solutions
4. Implementation Vision: Realistic pathway from prototype to production

The strongest candidates demonstrate not just technical skills, but also practical thinking about how AI can meaningfully enhance business processes. 

This is a new field. Hiring managers can’t just look for people who have done this before; they need to find people that know how to learn, think, and adapt in new areas. On the other end of the spectrum, hiring managers can’t expect employees to not use AI at all. It’s unrealistic in many fields, particularly for employees who are working for leadership teams that expect AI usage to be integrated into their work processes.

The key to success is coming to expect and understand that many employees will have integrated healthy uses of AI into their workflows. And your job, as a hiring manager, is to assess their ability to harness the power of AI in responsible and effective ways at your organization.