---
title: Ashby AI Application Review: Criteria and Human Checks | Remqa
url: https://remqa.com/blog/ashby-ai-assisted-application-review
updated: 2026-11-16
---

[Home](https://remqa.com/) / [Blog](https://remqa.com/blog) / Ashby AI Application Review: Criteria and Human Checks

Recruiting AI

# Ashby AI-assisted application review: criteria, opt-outs and human checks

Set up Ashby AI-assisted application review with clear criteria, correct evaluation columns, opt-out handling and recruiter checks before decisions.

Remqa · November 16, 2026 · 5 min read

**Short answer:** Ashby AI-assisted application review evaluates candidates against job-specific criteria and returns meets, does not meet or undecided results. Configure clear criteria, validate them, review the reasoning and handle skipped evaluations separately before taking recruiting actions.

- Keep each criterion singular and supported by candidate evidence
- AI criteria results and human Average Score are separate measures
- Opt-out and unseen-notice cases need a separate review path
- Re-evaluate active job considerations after changing criteria

By the Remqa engineering team · Last updated November 16, 2026

A recruiter opens an Ashby review queue and sees a candidate with no human score, several AI criteria results and a skipped evaluation on another application. Treating all three as the same signal creates bad decisions. The setup needs a clear meaning for every result and an owner for cases the AI has not evaluated. This guide covers the native Ashby controls and a recommended operating process around them.

## Start with the native feature, not another screening system

Ashby has an AI-assisted application review feature. An Org Admin enables AI Features under Admin, Organization Setup, Opt-In Features and accepts the relevant terms. AI-Assisted Application Review is then enabled from the feature tiles. Job criteria are configured under the selected job's AI Features tab. Editing those criteria requires the appropriate administrator access for that role.

Use the built-in workflow as the starting point. Adding an external agent should solve a defined gap, such as preparing a review packet across approved systems. It should not create an untracked second screening result. Record who owns the criteria and who can approve changes before the queue becomes part of daily recruiting work.

## Write criteria that a recruiter can verify

Ashby supports suggested resume criteria based on the job posting, criteria reused from accessible jobs and custom criteria. Custom criteria have a short title and a prompt. The short title appears in settings, application review and filters. A vague title or prompt can therefore spread the same ambiguity through the entire queue.

Keep each requirement singular. Separate Python experience from AWS experience rather than asking one criterion to evaluate both. State what evidence would satisfy the requirement and what information would be insufficient. This is a recommended drafting method, not a guarantee that a model will interpret every resume correctly.

| Weak requirement | Better review question | Human check |
| --- | --- | --- |
| Strong technical background | Does the supplied history describe building Python services? | Check the cited role and actual responsibility |
| Good communicator | Does the history describe writing technical documentation for customers? | Do not infer communication quality from a title |
| Python and cloud experience | Use separate criteria for Python and the required cloud work | Check whether each requirement is necessary for the job |
| Great culture fit | Replace vague impressions with job-related evidence | Have the hiring owner approve the requirement |

These are examples for designing a review, not universal hiring requirements. Agree the actual job duties with the hiring manager. Do not add proxies for personal traits merely because they are easy to extract from a profile.

## Separate answered questions from evaluated qualifications

The application form criteria section can check whether an applicant answered a long, unformatted text question. Blank or unrelated answers are marked as not meeting that criterion. That control does not establish that the answer is correct, persuasive or sufficient for the role. A completed field and a demonstrated qualification are different things.

Ashby can also consider education and work history on the candidate profile when there is no resume. During testing, include those cases. Reviewers should know which underlying information was available instead of assuming every evaluation used an attached document.

## Validate, then test with recruiter judgments

Ashby validates criteria for clarity and objectivity and scans for implicit or explicit Equal Employment Opportunity issues. Flagged criteria show warnings and details. The product allows proceeding with warnings, but its guidance recommends updating or removing the flagged criteria. Do not treat the ability to continue as an approval of the requirement.

After validation, Save and Evaluate all active candidates starts the initial evaluation. If criteria change later, use Save and re-evaluate all active Job Considerations. As an operating control, keep a private record of the approved criteria, the change reason and the date reviewers began using refreshed results.

> **Test disagreement before processing the queue.** Have a recruiter independently review varied examples, including missing information and indirect experience. Compare the reasoning, not just the final label. Revise unclear criteria before expanding use.

## Read the correct results column

Ashby returns a result for each criterion: meets, does not meet or undecided. Reviewers can see full evaluations in the candidate profile's Feedback tab and expand the reasoning. The Application Review stage also shows evaluation summaries. Filters can select criteria and statuses, and bulk actions can be performed on the resulting list.

Average Score is different. It reflects ratings given by human reviewers during application review and is calculated independently from AI evaluations. An empty Average Score can mean no human reviewer has scored the application. It does not mean the candidate failed the AI review.

The AI Job Criteria Met Percentage column reports the proportion of criteria met. Use it carefully. As a recommended review policy, do not make that number a universal rejection threshold. Two criteria may carry very different job relevance, and an undecided result can point to missing evidence rather than missing ability.

## Build a separate queue for skipped evaluations

Ashby's Automated Processing Legal Notice is configured under Admin, Data Management, Data Privacy and Compliance, Privacy & Legal. The notice can apply to all jobs or a defined scope. With Hide Evaluations if a Candidate Hasn't Seen this Legal Notice enabled, evaluations are hidden for candidates who have not seen the required notice. Candidates who opt out have their evaluation skipped.

A direct application through the Ashby-hosted form is deemed to have seen the notice. Clicking Learn More is optional. Other entry routes, including manual additions and referrals, can produce unseen-notice cases. Review those separately rather than assuming every record passed through the hosted form.

The pipeline can filter AI opt-out status with advanced filters. Ashby does not send a proactive opt-out alert by default, though reporting can be configured for alerts. Assign a recruiter to the skipped queue. Any separate integration should respect the review restriction rather than sending the same candidate to another model. That integration rule is a recommended design control, not a legal conclusion.

## Check actions before the queue becomes operational

1. Confirm the job, criteria and available candidate information match the intended review.
2. Inspect the reasoning behind negative and undecided results.
3. Keep skipped evaluations distinct from unmet criteria.
4. Have a recruiter approve candidate actions and any associated messages.
5. Record confirmed actions so a retry does not repeat a rejection or email.

Common mistakes include sorting on the human score while calling it an AI ranking, changing criteria without refreshing active evaluations and treating a filtered list as a hiring decision. Keep action ownership explicit. Our guide to [resume screening with human review](https://remqa.com/blog/ai-resume-screening-human-review) covers the broader decision boundary.

## How Remqa helps with Ashby review workflows

Remqa builds [AI agents for recruiting teams](https://remqa.com/ai-recruiting-agents) around their existing systems. A useful starting scope is one role, an approved criteria set, a manual-review path and a recorded action boundary. If the workflow spans other systems, we can help design permission-aware handoffs without presenting custom controls as native Ashby behavior. Walk an engineer through your review queue and the cases your team cannot confidently resolve.

## Questions people ask.

**Q: Who can enable Ashby AI-assisted application review?**
An Org Admin enables the feature. Editing job criteria requires Organization Admin access or Elevated Access - Admin or Elevated Access - Admin Private for the specific role.

**Q: Does Ashby Average Score include AI criteria results?**
No. Average Score reflects ratings from human application reviewers. AI Job Criteria Met Percentage is a separate column for the proportion of AI criteria met.

**Q: What does the application form criterion check?**
For long, unformatted text fields, the option checks whether the applicant answered the question. Blank or unrelated answers do not meet that criterion. This is not a general assessment of answer quality.

**Q: What happens when a candidate opts out of AI processing?**
Ashby skips the AI evaluation and shows the reason in the product. Teams can filter on AI opt-out status. Do not treat a skipped evaluation as a failed job criterion.

**Q: Can changed criteria be applied to existing candidates?**
Yes. After editing prompts or criteria, use Save and re-evaluate all active Job Considerations. Review the refreshed results before using them to guide actions.

Book a free 30-minute call with an engineer: https://calendly.com/meghanaj-connect/30min
