aamc.org does not support this web browser.

Use Case 3: LLM-Assisted Socioeconomic Context Analysis

Challenge

Recent Supreme Court rulings have restricted certain considerations in admissions, disrupting traditional holistic review practices that often relied on explicit demographic factors. In this evolving legal landscape, programs still need a robust way to evaluate each applicant’s varied life experiences (accomplishments and hardships), while staying compliant with new requirements. However, mining multiple essays and supplemental materials for relevant socioeconomic details is time-consuming and prone to human oversight — especially at scale.

Solution

Implement a large language model- (LLM-) based tool that reviews written materials and synthesizes socioeconomic context (e.g., financial background, geographic barriers, educational hurdles) in an evidence-focused manner. By surfacing the applicant’s unique journey, this approach allows staff to continue a form of holistic review without relying on factors that may be legally constrained, ensuring the institution can still recognize applicant history and future potential for resilience, resourcefulness, and community impact.

How it Works

  • Build prediction model using historical data (USMLE scores, publications, research experience).
  • Generate an "Interview Likelihood Score" (0-100) based on past successful candidates.
  • Identify key statistical factors influencing predictions.
  • Process personal statements, activity descriptions, and letters.
  • Programs first define priority areas for summarization.
  • Create targeted summary highlighting, for example:
    • Program value alignment (e.g., community service, research excellence).
    • Socioeconomic context (e.g., education background, financial circumstances).
    • Key experiences and achievements.
    • Notable characteristics or qualities.
    • Unique background elements.
  • Include relevant quotes as evidence.
  • Interview Likelihood Score: 85/100.
  • Statistical factors.
    • USMLE Step 1: 245 (top 15% of past interviewees).
    • USMLE Step 1 attempts: 2.
    • Research: 2 first-author publications.
    • Clinical experience: 1,000-plus hours.
  • Program value alignment.
    • Community focus: “Created mobile health clinic for underserved areas.”
      • Found in: Activities section, entry #3.
      • Reasoning: Demonstrates initiative in addressing health care access.
    • Research excellence: “Led quality improvement study on ED wait times.”
      • Found in: CV research section and personal statement paragraph 2.
      • Reasoning: Shows both leadership and research methodology skills.
    • Educational innovation: “Peer tutoring program for premed students.”
      • Found in: Activities section, entry #7.
      • Reasoning: Indicates commitment to medical education.
  • Context and Background.
    • First-generation college student.
      • Found in: Secondary application essay #2.
      • Reasoning: Explicitly stated in response about challenges.
    • Worked 20-plus hours per week during undergrad.
      • Found in: CV employment history and referenced in personal statement.
      • Reasoning: Indicates financial need and time management skills.
    • Rural health care experience in medical desert region.
      • Found in: Personal statement opening paragraph and activities #4.
      • Reasoning: Shows exposure to underserved health care settings.
  • Key Experiences.
    • ED quality improvement project leader
      • Found in: Research experience section and LOR from ED director.
      • Reasoning: Major leadership role with measurable impact.
    • 3 years EMT experience.
      • Found in: CV clinical experience section.
      • Reasoning: Sustained clinical commitment in premedical school.
    • Health care disparities research focus.
      • Found in: CV research section and personal statement theme.
      • Reasoning: Consistent thread across multiple experiences.
  • Contextual review of activities. Assesses activities considering context, such as:
    • Work experiences.
    • Clinical exposure opportunities.
    • Research access and opportunities.
    • Leadership development.
    • Community engagement.
  • Define key action points. Highlights high-impact areas to focus on, such as “overcome key obstacles” or “unique community contributions,” rather than a lengthy narrative.
  • Dashboard-style summaries. Showing insights like “high-impact experiences” and “noteworthy adversities,” each with linked evidence for easy verification.
  • Time savings metrics. Quantifies time saving by comparing AI-driven analysis to traditional manual review, making it ideal for high-volume programs.
  • Built-in evidence retrieval. Ensures an evidence-based approach, with direct quotes from application materials, making verification quicker and easier.
  • Context adaptability. Adapts to program-specific priorities, such as an increased emphasis on rural health care initiatives or financial hardship.
  • Reviewer-guided learning. Can refine what information is highlighted based on reviewer feedback, ensuring alignment with evolving holistic review goals.
  • Provides organized summaries. Delivers organized profiles that outline:
    • Key background factors.
    • Challenges overcome.
    • Significant experiences with context.
    • Supporting evidence from multiple documents.
  • Background context analysis for Frankie Chen-Jones. Instead of manually piecing together context from various documents, a reviewer receives an organized, evidence-supported analysis.
    • Educational journey
      • Finding: First-generation student, community college transfer.
      • Location: Secondary essays and AMCAS® application.
      • Reasoning: Demonstrates nontraditional path, educational barriers.
    • Financial background
      • Finding: Worked 20-plus hours per week, self-funded MCAT® prep.
      • Location: Work/Activities section and personal statement.
      • Reasoning: Shows financial constraints, time management skills.
    • Geographic context
      • Finding: Rural area, 60-mile drive to nearest hospital.
      • Location: Biographical info and experiences essay.
      • Reasoning: Indicates health care access barriers.
    • High school profile
      • 67% free/reduced lunch eligible.
      • 3 AP courses (bottom 10th percentile).
      • 5% medical school matriculation rate.

Key Takeaways

Core Benefits

  • Contextual understanding. Comprehensive analysis of applicant circumstances.
  • Legal adaptability. Approach mindful of evolving admission requirements.
  • Systematic review. Consistent evaluation of background factors.
  • Evidence-based. Clear documentation of context indicators.

Resource Requirements

  • Technical: Robust LLM infrastructure, data integration tools.
  • Personnel: Subject matter experts for context definition, technical team.
  • Effort: High initial investment in context assessment framework.

Challenges, Solutions, and Information Triangulation

Building on Table 1's challenges around example libraries and expert consensus, Table 3 provides a non-exhaustive list of challenges building with LLMs. Given LLMs' unpredictable nature, maintaining consistent standards demands rigorous oversight. Teams often fall into common traps: implementing generative AI unnecessarily, choosing overly complex solutions initially, and placing too much faith in early demos. The key to success lies in systematic human evaluation — the same careful approach needed for building reliable example libraries and achieving expert consensus.

Table 3. LLM -Assisted Context Analysis: Challenges, Solutions, and Information Triangulation.
Topic Challenge Solution Information Triangulation
Infrastructure: Cost, Staffing, and Deployment Self-hosted AI promotes privacy but requires higher investment, while API-based models are easier but may risk data exposure. Consult IT partners to consider using cloud AI strategically, optimize AI costs, and consider a hybrid approach that balances security with usability. N/A
Evaluating LLM Outputs AI may generate plausible but incorrect details. Poorly designed prompts can reduce accuracy. Design annotation platform to systematically evaluate AI outputs, ensuring alignment with detailed rubrics. Compare AI-generated responses with human feedback across documents
Transparency and Interpretability Admissions officers need clear, explainable insights. Structure AI-generated insights in readable formats and provide clear summaries. Compare AI-generated responses with human feedback across documents
Customization vs. Flexibility AI fine-tuning can improve accuracy but requires technical expertise and resources. Design annotation platform to systematically evaluate AI outputs, ensuring alignment with detailed rubrics. Compare AI-generated responses with human feedback across documents
Bias and Fairness AI models may reflect biases in training data, potentially disadvantaging certain applicants. Consider open-source bias evaluation tools such as LangFair to assess and mitigate fairness risks. Examine AI-generated insights across applicant documents for different groups
Scalability and Efficiency AI must process large application volumes quickly. Consult IT partners to optimize AI processes. Cross-check scalability and efficiency across applicant documents
Consistency and Model Updates AI model updates can alter outputs, leading to inconsistencies in applicant evaluation within cohorts. Consult IT partners to design infrastructure to constrain model version to promote consistent decision-making. Use same AI model for evaluating all document types

Note. Table 3 challenges build upon those in Table 1, particularly for developing example libraries and achieving expert consensus. That is, evaluating LLM outputs require the same rigorous evaluation frameworks established for competency-based review.

Best suited for

  • Programs prioritizing holistic review.
  • Institutions seeking legally conscious evaluation methods.
  • Teams with resources for comprehensive implementation.
  • Programs handling high application volumes.

Bottom Line

LLM-assisted socioeconomic context analysis provides systematic background evaluation with consideration of evolving legal considerations. This method surfaces relevant background factors while remaining adaptable to changing requirements. Implementation requires significant expertise for setup and ongoing oversight. This approach works best for programs prioritizing holistic review with resources for technology infrastructure.