Patient-Trial Matching Challenge @ IEEE ICHI 2027

About the challenge

IEEE ICHI 2027 will host the Patient-Trial Matching Challenge, a shared task on automated clinical-trial eligibility screening from electronic health records (EHRs). Systems read the original inclusion and exclusion criteria of a trial, decide for each patient whether each criterion is met, and cite the records that support the decision. The data are fully synthetic, so no real patient information is involved.

Existing benchmarks test parts of this problem: notes only with yes/no labels (n2c2 2018), short case summaries judged at the trial level (TREC Clinical Trials 2021-2023), or criterion-level reasoning on brief single-note summaries (TrialGPT). This challenge evaluates longitudinal structured data and clinical notes together, separates missing evidence from contradictory evidence, and requires every decision to cite its evidence.

Fast facts

• What: criterion-by-criterion eligibility screening, in three tracks: Structured Data, Clinical Notes, and Multimodal EHR (primary).
• How it works: teams submit a runnable system; the organizers run it on held-out patients and trials with a designated large language model, and score the quality of its labels and evidence, its cost, and its runtime.
• Who can join: students, researchers, and industry practitioners worldwide. Participation is free.

Important dates

Aligned with the IEEE ICHI 2027 main paper track. Dates are tentative and may be adjusted; registered teams are notified of any changes.

Tasks

Participants build fully automated systems that screen patients against clinical trial eligibility criteria. For each trial, a system receives the inclusion and exclusion criteria and a cohort of patient records, and for every patient × trial × criterion assigns one of four labels and cites the supporting records.

Input

A trial's original eligibility criteria (unchanged from the protocol), and for each patient a synthetic EHR as of a fixed screening date. The three tracks differ only in the data available to the system:

• Track 1 · Structured Data: OMOP CDM v5.4 tables only.
• Track 2 · Clinical Notes: clinical notes and patient demographics only.
• Track 3 · Multimodal EHR (primary track): OMOP tables and clinical notes together.

Output

A JSON Lines file with one line for every patient × trial × criterion: patient_id, nct_id, criterion_id, one label, and an evidence list. Each evidence item is an OMOP row (table name and row_id) or a clinical note (note_id). An empty list means no supporting evidence was found. Systems may cite only records visible in their track. A sample file is included in the starter kit.

Labels

A label describes whether the criterion text is satisfied, not whether the patient is eligible:

Timing

All records are as of one fixed screening date. A note counts only if it was written on or before that date, and a mention in a note is dated by the event it describes, not by the date of the note. A later finding that supersedes an earlier one is not a conflict.

Screening status (derived by the organizers)

The organizers derive screening status from the submitted labels, for each patient and trial. Participants do not submit statuses or rankings.

• Ineligible: any inclusion criterion is Unmet, or any exclusion criterion is Met.
• Eligible: every inclusion criterion is Met and every exclusion criterion is Unmet. (Eligible means eligible for further screening, not for enrollment.)
• Uncertain: otherwise.

The organizers also rank the patients for each trial from the labels. Teams may enter one, two, or all three tracks.

Data

Synthetic and privacy-free: longitudinal patient records generated for this challenge, in the OMOP CDM v5.4 with standard vocabularies (ICD-10-CM, SNOMED CT, RxNorm, LOINC), plus synthetic clinical notes across several encounter types. No real patient records are used.

• Development set: released November 20 to registered teams, with the trial criteria, patient records, reference labels, and evidence references for every criterion.
• Held-out set: different trials, from several clinical areas, and different patients. The trial texts, patient records, and answer keys stay private and are used only in the official runs.

Data scale and how the records and labels are generated and checked will be described on the Challenge Details page.

Evaluation

Each track has its own reference labels, determined solely by the information available in that track, and a separate leaderboard. Quality combines four components (provisional weights):

Q = 0.40 × criterion F1
  + 0.20 × screening F1
  + 0.20 × evidence F1
  + 0.20 × ranking nDCG

Scoring rules

• Only the first line for each patient × trial × criterion counts.
• A missing or malformed line gets no credit and counts as Not Assessable for screening and ranking.
• Citing a record that does not exist or is not visible in the track counts as a false positive.

Qualification

A run is ranked if it passes the build and model-use checks and reaches a quality score Q of at least 0.6 (provisional).

Ranking

Qualified runs are compared on quality, model cost, and runtime together. Runs that no other run beats on all three form the first tier; the next tier is formed from the remaining runs, and so on. Within a tier, higher quality ranks higher.

Leaderboard

Each track shows Q, its four components, cost, runtime, and 95% bootstrap intervals; the intervals do not change the official ranking. The organizers' baseline is shown but does not compete.

Final weights, limits, and the scoring code are released with the development data.

Participation & registration

• Register (free): by December 18, 2026, through the registration form on the Challenge Details page (team name, member names, affiliations, and a primary contact).
• Teams: up to five members; each person may join only one team.
• Starter kit: a Docker template, a baseline system, sample submissions, and the scoring script, released with the development data.
• Model access: during development, teams use the designated models through their own accounts at their own expense; the organizers provide model access for the official runs.

Submission, publication & recognition

System submission (by January 8)

Source code, a Dockerfile with pinned dependencies, and a two-page system description, one system per track. Source code stays private to the organizers; public release is encouraged but optional.

Official runs

The organizers check each submission for prohibited models, build it from source, and run it once per track on the held-out data, in the same CPU-only environment with a fixed API concurrency. Dependencies may be downloaded at build time; at run time the only network access is to the organizers' model proxy. Reruns happen only after an organizer-side failure, and no fixes are accepted after the deadline.

Challenge paper (optional, by February 1)

Top-performing and methodologically strong teams will be invited to submit a short paper describing their system and results through the ICHI 2027 main paper track. Standard peer review applies; an invitation does not guarantee acceptance. Accepted papers appear in the IEEE ICHI 2027 proceedings on IEEE Xplore.

Publication & recognition

Accepted papers appear in the IEEE ICHI 2027 proceedings on IEEE Xplore; at least one author must register for and present at the conference. Results and awards are presented at a challenge session during the conference.

Rules of conduct Preview

• Eligibility and teams: open internationally; team membership is declared at registration and fixed after December 18.
• Models: systems must use the designated large language model, and optionally the designated embedding model, announced with the development data. No other generative or embedding models are allowed in the official runs; declared rule-based methods, vocabulary lookups, and non-LLM NLP tools may be used. Every system must make at least one successful call to the designated language model.
• Data use: use the provided data and declared public knowledge resources such as vocabularies and guidelines. External patient datasets may not be used for training, fine-tuning, or evaluation. Attempts to recover hidden answers are prohibited.
• Organizer teams: teams from the organizers' groups may take part but are not eligible for awards.
• Integrity: submissions must be the team's own work; violations lead to disqualification.

Complete rules and technical requirements will be posted on the Challenge Details page.

Awards

Certificates and recognition for the top teams; additional distinctions may be announced by the ICHI 2027 organizers.

Prizes are awarded in a track only if at least three teams submit valid runs.

Frequently asked questions

Organizers & contact

• Data Challenge Chair
  Xing He, PhD, Indiana University
  hexing@iu.edu
• Challenge Lead
  Hongyu Chen, University of Florida
  hongyuchen1@ufl.edu

Challenge inquiries and registration: hexing@iu.edu

Sponsorship and industry contact: to be announced