Principal AI/ML Engineer

About Evidently

At Evidently, we are raising the quality of care for every patient by empowering clinicians with the right information at the speed of thought. Our Clinical Data Intelligence platform extracts meaning from an ocean of clinical notes, outside records, and scanned documents — and delivers it to clinicians in the moment it matters. We're a small, mission-driven team building something genuinely hard and genuinely impactful. Learn more at evidently.com.

Who We're Looking For

We're looking for a senior technical leader with deep AI/ML research and engineering foundations who can help set the technical direction for how we build intelligent systems. This isn't a pure IC role narrowly scoped to implementation. We want someone who can identify what we don't know, design the experiments to find out, and drive the full stack of AI work from research to production.

The ideal candidate has genuine research depth (likely a PhD or equivalent industry research background) and has spent significant time doing the hard work of getting AI systems to actually work reliably in production on messy, real-world data. You've built evals from scratch. You've grappled with LLM failure modes. You know the difference between a benchmark that flatters a model and a benchmark that tells you whether to trust it. You've formed strong views on what makes AI systems trustworthy, measurable, and safe to deploy in high-stakes settings.

We are an AI-first company, not just in what we build, but in how we work. We use AI tools aggressively across research, engineering, and operations to move faster and think more clearly. We expect everyone on the team to work this way, and we're looking for someone who models what that looks like at a senior level: using AI as a force multiplier for their own judgment, not as a substitute for it.

Healthcare is one of the most data-rich and most consequential domains for AI. We need someone who takes both of those facts seriously.

What You'll Do

Technical Direction & Research

• Drive the AI/ML technical strategy for Evidently's core clinical intelligence systems from information extraction and summarization to reasoning and retrieval, and drive that strategy end-to-end through execution.
• Identify gaps in our current systems: where performance is brittle, where we lack coverage, where our evaluation methodology isn't catching real-world failure modes.
• Stay at the frontier of relevant research (LLMs, clinical NLP, retrieval-augmented generation, RLHF/RLAIF, agents, structured prediction) and translate what's useful into production-ready systems.
• Define our model evaluation and benchmarking approach — building evals that measure what actually matters for patient care, not just what's easy to measure.

Systems & Engineering

• Build and improve production AI pipelines: clinical entity extraction, summarization, chart abstraction, and structured inference over unstructured clinical text.
• Design and improve our approach to RAG, prompt engineering, fine-tuning, and model selection with rigorous evaluation at each step.
• Drive secure deployment of AI systems in compliance with HIPAA and SOC 2, with particular attention to accuracy, auditability, and hallucination risk.

Leadership & Collaboration

• Be a technical anchor for AI decisions across the team and the person others turn to when the hard questions arise about what approach to try, why something isn't working, or whether a model is ready to ship.
• Work directly with clinical subject matter experts to understand how our outputs are used, where they fail in practice, and what "good" actually means in a clinical workflow.
• Mentor other engineers; raise the team's collective fluency in AI/ML best practices, including how to use AI tools effectively in day-to-day technical work.

What You'll Bring

Non-negotiables

• Deep expertise in modern AI/ML, LLMs, NLP, information extraction, or closely adjacent areas. This means hands-on experience at the frontier, not familiarity from a distance.
• PhD in ML, NLP, computer science, or a related field, OR equivalent depth demonstrated through research publications, significant open-source contributions, or impactful industry AI work.
• Experience building and owning evaluation frameworks; not just running evals, but designing the methodology: what to measure, how to avoid Goodhart's Law, how to detect regressions.
• Track record of taking AI systems from research/prototype to reliable production deployment, with an honest understanding of where things break and why.
• 8+ years of software engineering experience, with the bulk of that focused on AI/ML systems.

Technical Depth

• Fluency in Python, PyTorch or JAX, and the core ML stack (training, fine-tuning, inference, quantization).
• Hands-on experience with large language models: prompting strategies, RAG architectures, RLHF/preference tuning, instruction tuning, or model distillation.
• Strong command of data: SQL, data pipeline design, annotation tooling, dataset curation. You understand that AI quality is mostly a data quality problem.
• Experience with NLP libraries (Hugging Face, spaCy, etc.) and production serving infrastructure (vLLM, TGI, or similar).
• Proficiency with AI-assisted development tools such as Claude Code, Cursor, Copilot, or similar, as a skilled operator who directs these tools with precision rather than a passive consumer who accepts their output uncritically. You know when to push them harder, when to override them, and how to structure work so they're genuinely useful rather than subtly wrong.

Mindset

• You are genuinely AI-first in your approach because you've done the work to understand where AI methods create real advantages and where they don't. This extends to how you write code, conduct research, and think through problems day-to-day.
• You have the intellectual honesty to say when a model isn't good enough to ship, and the pragmatism to ship imperfect things when the risk is understood and bounded.
• You're comfortable with ambiguity at the research frontier and with the rigor required to convert that ambiguity into production systems.
• You care about the domain. Clinical AI is high-stakes and you take that seriously.

Bonus

• Familiarity with clinical NLP, healthcare data standards (HL7, FHIR, ICD/CPT coding), or prior work in healthcare AI.
• Experience with structured prediction, medical knowledge graph construction, or clinical information extraction.
• Publications or applied research contributions in NLP, LLMs, clinical AI, or evaluation methodology.

Tech Stack

Python · PyTorch · Hugging Face · PostgreSQL · BigQuery · dbt · FastAPI · Google Cloud Platform · LLM APIs (OpenAI, Anthropic, etc.) · Jupyter / Streamlit

Who You'll Work With

You'll join a small, senior team of AI leaders who have spent careers at the frontier of the field with research backgrounds, publications, and industry experience that are genuinely top-tier. This is a team that has done the hard work, not just talked about it, and that rigor shows in how we approach problems. Across AI, backend, and frontend, your work will have direct, visible impact. You'll collaborate closely with clinicians and hospital partners to understand real-world performance, not just benchmark performance. The team is concentrated in the Bay Area with a weekly San Francisco office day; remote candidates are welcome, though you must be able to work core Pacific hours and must be based in the US.

Compensation: $250k – $350k + equity