AI Red Team Services for Healthcare & HIPAA: What to Buy

You're about to ship a healthcare app, a fine-tuned model, or a clinical agent, and you need to decide whether to buy an AI red team service, tooling, or both. This guide covers what to ask before you sign, how the work differs from a penetration test, and when HIPAA actually applies.

Attack techniques and test design are in the AI red teaming guide for healthcare.

Service or tooling

Buy a tool for a check your own engineers can rerun. Buy a service for a scoped engagement, a person who follows a finding from the model into the systems around it, and a report you can set next to a risk analysis. Most healthcare teams need both.

When tooling fits

Garak and PyRIT are open-source tools for repeatable probes: Garak scans for LLM weaknesses, and PyRIT automates adversarial testing. Use them on a non-production build after prompt or model changes, and only on fixtures that are not live patient data. Otherwise the logs become another store of protected health information. The output is a baseline, not a scope decision or a clinical judgment.

When a service fits

Buy the service for the system you will ship: the fine-tune, its instructions, retrieval, agent tools, prompt storage, and behavior when the model is wrong. Hire the service when the agent can reach records, a patient, or a clinician, when a reviewer must see exclusions, or when nobody inside can separate a security failure, a privacy leak, and a clinical safety problem.

Why both

Tools cover changes between releases. A service covers the launch gate and the retest after fixes. A tool will not sign a business associate agreement or file a finding in the risk analysis. A one-time service will not see the next model update.

How this differs from a penetration test

Buy them as separate engagements, even when one firm can do both.

A penetration test is a time-boxed attempt to break controls on an application, its APIs, and the infrastructure around them. What to put in that statement of work, how to read the report, and how to close findings are in the penetration testing guide for SaaS companies. You are buying an answer about authentication, authorization, and configuration.

An AI red team asks whether the system can be steered past a refusal, whether an agent can misuse an allowed tool, and whether output leaks data or gives guidance it should not. If the statement of work never names the model, the agent, or the test data, it is not an AI red team. If it never looks at the application, it is not a penetration test. A product team usually needs both reports.

What HIPAA requires here, and what it does not

HIPAA does not name AI red teaming, large language models, agents, or fine-tuning. The Security Rule does not require a deliverable called an "AI red team."

What it requires, from a covered entity or a business associate that holds electronic protected health information (ePHI), is a risk analysis and safeguards. Risk analysis is required: an accurate and thorough assessment of the potential risks and vulnerabilities to the confidentiality, integrity, and availability of ePHI held by that covered entity or business associate (45 CFR 164.308(a)(1)(ii)(A)). Risk management requires security measures sufficient to reduce those risks and vulnerabilities to a reasonable and appropriate level to comply with § 164.306(a) (45 CFR 164.308(a)(1)(ii)(B)). Administrative, physical, and technical safeguards sit around that analysis. The evaluation standard calls for a periodic technical and nontechnical evaluation of whether security policies and procedures meet the rule, including after environmental or operational changes that affect the security of ePHI (45 CFR 164.308(a)(8)).

A red team can feed that analysis. It does not replace it, and the rule does not name the method. If the system does not create, receive, maintain, or transmit ePHI, do not buy the test in HIPAA's name. You may still want it for safety or for other sensitive data. Say which.

The hire also raises two Privacy Rule points. A vendor that will create, receive, maintain, or transmit PHI for you needs a business associate agreement before access (45 CFR 164.502(e) and 164.504(e); Security Rule contract rules at 164.308(b) and 164.314(a)). A confidentiality clause does not replace it. Subcontractors who will handle PHI need the same kind of agreement. If no PHI will be received, say so in the statement of work. Employees are workforce. A contractor with PHI in the prompts is not. The minimum-necessary standard (45 CFR 164.502(b)) applies to what you share with a testing vendor, so test on synthetic or de-identified data wherever you can and give the vendor only the PHI the test actually needs. Do not copy production for realism without a stated reason and an agreement.

Providers, health-tech companies, and pharmaceutical companies

A provider that transmits health information electronically in a standard HIPAA transaction is a covered entity. A company that handles PHI for that provider, for a regulated function or service, is a business associate. A software seller with no PHI access is not one for that reason alone. Either way, the provider needs material for the risk analysis and the business associate needs to show the safeguards it promised.

Making drugs does not make a company a covered entity. HIPAA applies to a system only where a covered entity or business associate handles PHI. Do not call other work a HIPAA control, and do not skip privacy or safety work on a PHI system because the parent is a manufacturer. This is not a legal opinion. If the role is unclear, ask counsel before you sign.

Questions to ask a vendor before you sign

If the answer is "we will decide at kickoff," you are not ready to sign. How to design the tests is in the AI red teaming guide for healthcare.

1. Scope, including PHI and test data. Name the model, fine-tune, prompts, retrieval, agent tools, and environment, and whether production is out. Say if data is synthetic, de-identified, or PHI. De-identification is a defined HIPAA result, so name who is accountable for that call. "The chatbot" misses an agent behind it.
2. Business associate agreement. If the vendor or a subcontractor will create, receive, maintain, or transmit PHI, in prompts, outputs, tickets, or logs, sign before access. If not, the statement of work says PHI is out of scope and what happens if it appears.
3. Who touches PHI. Name roles. State where prompts, outputs, and findings are stored, for how long, and who can open them later, including notes, tickets, and any model provider that receives prompts during the test.
4. Retest. Is a retest of fixes included, and which changes reopen it: a new model, tool, data source, or user group? The rule says periodic, not annual. "Annual" on a proposal is a commercial term.
5. Report format. Severity, enough detail to reproduce the finding in the test environment, and a clear in-scope and out-of-scope line. Separate an ePHI risk from a safety issue outside the Security Rule. No score without evidence, and no claim that the test made the system safe or compliant.
6. Model and agent coverage. Base model, fine-tune, and agent are different targets. List what will not be exercised. If the agent can act, test those actions in a contained environment or say they were not tested.
7. Safety, security, and privacy-leak testing, as three lines. Safety: harmful or misleading guidance or actions, including a failure to refuse or hand off. Security: unauthorized access or actions. Privacy leaks: PHI or other sensitive data leaving through the model, a tool, or logs. Map each to a deliverable. A jailbreak score covers only one of these three lines. A bias study is extra.

A pre-deployment checklist

The questions above are what you ask the vendor. These are the launch decisions.

• State whether this workflow creates, receives, maintains, or transmits ePHI, and whether you are the covered entity, a business associate, or outside HIPAA.
• Choose the test data before any outsider sees the system. Sign the business associate agreement first if PHI is required. Otherwise keep PHI out of prompts, logs, and vendor tickets.
• List the fine-tune, its instructions, the data it can retrieve, and every action the agent can take. No list means you do not know what you are buying.
• If output can reach a patient, a chart, or a clinical decision, a clinically responsible person reviews safety findings before launch. That review is an operating choice, not a Security Rule specification.
• Assign prompt and output logs an owner, an access list, and a retention limit before the first real record is used.
• Do not launch an agent that can retrieve records or take actions on a scanner export alone.

FAQ

Should a healthcare company buy AI red team services or tooling under HIPAA?

Most healthcare teams need both. Use tooling for checks your own engineers can rerun on a non-production build with test data, not live patient data, and treat the output as a baseline rather than a scope decision. Buy a service for a scoped engagement, someone following a finding from the model into surrounding systems, and a report beside a risk analysis. The Security Rule does not require an "AI red team" deliverable. A red team can feed the risk analysis and does not replace it. If the system does not create, receive, maintain, or transmit ePHI, do not buy the test in HIPAA's name.

How is AI red teaming different from a penetration test for a healthcare app?

Buy them as separate engagements, even when one firm can do both. A penetration test is a time-boxed attempt to break controls on an application, its APIs, and infrastructure. The answer covers authentication, authorization, and configuration. An AI red team asks whether the system can be steered past a refusal, an agent can misuse an allowed tool, and output leaks data or gives guidance it should not. If the statement of work never names the model, agent, or test data, it is not an AI red team. If it never looks at the application, it is not a penetration test.

What Jacobian delivers

AI model risk management is the red-team service: a NIST AI RMF assessment (Govern, Map, Measure, Manage). Automated probing uses Garak, PyRIT, and Promptfoo. Manual work covers prompt injection, jailbreaks, RAG poisoning, and authorized model-extraction probes. Findings include reproduction prompts and severity scoring for foundation models, fine-tunes, RAG pipelines, and tool-using agents. The service implements defenses: output filtering, prompt-injection detection, RAG content provenance, and abuse-tied rate limits.

Penetration testing is a separate application, API, and network engagement. Web tests follow OWASP WSTG v4.2, API tests follow the OWASP API Security Top 10, and a retest of remediated findings is included. Web-application methodology does not cover prompt injection, jailbreaks, or data exfiltration.

GenAI compliance is the policy layer, not the model test: a use inventory, acceptable-use and data-handling rules, vendor-assessment criteria, and a gap map to the EU AI Act and the NIST AI RMF. If the workflow handles ePHI, HIPAA compliance is a risk assessment, safeguard implementation, and business associate agreement review. A HITRUST assessment is a scoped e1, i1, or r2 review. None of these replaces the red team.

Before you sign

Contact Jacobian Engineering for a scoping call. We review the system diagram, whether this workflow creates, receives, maintains, or transmits ePHI, and any draft statement of work you already have. We then deliver a statement of work covering scope, deliverables, exclusions, and what we need from you.

This article is general information, not legal advice. Consult the regulations and your own counsel.