Eight criteria for evaluating AI development vendors, with copy-paste RFP questions for each. Use it as a vendor-meeting prep doc, an RFP rubric, or just a sanity check on your AI partner shortlist.
Picking a generalist software outsourcing partner is hard. Picking an AI development partner is harder, because the surface signals look almost identical to generalist outsourcing — same case studies pages, same Clutch ratings, same pitches. The difference shows up only when you ask AI-specific questions, and most buyers don't know which questions to ask.
This framework covers 8 AI-specific criteria layered on top of standard vendor evaluation (engineering quality, contracts, communication — covered separately in our generalist software vendor guide). Together they form a complete RFP rubric for AI-first engagements.
The single best predictor of whether a vendor can ship AI is whether they have shipped AI recently. Production deployments older than 18 months tell you the team had the capability — but AI tooling, model selection, and deployment patterns shift fast. "We did some great LLM work in 2024" is a yellow flag in 2026 unless they can also point to recent work. Ask for 5 production AI deployments shipped in the last 18 months, with the LLM provider, model, scale (requests/month), and a measurable outcome each.
RFP question
List 5 production AI systems your team shipped in the last 18 months. For each: client industry, LLM model + version, monthly request volume, and one quantified outcome.
The strongest signal of mature AI engineering: when you ask about a project, the vendor talks about eval methodology before they talk about models or prompts. Weak signal: they describe "we used GPT-4" and "the customer was happy." Strong signal: they describe "we built a 200-case frozen eval set, scored on faithfulness + answer relevance, CI-gated releases at 0.92 minimum." The eval framework is the operating system of production AI; vendors without one are shipping prototypes.
RFP question
Walk us through the eval framework for your most recent AI deployment. How many test cases? What scoring functions? What's the regression-detection cadence?
For agent or RAG projects, the architecture details determine whether the system survives past month two. Ask about memory tiers (working / short-term / long-term separation), retrieval methodology (pure semantic vs hybrid + reranking), and cold-start handling. A vendor that answers "we use a vector database" without specifying which tier of memory it serves or how retrieval is structured is treating memory as a single-shot solution — which fails at scale.
RFP question
Describe the memory tier architecture you'd build for our use case. Where does pgvector / Pinecone / etc sit? What's the retrieval-then-reranking pattern?
Most production AI is running 70-90% over its optimal cost because the team never went back to optimize. A mature AI vendor talks about model routing (classifier dispatching cheap tasks to cheap models), prompt caching (Anthropic + OpenAI cache discounts), batch tier (50% off for async workloads), and per-task hard caps (preventing $4 pathological tasks). If they don't mention these unprompted in the discovery, your project will be 5x more expensive to run than necessary.
RFP question
What's your default cost-optimization checklist for new AI deployments? Walk us through 3 specific patterns you'd apply.
MLOps separates teams that can debug a 3am model regression from teams that can't. Ask about: model registry (MLflow, SageMaker Model Registry, custom), CI/CD for model code, drift detection, prediction logging, kill switch (single config toggle to disable AI calls in incident). "We deploy to AWS" is not an answer. "We use MLflow Model Registry, OpenTelemetry traces with phi=true tagging for sensitive workloads, Datadog dashboards, kill switch wired to PagerDuty" is.
RFP question
Describe your production observability stack for AI systems. How would I as the client replay a specific failed prediction at 11pm?
GDPR, HIPAA, PDPA, PIPEDA, APPI, EU AI Act are all distinct regimes. Vendor experience in one does not transfer cleanly to another. Ask for: which compliance regimes your team has shipped under in the last 18 months, with the client jurisdiction confirmed. "We're GDPR-aware" is not the same as "we've shipped GDPR-compliant AI for 3 EU clients in the last 12 months and can walk through our DPIA template."
RFP question
Which compliance regimes have you shipped AI deployments under in the last 18 months? For each, how many client deployments, and can we see your DPA template?
Production AI systems with user-provided input (or LLM input from emails, scraped pages, etc.) face prompt injection attacks. A vendor that doesn't bring this up before you do is not thinking about adversarial inputs. Ask about: input sanitization, delimiter-based prompt isolation, LLM model selection (some are RLHF-resistant to injection, others aren't), output filtering, and incident response when an injection succeeds.
RFP question
Walk us through how you'd defend our agent against prompt injection. What's your standard pattern for sanitizing user inputs that flow into LLM context?
An AI deployment is the start of a 24-36 month operating lifecycle, not the end. Ask: what does your handoff include? A 2-page Confluence page is not a handoff. A real handoff includes: runbook for the on-call team, incident-response procedures specific to AI failures, eval framework that your team can extend, model-refresh playbook, and monthly retainer option for continued tuning. Vendors who optimize for handoff-and-disappear ship brittle systems that rot in 6 months.
RFP question
Describe your handoff deliverables. What does the runbook look like? Do you offer ongoing retainer for model tuning, or do we operate the system entirely after delivery?
For each criterion, score each vendor 1–5:
Decision rule: a vendor scoring 3+ across all 8 criteria is shortlist-worthy. A vendor scoring 2 or below on more than two criteria is a likely pass, even if their sales process feels strong. The sales process is selecting for what they can demo, not what they ship.
Free 30-minute call: walk through your AI vendor shortlist with a NKKTech senior engineer. We'll review the responses you've gotten and tell you which red flags to weight more heavily for your specific project. Even if NKKTech isn't the right vendor, we'd rather help you pick well.
Book a free RFP review