How to evaluate a new vendor's security

A vendor review often begins with a questionnaire and ends when every cell has text in it. That produces a completed spreadsheet, which is not necessarily the same thing as a completed analysis. Security review starts when you ask what each answer proves, whether the evidence applies to the service you will use, and what happens if the answer is wrong.

Start with the proposed use, not the vendor

Before requesting a document, write down the business context. Identify the service owner, intended users, data classes, system integrations, access privileges, deployment model, geographic requirements, availability needs, and contractual obligations. Diagram the important data flows. Note whether the vendor can affect production, identity, finance, legal commitments, or safety. This is consistent with the supply-chain due-diligence questions in NIST SP 1326.

This first step determines the depth of review. A scheduling tool with public meeting information deserves a different burden of proof than a platform processing customer records and authenticating through your identity provider.

Request evidence that can answer the questions

• A current SOC 2 Type II report, including the auditor's opinion, system description, tests, results, exceptions, subservice organizations, and complementary user entity controls.
• A recent penetration-test executive summary with scope, dates, methodology, severity definitions, material findings, and remediation status.
• Architecture and data-flow documentation covering storage, processing, encryption, tenancy, backups, and external dependencies.
• Policies or precise questionnaire answers for access control, vulnerability management, secure development, incident response, business continuity, retention, deletion, and third-party risk.
• Contractual terms for breach notification, data use, subprocessors, deletion, audit rights, availability, and termination assistance.

Trust-center badges are useful routing signs. They are not the destination. A certification name without the scope, period, system, and result cannot support much of a decision.

Evaluate the evidence manually

1. Verify identity, time, and scope. Confirm the legal entity, product, infrastructure, locations, audit period, and relevant subservice organizations. Check whether the evidence is current enough for your decision.
2. Read the opinion and management assertion. Note qualifications, carve-outs, and the exact criteria covered. For SOC 2, do not skip directly to the control table.
3. Map your requirements to evidence. For each requirement, record whether it meets, does not meet, partially meets, remains pending, or has been explicitly risk accepted. Cite the page or response supporting that status.
4. Review exceptions and remediation. Understand what failed, how often, for how long, what population was affected, and whether remediation was independently tested.
5. Compare documents. Look for contradictions in retention periods, encryption descriptions, incident timing, privileged access, backup testing, and subprocessor use.
6. Ask focused follow-ups. A good question names the gap and requests evidence. "Please provide evidence that administrative MFA covers support and break-glass access" is more useful than "Do you use MFA?"
7. Write a recommendation. Approve, conditionally approve, hold pending evidence, or deny. State conditions, owners, dates, residual risks, and a reassessment trigger.

Control areas that should survive the shortcut

Whatever framework you use, make sure the review covers identity and privileged access, data protection and lifecycle, application and infrastructure security, vulnerability and patch management, secure development, logging and monitoring, incident response, resilience and recovery, personnel and third-party controls, privacy commitments, and evidence quality. Those categories map to the control families in NIST SP 800-53 Rev. 5 and the supplier-risk practices in NIST SP 800-161 Rev. 1.

For source-readable components supplied by the vendor, add code-level review of authentication and authorization, data protection, injection attacks, cryptography, error handling, configuration and deployment, code quality, dependency risk, third-party components, advisories, malware signals, install hooks, provenance, and artifact consistency.

A worked decision: the payroll integration

Imagine a vendor that will synchronize employee records between payroll and the identity platform. The vendor supplies a current SOC 2 Type II report, a penetration-test summary, a security questionnaire, and a data-processing agreement. On the surface, this is an unusually complete evidence package.

The first pass finds several strengths. The report covers the production service, workforce multifactor authentication, change control, vulnerability scanning, backups, and incident response. The penetration test includes the public application and API. The contract restricts data use and includes a defined notification period. Those conclusions belong in the decision record with their page citations.

The second pass finds a gap. The integration requires a service account that can create, disable, and modify users. The SOC 2 control describes employee access to the vendor environment, but it does not establish how customer integration credentials are stored, rotated, or constrained. The penetration test excluded the connector. The questionnaire says credentials are "encrypted," which does not answer who can retrieve them or what happens when a support engineer troubleshoots a failed synchronization.

That gap does not automatically require denial. A focused follow-up can request the connector architecture, credential-storage design, support-access workflow, audit logs, rotation support, and evidence that tenant boundaries apply to the integration service. The business can also contribute mitigating controls: a dedicated least-privilege account, network restrictions, automated reconciliation, alerting on privileged changes, and a rapid revocation procedure.

The decision may be conditional approval rather than a generic medium-risk score. Conditions could require the dedicated account, quarterly credential rotation, change alerts, and written confirmation of support-access controls before production. The business owner accepts the residual availability risk; the identity owner verifies configuration; security closes the evidence gap. If the vendor cannot answer, the status remains pending. Missing evidence is not proof of failure, but it is not proof of control either.

What a SOC 2 report can and cannot tell you

SOC 2 examinations are performed by independent licensed CPA firms. The AICPA describes the SOC suite and the CPA examination role. That audit discipline tests whether management's description is fairly presented and whether stated controls were suitably designed and, for a Type II report, operated over the review period. I use that as evidence of control performance.

My caution is about the next inference. A clean result does not automatically establish that the control set is sufficient against your threats or for your intended use. The auditor evaluates the stated system and criteria. The security reviewer still has to evaluate whether the controls that passed are the controls the decision requires.

That is not a criticism of accounting or audit. It is a division of labor. I want the auditor's disciplined testing, and I want a security practitioner to evaluate security sufficiency using the tested controls, exceptions, scope, and business context.

When a shorter review is reasonable

Vendor review can delay useful tools and exhaust small teams. A tiered process is the practical answer. Use fast, documented screening for low-risk services. Increase evidence requirements when data sensitivity, privilege, exposure, criticality, or regulatory obligations increase. Permit exceptions, but require an owner and expiry.

How to interpret silence, age, and contradiction

Vendor evidence rarely arrives as a single coherent story. A policy may be current while the penetration test is eighteen months old. A questionnaire may claim annual recovery testing while the audit report describes only backup monitoring. A contract may promise deletion while the architecture diagram shows data flowing to a subprocessor that is not named in the retention answer.

Treat those differences as review inputs, not opportunities to choose the most favorable sentence. Record the date, scope, author, and authority of each item. A tested control normally carries more weight than a policy statement for the period and population tested. A contract can create a commitment, but it does not prove that the technical process already works. A recent vendor answer may explain a post-audit change, but it should not silently rewrite the historical audit evidence.

NIST SP 1326 asks organizations to consider supplier provenance, stability, foundational security practices, and product security. Use that structure to decide what additional evidence is proportionate. The purpose is not to punish every discrepancy. It is to prevent a decision from depending on an assumption nobody realized they had made.

Sources and further reading

• NIST SP 1326, Cybersecurity Supply Chain Risk Management: Due Diligence Assessment Quick-Start Guide
• NIST SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices
• NIST SP 1305, Cybersecurity Framework 2.0 Quick-Start Guide for Cybersecurity Supply Chain Risk Management
• NIST SP 800-53 Rev. 5, Security and Privacy Controls
• AICPA, System and Organization Controls Suite of Services

Managing repeated vendor reviews

Can I Run That? can organize vendor evidence, requirement mapping, citations, contradictions, follow-ups, and decision history across many reviews. The reviewer still resolves uncertainty, applies business context, and owns the recommendation.

A useful review ends with a decision record

The deliverable is not the questionnaire, SOC 2 report, or meeting notes. It is a record showing what was proposed, what evidence was reviewed, what the evidence supports, what remains uncertain, what conditions apply, who owns them, and when the decision should be revisited.