Vendor risk review vs. vulnerability scanning

Security tools are easiest to misuse when their output looks definitive. A scan can return a reassuring page of green checks while saying nothing about incident notification, privileged support access, backups, tenant isolation, or what the vendor does with customer data.

What each method evaluates

Vulnerability scanning

Scanning tests observable assets, software versions, configurations, dependencies, or code patterns for known or detectable weaknesses. Its quality depends on target coverage, credentials, signatures, configuration, timing, and whether the relevant attack surface can be observed. NIST SP 800-115 describes vulnerability scanning as one technical assessment technique and explains the limitations of scanner output.

Vendor risk review

Vendor review evaluates evidence about the service and the organization operating it: identity controls, data protection, secure development, vulnerability management, incident response, resilience, privacy, subprocessors, contractual commitments, exceptions, and unanswered questions. It also applies the customer's business context. NIST SP 800-161 Rev. 1 treats supplier risk as a lifecycle and governance problem, not merely a vulnerability count.

Scanning is not one thing

"We ran a scan" is not enough information to interpret a result. An unauthenticated external scanner sees what an unknown internet user can reach. An authenticated infrastructure scan may inspect patch and configuration state inside an operating system. A dependency scanner compares an inventory with advisory records. Static analysis examines code patterns without executing the application. Dynamic testing interacts with a running target. Each technique can be useful, and each has a different blind spot.

NIST SP 800-115 separates technical testing and examination methods and emphasizes planning, target selection, rules of engagement, and analysis. Those details decide what a result means. A clean unauthenticated scan does not establish that authenticated functions are safe. A dependency scan does not establish that proprietary application logic enforces authorization. A static finding does not establish exploitability without a path through the deployed system.

Before accepting a scan as evidence, record the target, date, tool or method, authentication level, source inventory, exclusions, failure messages, and whether remediation was retested. If a scanner could not authenticate to half the systems, the correct outcome is partial coverage, not a smaller vulnerability count.

Use both in a manual decision workflow

1. Define the proposed use, data, access, exposure, criticality, and owner.
2. Collect organizational evidence such as SOC 2, penetration testing, policies, architecture, questionnaires, and contracts.
3. Run authorized technical testing against the relevant surface or examine available source and packages.
4. Map all evidence to requirements and preserve citations, code references, scope, dates, and coverage limitations.
5. Reconcile conflicts. A clean scan does not erase a weak incident process. A strong policy does not erase an exposed service.
6. Record gaps, mitigating controls, follow-ups, and an approve, conditional, pending, or deny decision.

A worked decision: the vendor with the clean scan

Consider a scheduling vendor that will connect to the corporate directory, read employee calendars, and send meeting data to an artificial-intelligence feature. The vendor provides a recent external penetration-test letter stating that no critical or high-severity findings remained after remediation. The sales team calls this proof that the platform is secure.

The letter supports a narrower conclusion. A defined target was tested during a defined period, findings were classified under a stated method, and the reported severe findings were remediated. It does not tell you whether support engineers can impersonate users, whether deleted calendar content remains in backups, whether the AI subprocessor can train on prompts, or whether directory tokens are limited and rotated.

The vendor review supplies another set of evidence. The SOC 2 report shows workforce access testing and incident-response exercises. The architecture diagram identifies the calendar connector but excludes the new AI service because it launched after the audit period. The contract permits subprocessors but does not name a notification period for additions. The questionnaire says customer data is not used for training, but the product terms contain a broader data-use clause.

The scan did not fail. It answered its question. The review now has three pending questions: the AI service boundary, the controlling data-use term, and the connector token's privilege. A reasonable decision may allow a pilot with synthetic calendars, block directory-wide consent, and require written evidence before production data is enabled. If the business needs an immediate scheduling tool without the AI feature, that narrower deployment may be approvable sooner.

Technical testing can contradict an overly optimistic policy, and organizational evidence can reveal risks the scanner could never observe. Neither should be forced to impersonate the other.

Common substitution errors

• Treating a recent penetration test as proof of continuous vulnerability management.
• Treating a SOC 2 control for scanning as proof that no exploitable vulnerability exists.
• Treating a dependency advisory scan as a review of application logic.
• Treating the absence of findings as evidence of complete coverage.
• Treating vendor policy language as evidence that the control operated.

Even vulnerability records require interpretation. The CVE Program explains what a CVE record does and does not represent, FIRST explains how CVSS communicates severity, and the CISA Known Exploited Vulnerabilities Catalog adds evidence of exploitation in the wild. None of those sources supplies your deployment context.

Build one coverage map instead of two disconnected reports

Put requirements in rows and evidence sources in columns. A row for administrative access might cite a SOC 2 control test, a questionnaire answer about support impersonation, a product configuration guide, and the customer's planned SSO setting. A row for application vulnerabilities might cite the penetration-test scope, retest results, dependency inventory, and any authorized scanning. Mark the state of the requirement and the coverage of each source separately.

This makes overlap useful. If the vendor says it performs authenticated scanning, the SOC 2 report may show that scans occurred on schedule while a penetration test examines whether a particular weakness was discoverable. If your own scan finds an exposed service, the vendor's vulnerability-management evidence helps identify whether the asset was in inventory and whether the process should have found it. The sources answer different parts of the same requirement.

The map also makes absence visible. There may be no technical test of tenant isolation, no audit-period evidence for a newly released AI feature, or no contract term that preserves customer notification. Those are not necessarily failures. They are pending questions whose importance depends on the proposed use. NIST SP 1326 organizes supplier due diligence around information that supports risk decisions, which is a better objective than collecting the largest possible stack of reports.

When evidence conflicts, preserve the disagreement

A vendor may claim monthly authenticated scanning while the audit describes quarterly external scans. A dependency tool may report a critical advisory while the vendor says the affected function is not used. A penetration test may find no issue in the public application while source review finds an authorization path in an administrative API that was out of scope.

Do not resolve these conflicts by choosing the document with the strongest logo or the number with the most decimal places. Write down both claims, their dates and scopes, and the evidence each provides. Ask for the missing bridge: scanner configuration, inventory coverage, reachability analysis, API scope, remediation commit, or retest result. If the bridge never arrives, leave the requirement partial or pending and decide whether business context permits that uncertainty.

Severity is one input. The FIRST CVSS user guide explains that the score represents characteristics of a vulnerability, while environmental considerations can change its relevance to a consumer. The CISA KEV Catalog adds evidence that particular vulnerabilities have been exploited. Neither decides whether the affected component exists in your vendor's service or whether your planned integration exposes it. That final mapping is review work.

Sources and further reading

• NIST SP 800-115, Technical Guide to Information Security Testing and Assessment
• NIST SP 800-161 Rev. 1, Cybersecurity Supply Chain Risk Management Practices
• CISA, Known Exploited Vulnerabilities Catalog
• CVE Program, Frequently Asked Questions
• FIRST, Common Vulnerability Scoring System User Guide

Keeping the evidence paths separate

Can I Run That? analyzes vendor documents separately from source, dependencies, advisories, malware signals, provenance, and code-level risks. Reviewers can see what each evidence path supports before making a decision.