What do black-box, gray-box and white-box mean?
They describe the tester's starting knowledge. The PCI SSC's penetration testing guidance defines them in almost identical terms: black-box testing is done without prior knowledge of the internal structure, design or implementation of the target; gray-box with partial knowledge; white-box with knowledge of it. NIST's glossary uses the same split, and SP 800-115 describes gray-box as the combination of white-box and black-box techniques.
The labels describe knowledge, and they are separate from two other choices that often get mixed in:
- Credentials. Whether the tester logs in is a separate question. A gray-box test is usually authenticated, but a tester can hold source code and still attack only the public surface. See authenticated penetration testing.
- Staff awareness. NIST SP 800-115 calls testing done without the IT staff's knowledge, but with upper management's permission, "covert," and testing with their knowledge "overt." A black-box test can be overt.
Vendors also stretch the terms. Some call a test with only a URL and logins "black-box" because no documentation was shared; under the PCI definitions that is closer to gray-box. When comparing quotes, ask exactly what the tester receives.
How do the three compare?
| Aspect | Black-box | Gray-box | White-box |
|---|---|---|---|
| Tester receives | A target (domain, IP range, app URL) and permission | Accounts for each role, API specs or schemas, architecture notes | Source code, infrastructure-as-code, configs, design docs, plus accounts |
| Simulates | An outside attacker with no insider help | A customer, a partner, or an attacker with a stolen login | An insider, or an attacker who has already obtained the code |
| Strong at finding | Exposed services, forgotten hosts, unauthenticated flaws, what leaks to the internet | Broken access control between roles and tenants, business logic, injection behind login | Flaws in code paths hard to reach from outside, hardcoded secrets, weak crypto use, unsafe deserialization, logic that only reads wrong in source |
| Tends to miss | Everything behind login, deep logic, rarely used features | Code paths no UI or spec reveals | Problems that only appear in the deployed system: misconfigured infrastructure, component interactions |
| Time spent | Much of it on reconnaissance and guessing | Mostly on attacking | Split between reading code and confirming in the running app |
| Cost for equal coverage | Highest | Middle | Often the most efficient per finding for custom code, if testers can read your stack |
NIST's Appendix C makes a similar point about application testing: white-box techniques tend to be more efficient for finding defects in custom code, but they cannot detect problems in the interfaces between components or those introduced at build and installation time, which is what black-box techniques are for. Hence gray-box, which does both.
Which one should you pick?
For a SaaS product, gray-box is the usual default: accounts for every role, the API spec, and a short architecture walkthrough. It spends the tester's hours on attack, not reconnaissance. A few rules of thumb:
- Choose black-box when the question is "what can someone with nothing find?" This fits an external attack surface review, or a check on whether an acquisition's forgotten hosts are exposed.
- Choose gray-box for most application and API tests, and for any test meant to evidence access control.
- Choose white-box when the code is the risk: authentication and payment logic, cryptography, a new framework, or a team that wants findings tied to file and line.
- Combine them when budget allows: a short black-box reconnaissance phase, then gray-box or white-box testing on what matters.
Whatever you choose, write it into the rules of engagement and make sure the report states it. Reviewers reading a penetration test attestation letter look for this line.
Which approach do compliance frameworks expect?
Mostly the frameworks leave it to you, with two notable exceptions in guidance documents.
- PCI DSS. The PCI SSC guidance says PCI DSS penetration tests are typically performed as white-box or gray-box assessments, because they yield more accurate results than pure black-box testing. It adds that a black-box assessment may require more time, money and resources to meet PCI DSS requirements. This is guidance, not a requirement in the standard itself.
- FedRAMP. The Penetration Test Guidance has cloud providers supply privileged accounts in production, and tenant-to-management-system attacks run with the highest permissions available to customer users. A purely black-box test would not meet that.
- SOC 2 and ISO 27001. Neither names a testing approach. Auditors generally look at whether scope and method are stated and match the system in scope.
Common mistakes
- Buying a black-box test to save money, then spending the engagement on reconnaissance you could have handed over in an hour.
- Calling a test white-box when the tester received the repo but no time to read it.
- Omitting the approach from the report, which leaves reviewers to assume the weakest one.
[ Sources ]
- PCI SSC Information Supplement: Penetration Testing Guidance v1.1 (2017), sections 1.3 and 2
- NIST SP 800-115: Technical Guide to Information Security Testing and Assessment (Appendix C)
- NIST CSRC Glossary: gray box testing
- NIST CSRC Glossary: black box testing
- FedRAMP Penetration Test Guidance v3.0 (2022)
Written by Parameter · Last reviewed
