Tangible Security, a leading cybersecurity firm, conducted a controlled bakeoff test to evaluate the results of external and internal penetration testing between two widely used automated penetration testing platforms and a manual penetration test for a target network. For anonymity, the platforms are referred to as Platform A and Platform B, and the target network as Acme Corporation. This blog post compares the results, highlighting the strengths and limitations of each approach, providing examples of when automated testing is suitable, emphasizing the depth of manual testing, and exploring how a hybrid approach can bridge the gaps.

Overview of the Penetration Tests

Vulnerability	Automated Platform A	Automated Platform B	Manual Tester
Default or Weak Credentials	X	✓	✓
Misconfigured Services	✓	✓	✓
Network Protocol Weaknesses	✓	X	✓
Outdated Software	✓	X	✓
Privilege Escalation	X	X	✓
Sensitive Data Exposure	✓	✓	✓
System Exploitation	X	X	✓

 

Automated Penetration Testing: Platform A

We ran Platform A for one day and identified vulnerabilities in Acme Corporation’s network across several categories:

• Network Protocol Weaknesses: Issues allowing potential spoofing or interception.
• Outdated Software: Systems vulnerable to known exploits due to lack of updates.
• Misconfigured Services: Insecurely configured network services and protocols.
• Sensitive Data Exposure: Potentially sensitive information in unauthenticated shares, detected using a keyword list against file names.

Platform A’s reporting was clear with actionable mitigation recommendations. No accounts were compromised.

Automated Penetration Testing: Platform B

We ran Platform B for one day and identified:

• Misconfigured Services: Insecure network services and access controls, overlapping with Platform A.
• Default or Weak Credentials: Systems using unchanged or easily guessable credentials.
• Application Vulnerabilities: A reported issue in a web application, later confirmed as a false positive.
• When provided with unprivileged user credentials, Platform B found additional issues in authentication weaknesses and directory service vulnerabilities, plus an unconfirmed system service vulnerability.
• Platform B effectively identified sensitive data exposure in shares by downloading and locally analyzing files.

Platform B’s reporting was easy to read with actionable recommendations.

Manual Penetration Testing by Tangible Security

Tangible Security’s manual penetration test was conducted over a week by skilled and experienced testers and included both external and internal assessment of the same target network evaluated by Platforms A and B. The manual test was performed prior to the automated tests to ensure an unbiased comparison, with no advantage gained from visibility into the results generated by the automated platforms.

Key findings included:

• Credential Compromise: An account was cracked, used to access a collaboration platform, and uncovered additional credentials in shared files.
• Contextual Analysis of Sensitive Documents: Testers reviewed accessible documents and were able to understand which documents were significant, such as strategic documents or customer data, which the surface-level detection used by the automated tools was unable to perform.
• Privilege Escalation: Discovered credentials were used to gain higher-level access.
• System Exploitation: Compromised multiple systems via a critical vulnerability, undetected by automated platforms.
• Weak Web Credentials: Unchanged credentials on web interfaces across the network.

The manual test identified all vulnerability categories found by both automated platforms, plus additional vulnerabilities and positive findings highlighting Acme Corporation’s strengths, such as effective network defenses or secure backups.

Comparing Automated and Manual Penetration Testing

Strengths of Automated Penetration Testing

Tangible Security noted that automated platforms excel in speed, consistency, and broad coverage:

• Speed: Both platforms completed scans in one day, ideal for quick assessments with broad coverage.
• Standardized Detection: They reliably identified known vulnerabilities, such as misconfigured services and outdated software using prebuilt signatures.
• Sensitive Data Detection: Platform A used keyword-based detection, while Platform B analyzed file contents locally, both identifying exposed data.
• Actionable Reporting: Clear mitigation steps enabled Acme Corporation’s IT team to prioritize remediation.
• Credentialed Scanning: Platform B’s use of unprivileged credentials uncovered deeper issues like authentication and directory service vulnerabilities.

Limitations of Automated Penetration Testing

Tangible Security noted that automated platforms struggled with limited depth, false positives, narrow scope, limited real-world attack simulation, and unclear compliance alignment:

• Limited Depth: Neither platform detected system exploitation vulnerabilities nor performed privilege escalation, relying on known signatures.
• False Positives: Platform B’s reported application vulnerability was a false positive, and its system service vulnerability was unconfirmed, requiring manual verification.
• Surface-Level Analysis: Automated tools couldn’t contextually analyze sensitive documents or use discovered credentials for privilege escalation, limiting real-world attack simulation.
• Unclear Compliance Alignment: It is not currently clear whether automated penetration testing fully aligns with cybersecurity compliance standards.

Strengths of Manual Penetration Testing

Tangible Security’s manual testing provided a deeper, contextual, and comprehensive approach that aligns with a real-world attack simulation:

• Contextual Document Analysis: Testers reviewed the content of sensitive documents, revealing their true impact, unlike automated scans.
• Privilege Escalation: Using discovered credentials, testers escalated access, demonstrating attack progression.
• Creative Exploitation: A critical vulnerability compromised multiple systems, requiring manual analysis missed by automated tools.
• Chained Attacks: Cracking an account led to platform access and further credential discovery.
• Positive Feedback: Highlighting strengths fostered a proactive security culture.
• Custom Discovery: Testers identified weak web credentials through manual exploration.

Limitations of Manual Penetration Testing

Tangible Security noted:

• Time-Intensive: Manual penetration testing occurred over several weeks.
• Cost: Manual penetration testing tends to be more costly than automated testing due to the need for highly skilled professionals and the greater time investment required.
• Rapidly Changing Networks: Manual testing (if performed annually) is less suited for rapidly changing networks.

When Automated Penetration Testing Makes Sense

Tangible Security recommends automated testing for:

• Frequent Scans: Monthly checks for misconfigured services or outdated software.
• Rapidly Changing Networks: Routine scanning of rapidly changing networks.
• Initial Assessments: Cost-effective starting point before manual testing.
• Continuous Monitoring: Integration into DevSecOps for ongoing vulnerability detection.

For example, Acme Corporation could use Platform A to monitor outdated software or Platform B to check for weak credentials in new deployments.

The In-Depth Approach of Manual Penetration Testing

Tangible Security’s manual testing simulates real-world attacks with creativity:

• Human Intuition: Chaining vulnerabilities (e.g., account compromise to platform access) and escalating privileges mimics sophisticated adversaries.
• Contextual Analysis: Reviewing the content of sensitive documents revealed their impact, unlike automated scans.
• Custom Exploitation: Critical vulnerabilities required manual analysis.
• Contextual Reporting: Positive findings informed Acme Corporation’s security investments.

In addition to its core capabilities, manual penetration testing can be tightly integrated with other assessments such as wireless security testing, social engineering exercises, and physical security evaluations to simulate realistic, multi-vector attacks. This approach mirrors how adversaries operate across domains, using one foothold to gain another. In contrast, automated platforms operate in isolation and lack visibility into these adjacent attack surfaces, making it difficult to chain attacks across services. By combining multiple vectors into a coordinated assessment, organizations can more effectively evaluate their true security posture.

Bridging the Gap with a Hybrid Approach

Tangible Security advocates a hybrid approach:

• Initial Automated Scans: Use Platform A or B to identify common vulnerabilities, reducing the scope for manual testers.
• Manual Deep-Dive: Focus on high-risk areas such as web applications, physical security, or privilege escalation.
• Credentialed Testing: Platform B’s credentialed scans uncovered deeper issues, complementing manual efforts.
• Iterative Validation: Manual testers verify automated findings to eliminate false positives.

Acme Corporation could run Platform B monthly for monitoring and conduct annual manual tests by Tangible Security for in-depth attack simulations.

Conclusion

Tangible Security’s evaluation for Acme Corporation shows that automated platforms like A and B offer speed and cost-effectiveness for frequent scans but miss critical vulnerabilities like system exploitation, contextual document analysis, and privilege escalation. Manual testing provides unparalleled depth and insight into complex attack paths, including the ability to chain vulnerabilities, evaluate contextual risk, and simulate real-word adversary behavior. Manual testing can also be integrated with complementary services such as wireless assessments, social engineering and physical security testing to create multi-vector attack scenarios that reflect how threats unfold across different domains. A hybrid approach that combines automated tools for routine and regular assessments with manual testing for in-depth analysis provides comprehensive security coverage while optimizing both effectiveness and efficiency.