Overview
StrongDM’s AI team has implemented a “Software Factory” approach where coding agents write all code without any human review or intervention. They solve the quality assurance problem by using scenario-based testing with probabilistic satisfaction metrics rather than traditional unit tests.
The Breakdown
- No human code review policy - agents write and deploy code directly based on specs and scenarios, with humans never examining the generated code
- Scenario-based validation system - replaces traditional boolean tests with end-to-end user stories that measure probabilistic satisfaction across multiple execution paths
- $1,000+ daily token spending per engineer - indicates the computational intensity required to make fully automated coding workflows reliable and effective
- Transition from boolean to probabilistic success metrics - moves away from simple pass/fail tests to measuring what fraction of user scenarios are likely satisfied
- Claude 3.5 Sonnet as the inflection point - the October 2024 model revision enabled long-horizon coding workflows that compound correctness rather than accumulating errors