I was able to figure this out from the error message in web.log that pointed to Troubleshooting OAuth app access token request errors - GitHub Docs so score one for good logging info.
However, I would have expected that test configuration (and the bulk configuration of users and permissions) would not have succeeded with the incorrect value for the secret key.
- Developer Edition
- Version 10.2.1 (build 78527)
- Zip install on a vmware VM