Hey @Liver
The documentation on custom rules development is usually the right place to start.
To build the rule you are talking about, I would look in the examples at the NoIfStatementRule that tests if there are if statements in Junit4 code.
If you want to dive deeper in the code, the CognitiveComplexityVisitor counts different elements in the code and takes the level of nesting into account.