The upcoming 3.2 release will include software metrics, one of which is Cyclomatic Complexity. Cyclomatic Complexity is used in finding the C.R.A.P. index of a method. The Change Risk Analysis and Predictions (CRAP) index is to give some idea how difficult it would be to maintain the code base. It is not to be used in evaluating how "beautiful" the code is.
Crap Index
I'm handling entire files and functions, which PHPUnit does not support, I had to port PHPUnit's code to allow for this. Note, with some tests, it appears that using the Change Risk Analysis and Predictions (CRAP) index on an entire file does not give expected results. During tests with 0 code coverage, scored better than with 30% code coverage. It also appears, that ironically, I needed to unit test the below functions.
Once a file closes on 100% code coverage, the results are better. However, the C.R.A.P index is going to be worse until then. So this gives us the following table.
0%: Good 1% through 94%: Bad 95%+: Better
This gives a bad impression of Unit Tests for entire files (which unit tests aren't meant for in the first place, nor is the CRAP index for that matter). 30% code coverage would appear to be worse than having 0%, when it is in fact still better.
I would assume this means that if you don't have any code coverage then you assume the worse, when testing for bugs. If you have some code coverage, a new developer might then assume that the unit tests cover everything and blindly expect that if something breaks, that it would be covered. If there is ~100% code coverage, then it can be expected that bugs not covered will be a smaller amount and a new developer can expect that most changes will be covered.
What is Your C.R.A.P. Index?
I'm going to use WordPress as an example for testing entire files. It would be interesting if PHPUnit added Change Risk Analysis and Predictions as an metric for methods, when it was finished and stabilized. At least it wouldn't be difficult to add it.
I thought it would be interesting to test S9Y for a comparsion, but since it doesn't include any unit tests, I'm not.
Code
[php] /** * Based off of http://www.artima.com/weblogs/viewpost.jsp?thread=210575 * comp(m) = cyclomatic complexity * cov(m) = code coverage * * C.R.A.P. Level is based off of the file and not the method. */ function getCrapIndex($cyclomatic_complexity, $code_coverage) {
if($code_coverage == 0) { // comp(m)^2 + comp(m) if(function_exists('gmp_pow')) { return gmp_pow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } else if(function_exists('bcpow')) { return bcpow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } else { return pow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } } else if($code_coverage >= 95) { // comp(m) return $cyclomatic_complexity; } else { // comp(m)^2 * (1 – cov(m)/100)^3 + comp(m) if(function_exists('gmp_pow')) { return gmp_mul( gmp_pow($cyclomatic_complexity, 2), ( gmp_pow( ( gmp_sub(1, $code_coverage/100) ), 3) + $cyclomatic_complexity ) ); } else if(function_exists('bcpow')) { return bcmul( bcpow($cyclomatic_complexity, 2), ( bcpow( ( bcsub(1, $code_coverage/100) ), 3) + $cyclomatic_complexity ) ); } else { return pow($cyclomatic_complexity, 2) * (pow(1-$code_coverage/100, 3) + $cyclomatic_complexity); }
}
} [/php]
The above finds the Change Risk Analysis and Predictions (CRAP) index. The crap index reduces when the code coverage is 0% or 100%. Since it is easier and quicker, I added it in based on the web site. The last one uses BCMath, for fairness and accuracy. If you don't have BCMath extension, then it is
[php] $crapLevel = pow($cyclomatic_complexity, 2) * (pow(1-$code_coverage/100, 3) + $cyclomatic_complexity); [/php]
[php] // Ported from PHPUnit // http://www.phpunit.de/browser/phpunit/branches/4.0/PHPUnit/Util/Metrics/Method.php function getCCLevel($file) { $tokens = token_get_all(file_get_contents($file));
$ccn = 1;
foreach ($tokens as $i => $token) { if (is_string($token)) { continue; }
list ($token, $value) = $token;
switch ($token) { case T_IF: case T_FOR: case T_FOREACH: case T_WHILE: case T_CASE: case T_CATCH: case T_BOOLEAN_AND: case T_LOGICAL_AND: case T_BOOLEAN_OR: case T_LOGICAL_OR: { $ccn++; } break; } }
return $ccn; } [/php]
As you can see, the only difference between PHPUnit and my function getCCLevel() is that I'm inserting the entire file.
WordPress Crap Index By File
With 0% code coverage.
The files with CRAP index of 2 are files that include other files and only have a single if statement for including wp-settings.php. Some files, such as xmlrpc.php has internal functions and classes which would have to be removed for unit testing to provide the whole picture.
A lot of the WordPress root directory files and wp-admin directory files has presentation and business logic coupled. Because of this, the numbers are even more skewed. wp-admin files weren't included because of this. Most developers shouldn't mess with the wp-admin files.
WordPress includes enough hooks to where most developers won't be required to modify the core of WordPress, so these numbers mean nothing to plugin developers. The only exception, perhaps, are the undocumented hooks and filters, that would have to be tracked down. In which case, these numbers will paint a pretty good picture of the difficulty.
As with any metric, take what you will from them, but don't take them as a serious indicator. The Change Risk Analysis and Predictions is meant for methods (and probably functions also), so these are just for fun and shouldn't be (mis)used against WordPress as a whole.
The colors mean nothing and just for making the numbers look pretty.
Base of WordPress
| C.R.A.P. | File |
|---|---|
| 2 | index.php |
| 2 | license.txt |
| 2 | readme.html |
| 2 | wp-register.php |
| 2 | wp-config-sample.php |
| 6 | wp-atom.php |
| 6 | wp-commentsrss2.php |
| 6 | wp-pass.php |
| 6 | wp-rdf.php |
| 6 | wp-rss.php |
| 6 | wp-rss2.php |
| 6 | wp-feed.php |
| 20 | wp-blog-header.php |
| 90 | wp-links-opml.php |
| 110 | wp-cron.php |
| 182 | wp-comments-post.php |
| 306 | wp-trackback.php |
| 702 | wp-mail.php |
| 2970 | wp-settings.php |
| 3660 | wp-login.php |
| 20,022 | wp-app.php |
| 39,006 | xmlrpc.php |
WordPress Includes
| C.R.A.P. | File |
|---|---|
| 2 | version.php |
| 2 | rss-functions.php |
| 2 | registration-functions.php |
| 6 | default-filters.php |
| 12 | feed-rss.php |
| 20 | feed-atom.php |
| 20 | feed-rdf.php |
| 20 | feed-rss2.php |
| 30 | locale.php |
| 56 | feed-rss2-comments.php |
| 72 | feed-atom-comments.php |
| 72 | vars.php |
| 90 | streams.php |
| 210 | l10n.php |
| 380 | compat.php |
| 462 | registration.php |
| 600 | category.php |
| 600 | plugin.php |
| 650 | feed.php |
| 702 | author-template.php |
| 812 | bookmark.php |
| 1260 | deprecated.php |
| 1260 | user.php |
| 1332 | cron.php |
| 1560 | gettext.php |
| 1560 | template-loader.php |
| 1722 | wp-db.php |
| 1722 | comment-template.php |
| 2550 | kses.php |
| 2756 | script-loader.php |
| 3306 | bookmark-template.php |
| 4422 | cache.php |
| 4830 | capabilities.php |
| 5112 | category-template.php |
| 5852 | post-template.php |
| 6806 | class-pop3.php |
| 6806 | theme.php |
| 7832 | link-template.php |
| 9702 | class-smtp.php |
| 11990 | pluggable.php |
| 11990 | rewrite.php |
| 12210 | formatting.php |
| 14520 | class-IXR.php |
| 15006 | rss.php |
| 16770 | comment.php |
| 17556 | class-phpmailer.php |
| 20306 | classes.php |
| 24492 | widgets.php |
| 28730 | taxonomy.php |
| 28730 | general-template.php |
| 37442 | class-snoopy.php |
| 41006 | functions.php |
| 66306 | query.php |
| 95790 | post.php |
Total Crap Level: 598398
Sources
1. http://www.artima.com/weblogs/viewpost.jsp?thread=210575 2. WordPress Subversion Repository.