What is your C.R.A.P. Index?

The upcoming 3.2 release will include software metrics, one of which is Cyclomatic Complexity. Cyclomatic Complexity is used in finding the C.R.A.P. index of a method. The Change Risk Analysis and Predictions (CRAP) index is to give some idea how difficult it would be to maintain the code base. It is not to be used in evaluating how "beautiful" the code is.

Crap Index

I'm handling entire files and functions, which PHPUnit does not support, I had to port PHPUnit's code to allow for this. Note, with some tests, it appears that using the Change Risk Analysis and Predictions (CRAP) index on an entire file does not give expected results. During tests with 0 code coverage, scored better than with 30% code coverage. It also appears, that ironically, I needed to unit test the below functions.

Once a file closes on 100% code coverage, the results are better. However, the C.R.A.P index is going to be worse until then. So this gives us the following table.

0%: Good 1% through 94%: Bad 95%+: Better

This gives a bad impression of Unit Tests for entire files (which unit tests aren't meant for in the first place, nor is the CRAP index for that matter). 30% code coverage would appear to be worse than having 0%, when it is in fact still better.

I would assume this means that if you don't have any code coverage then you assume the worse, when testing for bugs. If you have some code coverage, a new developer might then assume that the unit tests cover everything and blindly expect that if something breaks, that it would be covered. If there is ~100% code coverage, then it can be expected that bugs not covered will be a smaller amount and a new developer can expect that most changes will be covered.

What is Your C.R.A.P. Index?

I'm going to use WordPress as an example for testing entire files. It would be interesting if PHPUnit added Change Risk Analysis and Predictions as an metric for methods, when it was finished and stabilized. At least it wouldn't be difficult to add it.

I thought it would be interesting to test S9Y for a comparsion, but since it doesn't include any unit tests, I'm not.

Code

[php] /** * Based off of http://www.artima.com/weblogs/viewpost.jsp?thread=210575 * comp(m) = cyclomatic complexity * cov(m) = code coverage * * C.R.A.P. Level is based off of the file and not the method. */ function getCrapIndex($cyclomatic_complexity, $code_coverage) {

if($code_coverage == 0) { // comp(m)^2 + comp(m) if(function_exists('gmp_pow')) { return gmp_pow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } else if(function_exists('bcpow')) { return bcpow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } else { return pow($cyclomatic_complexity, 2) + $cyclomatic_complexity; } } else if($code_coverage >= 95) { // comp(m) return $cyclomatic_complexity; } else { // comp(m)^2 * (1 – cov(m)/100)^3 + comp(m) if(function_exists('gmp_pow')) { return gmp_mul( gmp_pow($cyclomatic_complexity, 2), ( gmp_pow( ( gmp_sub(1, $code_coverage/100) ), 3) + $cyclomatic_complexity ) ); } else if(function_exists('bcpow')) { return bcmul( bcpow($cyclomatic_complexity, 2), ( bcpow( ( bcsub(1, $code_coverage/100) ), 3) + $cyclomatic_complexity ) ); } else { return pow($cyclomatic_complexity, 2) * (pow(1-$code_coverage/100, 3) + $cyclomatic_complexity); }

}

} [/php]

The above finds the Change Risk Analysis and Predictions (CRAP) index. The crap index reduces when the code coverage is 0% or 100%. Since it is easier and quicker, I added it in based on the web site. The last one uses BCMath, for fairness and accuracy. If you don't have BCMath extension, then it is

[php] $crapLevel = pow($cyclomatic_complexity, 2) * (pow(1-$code_coverage/100, 3) + $cyclomatic_complexity); [/php]

[php] // Ported from PHPUnit // http://www.phpunit.de/browser/phpunit/branches/4.0/PHPUnit/Util/Metrics/Method.php function getCCLevel($file) { $tokens = token_get_all(file_get_contents($file));

$ccn = 1;

foreach ($tokens as $i => $token) { if (is_string($token)) { continue; }

list ($token, $value) = $token;

switch ($token) { case T_IF: case T_FOR: case T_FOREACH: case T_WHILE: case T_CASE: case T_CATCH: case T_BOOLEAN_AND: case T_LOGICAL_AND: case T_BOOLEAN_OR: case T_LOGICAL_OR: { $ccn++; } break; } }

return $ccn; } [/php]

As you can see, the only difference between PHPUnit and my function getCCLevel() is that I'm inserting the entire file.

WordPress Crap Index By File

With 0% code coverage.

The files with CRAP index of 2 are files that include other files and only have a single if statement for including wp-settings.php. Some files, such as xmlrpc.php has internal functions and classes which would have to be removed for unit testing to provide the whole picture.

A lot of the WordPress root directory files and wp-admin directory files has presentation and business logic coupled. Because of this, the numbers are even more skewed. wp-admin files weren't included because of this. Most developers shouldn't mess with the wp-admin files.

WordPress includes enough hooks to where most developers won't be required to modify the core of WordPress, so these numbers mean nothing to plugin developers. The only exception, perhaps, are the undocumented hooks and filters, that would have to be tracked down. In which case, these numbers will paint a pretty good picture of the difficulty.

As with any metric, take what you will from them, but don't take them as a serious indicator. The Change Risk Analysis and Predictions is meant for methods (and probably functions also), so these are just for fun and shouldn't be (mis)used against WordPress as a whole.

The colors mean nothing and just for making the numbers look pretty.

Base of WordPress

C.R.A.P. File
2 index.php
2 license.txt
2 readme.html
2 wp-register.php
2 wp-config-sample.php
6 wp-atom.php
6 wp-commentsrss2.php
6 wp-pass.php
6 wp-rdf.php
6 wp-rss.php
6 wp-rss2.php
6 wp-feed.php
20 wp-blog-header.php
90 wp-links-opml.php
110 wp-cron.php
182 wp-comments-post.php
306 wp-trackback.php
702 wp-mail.php
2970 wp-settings.php
3660 wp-login.php
20,022 wp-app.php
39,006 xmlrpc.php

WordPress Includes

C.R.A.P. File
2 version.php
2 rss-functions.php
2 registration-functions.php
6 default-filters.php
12 feed-rss.php
20 feed-atom.php
20 feed-rdf.php
20 feed-rss2.php
30 locale.php
56 feed-rss2-comments.php
72 feed-atom-comments.php
72 vars.php
90 streams.php
210 l10n.php
380 compat.php
462 registration.php
600 category.php
600 plugin.php
650 feed.php
702 author-template.php
812 bookmark.php
1260 deprecated.php
1260 user.php
1332 cron.php
1560 gettext.php
1560 template-loader.php
1722 wp-db.php
1722 comment-template.php
2550 kses.php
2756 script-loader.php
3306 bookmark-template.php
4422 cache.php
4830 capabilities.php
5112 category-template.php
5852 post-template.php
6806 class-pop3.php
6806 theme.php
7832 link-template.php
9702 class-smtp.php
11990 pluggable.php
11990 rewrite.php
12210 formatting.php
14520 class-IXR.php
15006 rss.php
16770 comment.php
17556 class-phpmailer.php
20306 classes.php
24492 widgets.php
28730 taxonomy.php
28730 general-template.php
37442 class-snoopy.php
41006 functions.php
66306 query.php
95790 post.php

Total Crap Level: 598398

Sources

1. http://www.artima.com/weblogs/viewpost.jsp?thread=210575 2. WordPress Subversion Repository.