My time was pretty much cut short before I was able to fully talk about all that I wanted to talk about, so I’ll apply more clarity to certain parts that may need it. To be honest, I didn’t think I would have enough material to cover the full hour, but from how it went it appears that I could have gone on for two hours and still not completely covered the full scope. I could have also done without the user testing and focused completely on the automated testing portions, since the majority of the people who weren’t developers left anyway.

I was probably better off having a smaller audience, because I wasn’t as nervous. I think I did well, once I talked past the introduction and became comfortable with speaking to the audience. I’m going to have to watch the video to make sure that is the case and I will update this post if there are any other parts I need to explain better.

The WPTests Tool

I think the first part is what my tool, WPTests is and what it does, and what it is meant to do. The WPTests tool is to check the status of the Automattic WordPress Tests without having to download Subversion, set up Subversion, set up the config, and fix other issues. I don’t include a readme or help, and I only fix one issue that I have.

My intention was to send Alex my patches for the Automattic WordPress Tests and then my issues would be fixed and so would others that use my system or just download that system. I also wanted a tool which would work on Windows and allow me to record the progress. I do need to optimize the SQL queries in order to increase the performance or at least create pages based on dates. This would allow for getting all of the test runs in a single day to be displayed and then have the last test run for that day for all of the other days.

The tool provides a history, to give context to what the failures mean. If all of a sudden the tests all fail, then you know you shouldn’t upgrade. The problem is that the test runs are only once a hour, two minutes past the hour, so an issue might be fixed within that time, but the person would have to wait an hour or slightly over an hour before they would be updated.

It would also be great to have historical data on what tests failed and when and when the test passed. I like the tool, but I have plans for another, which are more personalized for the user, which will allow for choosing which test to run, which plugins to run, different options that the Automattic WordPress Tests supports and finally see the results of that test run.

Both tools are based on the same code, so it isn’t difficult to maintain both and I do plan on maintaining both. The tool I presented at the session was mainly to prove to Nikolay of Automattic of my intentions and what I wanted, which is something akin to continuous integration, but not exactly.

The tool can be download at wordpress.svn.dragonu.net/WPTests and you will need Subversion to do so. I do not have tags, so you just check out the root and you are ready to go. I do still plan on giving this to Alex and hope that he accepts at least part of it in to the Automattic WordPress Tests.

My WPTests tool does not support PHP4, you must have PHP5.1+ and PDO_SQLITE extension enabled to run the tool. You can however change the DSN and since all of the SQL is supported by MySQL also, you shouldn’t have issues. What you will have issues with, if you switch over to MySQL is that I use the TEXT field type, so a lot of space will be taken up. I’ll change that later, so that both are supported at a later date.

Amount of Test Cases for a Function or Method

The one failed test case was not the only test case covering that method. If I did not have that test case, then I would not have been able to figure out that my method was not working correctly. For that method I had four test cases covering about 3 lines of code (I’m being serious). For the entire class, I had a total of 15 test cases covering 4 or 5 very small methods.

When you write test cases, you want to write at least one test for valid input and at least one test case for invalid input. For each branch (if statement, while loop, for loop, anything that might require brackets) needs to have at least one test devoted to just that.

So all of my test cases covered almost all of the possible cases that a user will be using my methods for. I also thought ahead and thought of twists to make sure that my assumptions weren’t wrong and then I’ll have issues down the road. The failed test case was an assumption I had that I was unsure about. I was providing support for PHP4 and I can’t exactly test with PHP4, because of the way I have the PHPUnit library set up and what extensions I require. In theory, it should have worked perfectly, however that was not the case.

It is perfectly okay to write just one test case per function, but do realize that with one test case for one function, it might give false impressions over the reliability of that function. A function having one test case is only useful if that function only has one line and is calling another function and that is all it does.

You also want to split assertions made in a test case into multiple test cases, if you have more than one assertion in a test case. The reason is that, once there is a failure of an assertion in a test case, no other assertions are tried in that test case. If you have 5 assertions in a single test case and the test case fails after the first one, then the other 4 will not be reported as failures or even show up in the total. If it makes sense, move the other 4 assertions to their own test cases. If the assertions are dependent on each other, then by all means include them in the same test case.

Skipped and Incomplete Tests

I had to skip over the explanation of both the skipped and incomplete tests and how to do that. There is an issue I have with the current Automattic WordPress Tests automated test suite, in which test cases exist for a nonexistent function. The solution to skip over that test is to set a constant to false. That is not the correct way to solve that problem, because I want to run the other known bugs and you should always run the known bugs no matter what.

The correct way it should have been done is to test for whether that function exists and if it doesn’t, then mark that test case as skipped.

Skipped tests aren’t so much a problem as failed tests and errors, because it just means that whatever the test cases was using is not supported and it can’t run without it. If that component is important, then it probably should bring down the entire test suite.

Incomplete tests are test cases, which the developer started, but did not yet finish. So you can use this to your advantage, you can design all of the test cases you want for a component, mark them as incomplete and then go through and finish them all later. I would just as rather not do that, because you shouldn’t have a lot of incomplete tests and mostly it doesn’t take long to look through a function and write test cases for that function or method.

What the Number of Failed Test Cases Means

I don’t think the explanation I gave matched what I wanted to say. If there is a failed test case, then you should assume that there is a problem. That is until you look at both the test case and the function or method it is testing. If you find that the problem is the test case, then the test case should be fixed. If you find that it is the function or method that is the problem, then it is a bug and needs to be filed. However, both the test case and what it tests should be looked at.

What I said, was basically, the formatting test suite covers a full range of test cases. Not all of those test cases are perfect. Some are feature requests, which haven’t been filed and probably won’t be fixed. Therefore, you can safely ignore those. There are also other problems, where the actual test case is wrong and needs to be fixed. There are or were other test cases which actually found bugs in formatting functions, but it has been a while, so I don’t know if they have actually be fixed or looked at.

It can also be assumed that unless the entire automated test suite is brought down in a fatal error, that you can successfully upgrade WordPress. It does mean that there will be issues with bugs or regressions, but those should be bought to the developers attention.

Names of the Methods Should Mean Something

I’m in the camp that method names should mean something for that test. Meaning I should be able to read the test case method name when it fails and get an overview of what happened before I actually look at the test case and function to see where the problem is. It is a convention and with any convention it is up to the developer. If you want to write the function name and just have test_function_name2() and test_function_nameN(), then by all means do so.

What is important is not the method name of the test case, but that the test case exists. I will not fault you for naming it how Alex does it in the Automattic WordPress Tests. Alex might even change my way of naming the test case method names to something he likes. I am not an elitist on the subject, even though I did slightly mock Alex during the Session, even if I didn’t purposely mean to. My purpose was only to question, which way you would do it and finally advocate mine.

You can name it any way which the test case name makes sense. Either way, I’ll be happy if you just do it.

A List of “Thank Yous”

1. Charles Stricklin – I wanted to thank him for allowing me to speak at the Fisco, Dallas WordCamp 2008, even though pretty much no one there knew who I was and for giving a really geeky presentation.
2. I want to thank Branson Tourism Center and Branson.com for sponsoring my trip down there. For my work to do that, is awesome. I don’t think some of the other places I’ve worked out at would had extended their hand (and wallets) in that way. Larry and Lianne are really great people to work for and I would say that, even if they fired me tomorrow and say that years after I do finally leave. Of all of the people I’ve worked for, there were very few who I actually appreciated working for and would do anything for. It isn’t just that they are nice, they are, but it is the respect and that is the most important when I work for a company.
3. I would like to thank the city of Frisco, Texas for allowing WordCamp Dallas 2008 to be held there and appreciate the technician who ran the cameras to record all of the sessions. That was really great of the city to allow all of us to use their facility for the event. The city is just really beautiful.
4. Everyone else who stayed for my session, thank you. I also thank all of the people who I talked to and told me about theirselves.

To Be Continued…

This post will be updated at a later time, when the video is available and I can go over more of what I missed and didn’t fully explain.

Possibly Related Posts:

• WordPress Plugin and Theme Scope
• What I’ve Learned From WordPress
• Why Do I Feel Like an Asshole When Criticizing WordPress?
• PHP5 Straw Man Argument
• WordPress 2.7 Changes Class for Plugins