Blog/ Drupal Testing Methodologies Are Broken - Here's Why

That's right! I said it. Currently available Drupal testing methodologies are broken. They are nowhere close to being enterprise-ready. All Drupal developers who have worked on long-term projects have felt this pain. Yes, Drupal 7 ships with Simpletest module. But there are two big problems with it:

Drupal 7 is not unit-test friendly

Although Simpletest supports unit testing, Drupal, especially custom code, which we really need to test, is not unit-test friendly. That's because unit testing requires that database not be intialized and all the input conditions be supplied as arguments to the function (what we call Dependency Injection in Symfony and Drupal 8). Since Drupal stores its configuration in the database, code assumes that the DB is available whenever it needs it. Drupal code also uses global variables such as $user and $base_url. Purists will say that we should write the code such that the DB connection or global variables are provided as arguments to the function as required. But then you are needlessly complicating the code. The other problem with unit testing Drupal's custom code is that almost everything happens in hooks. The input arguments to these hooks, such as hook_form_alter(), are deeply nested arrays and the hook changes one or two things in those arrays. If you want to unit test the hooks, you need to create such deeply nested arrays that you provide as arguments which the hooks can modify and that is really cumbersome.

As an example, consider a simple hook_form_alter() where we are changing the default value of the Greeting text field to "Hello" if the Article node we are editing has the highest nid among all the nodes in the system.

/**
 * Implements hook_form_FORM_ID_alter() for node_article_form().
 */
function mymodule_form_node_article_form_alter(&$form, &$form_state) {
  if ($form['#node_edit_form'] == FALSE) {
    return;
  }

  $nid = $form['#node']->nid;

  // Check if there is any other node in the system with a higher nid.
  $query = new EntityFieldQuery();
  $query->entityCondition('entity_type', 'node')
    ->entityCondition('entity_id', $nid, '>');
  $result = $query->execute();

  if (!isset($result['node'])) {
    // There is no node with a higher node id.
    $form['greeting'][LANGUAGE_NONE][0]['#default_value'] = t('Hello');
  }
}

It's nightmare to unit test this simple function. In fact, we can not unit test it because it uses a database query. We first need to refactor the function to the following:

/**
 * Implements hook_form_FORM_ID_alter() for node_article_form().
 */
function mymodule_form_node_article_form_alter(&$form, &$form_state) {
  if (!is_edit_form($form)) {
    return;
  }

  if (node_has_highest_nid($form['#node']->nid)) {
    modify_greeting_field($form);
  }
}

/**
 * Check whether the provided form is a node edit form.
 */
function is_edit_form($form) {
  return $form['#node_edit_form'];
}

/**
 * Check whether the node id provided is the highest node id in the system.
 */
function node_has_highest_nid($nid) {
  // Check if there is any other node in the system with a higher nid.
  $query = new EntityFieldQuery();
  $query->entityCondition('entity_type', 'node')
    ->entityCondition('entity_id', $nid, '>');
  $result = $query->execute();

  return !isset($result['node']);
}

/**
 * Set the default value of the Greeting field.
 */
function modify_greeting_field(&$form) {
  $form['greeting'][LANGUAGE_NONE][0]['#default_value'] = t('Hello');
}

You will notice that the number of lines of codes has increased from 21 to 39. Now we can unit test is_edit_form() and modify_greeting_field() functions like this:

/**
 * Test is_edit_form() function.
 */
function testIsEditForm() {
  $form = array();
  $form['#node_edit_form'] = TRUE;
  $this->assertTrue(is_edit_form($form), "A node edit form is misinterpreted as not not an edit form.");

  $form['#node_edit_form'] = FALSE;
  $this->assertFalse(is_edit_form($form), "A form which is not an edit form is misinterpreted as edit form.");

  unset($form['#node_edit_form']);
  $this->assertFalse(is_edit_form($form), "A form which is not an edit form is misinterpreted as edit form.");
}

/**
 * Test modify_greeting_field() function.
 */
function testModifyGreetingField() {
  $form = array();
  modify_greeting_field($form);
  $this->assertEqual("Hello", $form[LANGUAGE_NONE][0]['#default_value'], 'Greeting field value is not set to "Hello".');

  $form[LANGUAGE_NONE][0]['#default_value'] = t('Random');
  $this->assertEqual("Hello", $form[LANGUAGE_NONE][0]['#default_value'], 'Greeting field value is not set to "Hello".');
}

Even after all these extra lines of code, we are essentially testing less than half of the original code. Even in that, we are just testing array manipulation logic. The EntityFieldQuery() logic, which is the most important logic in this function, is not being tested at all. No wonder that most Drupal developers don't write automated tests and most clients don't want to pay for it.

Simpletest module requires reconfiguring the site from scratch

While doing functional tests, Drupal's simpletest module creates separate environment and database tables for each test and doesn't use the existing site configuration. In addition to being slow, this approach requires that before each functional test, you configure the site in code to replicate the existing site that is in production. This is nuts!!!! As an example, if you have a custom content type "Student" with 20 fields, you will have to create the content type and all those 20 fields in code before you start your functional tests. That's a lot of extra code! Yes, you can use Features for creating the content types with fields and you can enable that feature module before you start the functional test. But then you are managing a lot of feature modules for your site. You should try resolving conflicts in features modules if multiple developers are working on your code. It's a lot of fun! :) And to top it all off, not all the configurations work with Features so at some point you will have to write code to configure the site before you start the functional tests. This is not the most difficult problem to solve though. Some developers have tried modifying Simpletest so that it uses either the existing dev database or creates a new test database by uploading an SQL dump. I have not tried them but even if this issue is resolved, the issue of difficulty of writing unit tests for Drupal 7 still remains.

So what's the solution?

Forget about writing tests. Just kidding! :) As explained earlier, unit-testing won't cut it for Drupal because of how Drupal is written. We need to consider integration and functional tests. There are some good functional testing tools that work with Drupal such as Behat, Sahi and Selenium. We actually use these and they work pretty well. The problem is that functional tests are very slow because they use the browser. Even PhantomJS, a headless browser without the GUI, is much slower than unit tests or integration tests. This may not be because PhantomJS is slow, but because every single request, Drupal needs to do a complete bootstrap. For a medium complexity site, if you write all your tests as functional tests, it can easily take between 2 to 3 hours to complete one round of automated testing. That's agonizing! There has to be something better.

That's where integration tests come in. Integration tests use the database but not the browser. Although integration tests do need to bootstrap Drupal, they can be made to do it only once for the complete test suite. As a result, they are slower than unit tests but much faster than functional tests. Since integration tests have access to the database, developers don't have to modify their code to make it unit-testable. Here are some characteristics of an integration testing framework that will interest Drupal developers:

1. It needs to work with the existing site and database. Developers shouldn't need to configure their site in code before they start the tests.
2. It should help in reducing the time it takes for developers to write and maintain tests. This implicitly means that the test code should be readable and not too long.

Agreed that integration tests won't be able to test the site layout or JS. For these tests, we have to use functional tests but for other tests where we are testing business logic or site structure, integration tests are a much better solution that functional tests.

Since little over a year, we have been creating such a framework and using it with our large Drupal development projects. It is created specifically for Drupal and it understands Drupal so a lot of mundane background tasks of setting up the tests and the content types with fields are done automatically. Developer just has to concentrate on writing the business logic for the test. We are already on the fourth iteration and we feel that it's good enough and useful enough for general public. We have decided to make it open source so that the community can benefit from it and the brand of Drupal as a quality web development platform increases. Follow us on Twitter and LinkedIn or sign up below for our newsletter to know as soon it's released as Open Source!

EDIT: The integration test framework, named Red Test, is open-source now. If you have 15 minutes, start by writing your first integration test!