Capturing remote code coverage in API tests with PHPUnit

12 February 2022 — Comments

You can find an improved version of what’s described here in Capturing remote code coverage in E2E tests with PHPUnit

Capturing code coverage for a test suite is a very useful way to know which parts of your source code are actually getting executed by tests.

This is useful not only to know if you need to add more tests to your project (in case the coverage is too low), but also, if you want to apply technics like mutation testing, having a properly generated code coverage report will dramatically speed up te execution and help you know if a non-killed mutant is relevant.

Generating the code coverage

Capturing code coverage is usually easy in the most common types of tests, like unit tests, where the source code is executed in the same process.

Tools like PHPUnit, one of the most extended PHP testing frameworks, ships with tools like php-code-coverage, designed to capture the code under execution, just by providing a couple of lines of configuration.

However, generating a reliable code coverage report is not always that easy for other types of tests.

In contexts like E2E tests, where you may spin-up an API on a different process, and make your tests just call this API via HTTP and assert on the responses, PHPUnit will not be able to capture the code coverage, because the source code is actually being loaded on a different process.

Remote code coverage

A couple of weeks ago, I faced the issue described above. I had a web service for which I had a decent API test suite, but I wanted to capture code coverage for two main reasons:

  • Be able to merge this coverage with the one generated by other tests suites, to know what’s the “total” code coverage, regardless the test producing it.
  • Apply mutation testing to the API tests, as I was already doing that for the rest of the test suites in the project. For this, I also needed to know exactly what tests were covering every line of the source code.

After some investigation, I couldn’t find a built-in feature, where PHPUnit itself was able to “remotely” capture te code under test, so I had to think on a way to manually use php-code-coverage by myself.

Code coverage reports

In order to be able to capture code coverage, I had to make sure the instances of the objects provided by php-code-coverage where “spin-up” with the server, and then, the code coverage report was dumped at the end.

Also, you usually define the code to include/exclude from coverage in PHPUnit’s config, but since this time we are the ones creating the coverage objects, this was not possible.

I tried a couple of different approaches (more on this at the end), but the most accurate one for me, was to define two routes in my server, that were dynamically configured only when running the server in the context of these tests, and were responsible for creating and dumping the code coverage.

use PHPUnit\Runner\Version;
use SebastianBergmann\CodeCoverage\CodeCoverage;
use SebastianBergmann\CodeCoverage\Driver\Selector;
use SebastianBergmann\CodeCoverage\Filter;
use SebastianBergmann\CodeCoverage\Report\Html\Facade as Html;
use SebastianBergmann\CodeCoverage\Report\PHP;
use SebastianBergmann\CodeCoverage\Report\Xml\Facade as Xml;

// Define directories containing the code you want to cover
$filter = new Filter();
$filter->includeDirectory(...);
$filter->includeDirectory(...);
$coverage = new CodeCoverage((new Selector())->forLineCoverage($filter), $filter);

// This is "pseudo-code". Defining the routes depends on the framework you use
$app->get('/api-tests/start-coverage', function () use (&$coverage) {
    $coverage->start('API tests');
    return new EmptyResponse();
});
$app->get('/api-tests/stop-coverage', function () use (&$coverage) {
    $basePath = '...'; // Wherever you want the reports to be dumped
    $coverage->stop();

    // I generated coverage reports in a couple of different formats
    (new PHP())->process($coverage, $basePath . '/coverage.cov');
    (new Xml(Version::getVersionString()))->process($coverage, $basePath . '/coverage-xml');
    (new Html())->process($coverage, $basePath . '/coverage-html');

    return new EmptyResponse();
});

Then, I added the next snippet to my PHPUnit bootstrap.php file, to make sure the endpoints configured above are invoked at the proper times:

// when starting up the test suite, call the endpoint to start collecting coverage
$httpClient->request('GET', 'http://localhost:8080/api-tests/start-coverage');

// When the tests process shuts down, call the endpoint to stop the coverage
register_shutdown_function(function () use ($httpClient): void {
    $httpClient->request('GET', 'http://localhost:8080/api-tests/stop-coverage');
});

Learn more about how to configure PHPUnit’s bootstrap script.

This was fine from a pure code coverage point of view, but if you try it yourself, you will notice the report says all the code has been covered by API tests, which is what we provide when calling $coverage->start('API tests');

Invalid API coverage

To properly do mutation testing we need to know which are the exact tests that cover every part of the code. That requirement took me to the next approach.

Supporting mutation testing

To be more precise with the generated reports, I needed some way to notify the server about the specific test performing the request.

One way to do this is by sending a custom header when performing a request from your tests, and registering some event handler or middleware which starts the coverage for that request with the value coming from that header.

With that, I evolved the configuration above, and added a middleware handler (the framework I use is PSR-15 based, but the framework of your choice will probably have a way to do this).

use ...

$coverage = new CodeCoverage();

// This ensures the coverage is started and stopped on every request, using the "x-coverage-id" header as the ID.
$app->middleware(function (ServerRequestInterface $req, RequestHandlerInterface $handler) use (&$coverage) {
$coverage?->start('API Tests');
    $coverage->start($req->getHeaderLine('x-coverage-id'));

    try {
        return $handler->handle($req);
    } finally {
        $coverage->stop();
    }
});

// This route is no longer needed. You can also remove the request from your bootstrap script
// $app->get('/api-tests/start-coverage', ...);
$app->get('/api-tests/stop-coverage', ...);

Then, inside your test you can do something like this:

use PHPUnit\Framework\TestCase;

class MyApiTest extends TestCase
{
    private ClientInterface $httpClient;

    public function setUp(): void
    {
        $this->httpClient = ...;
    }

    public function testMyEndpoint(): void
    {
        $resp = $this->httpClient->request('POST', '...', [
            'headers' => [
                'x-coverage-id' => static::class,
            ],
        ])

        // Asserts go here...
    }
}

After this, the code coverage reports will properly show the actual tests covering them.

Valid API coverage

Alternative approach

The approach explained here is functional, but it’s a bit hacky. I had to do it like this because of limitations with how I serve the API with openswoole.

However, instead of defining routes that need to be actively called to start/stop the code coverage (or just to dump the report, as in the second approach), you could try to register the shutdown function directly in the server side.

If that works for you, it’s much more straightforward, and you won’t need to bother changing your tests bootstrap script.

Conclusion

Even though there’s room for improvement, this allows to generate code coverage reports for code that’s running on a different process.

I plan to apply something similar for E2E tests for a command line app, so maybe I extend this article at some point.

Also, I decided to write this article, because at the moment of having to implement this myself, I didn’t find too much useful information out there.

I hope this helped you, and now you can collect code coverage reports on your API tests.