<p>Two days ago I released the first major version in almost 2 years for <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-web-client">shlink-web-client</a>, a web UI for <a target="_blank" rel="noopener noreferrer" href="https://shlink.io">shlink</a>, my self-hosted URL shortener.</p><p>This new version introduces several improvements to the project, and I thought it was worth writing a post to explain all of them in depth.</p><h3>New design</h3><p>The project has been using the same design almost since the beginning. It was clean, but it laked a bit of contrast and consistency in many sections.</p><p>Now the main background is light-gray, with components composed in white cards with a slight box shadow, that makes them easier to spot inside the app.</p><p>Also, the main area has now a max-width that prevents components from stretching too much on wide screens.</p><div class="row"><div class="col-md-6  "><img alt="Before" src="/assets/img/shlink-web-client-3/design-before.png" class="gallery-item"/><p class="text-center"><small>Before</small></p></div><div class="col-md-6  "><img alt="After" src="/assets/img/shlink-web-client-3/design-after.png" class="gallery-item"/><p class="text-center"><small>After</small></p></div></div><h3>Overview page</h3><p>Up until now, after connecting to a Shlink server, you went directly to the list of short URLs.</p><p>With this new version, you now see an overview page first, which displays some general stats, a quick short URL creation form, and the latest 5 short URLs.</p><p>From there, you can navigate to the full list of short URLs, with filtering capabilities, or go to the advanced short URL creation form.</p><div class="row"><div class="col-md-8 col-md-offset-2 "><img alt="Overview" src="/assets/img/shlink-web-client-3/overview.png" class="gallery-item"/><p class="text-center"><small>Overview</small></p></div></div><h3>Improved short URL creation form</h3><p>One of the sections with more room for improvement was the short URL creation form. It had all the inputs mixed, displaying just the &quot;long url&quot; one, with the option to un-collapsing the rest at will.</p><p>The new design groups the inputs by context, which makes it more clear. It also displays the inputs all the time, since the &quot;quick form&quot; is now located in the overview page.</p><div class="row"><div class="col-md-6  "><img alt="Before" src="/assets/img/shlink-web-client-3/creation-form-before.png" class="gallery-item"/><p class="text-center"><small>Before</small></p></div><div class="col-md-6  "><img alt="After" src="/assets/img/shlink-web-client-3/creation-form-after.png" class="gallery-item"/><p class="text-center"><small>After</small></p></div></div><h3>Enhanced visits page</h3><p>The visits page had a problem, it was trying to display too much information in a single place.</p><ul><li>Visits over time</li><li>Visits from every platform</li><li>Visits from every location</li><li>A list of visits as a table</li></ul><p>The new approach splits those in subsections with their own sub-route each (<code>/by-location</code>, <code>/by-context</code>, etc). This helps to render a smaller amount of charts at once, since they are a bit CPU greedy.</p><p>Also, only the visits from the last 30 days are loaded by default, making it load way faster for URLs with a lot of visits.</p><div class="row"><div class="col-md-6  "><img alt="Before" src="/assets/img/shlink-web-client-3/visits-before.png" class="gallery-item"/><p class="text-center"><small>Before</small></p></div><div class="col-md-6  "><img alt="After" src="/assets/img/shlink-web-client-3/visits-after.png" class="gallery-item"/><p class="text-center"><small>After</small></p></div></div><h3>Outer sections</h3><p>The app has some sections which are not related to a specific server, like creating/editing servers, or the settings page.</p><p>These sections have also been adapted to the new style, and they also have a max-width for wide screens.</p><div class="row"><div class="col-md-6  "><img alt="Before" src="/assets/img/shlink-web-client-3/create-server-before.png" class="gallery-item"/><p class="text-center"><small>Before</small></p></div><div class="col-md-6  "><img alt="After" src="/assets/img/shlink-web-client-3/create-server-after.png" class="gallery-item"/><p class="text-center"><small>After</small></p></div></div><h3>Date range selector</h3><p>A new component allows selecting date ranges either by relative time ranges (last 30 days, last 90 days...) or by selecting absolute start and end dates.</p><p>This component is used to filter both short URL lists and visits lists.</p><div class="row"><div class="col-md-6  col-md-offset-3"><img alt="Date range selector" src="/assets/img/shlink-web-client-3/date-range-selector.png" class=""/><p class="text-center"><small>Date range selector</small></p></div></div><h3>Domain selector</h3><p>Shlink has had multi-domain support for some versions, but changing the domain from the web client required to set the value manually, which is a bit error-prone.</p><p>This version introduces a new component which allows selecting from domains already used in the past (which is what you will want in most of the cases), preventing accidental typos.</p><p>It also allows setting new values in case you want to use a new domain that you never used before. This domain will then appear in the list the next time.</p><div class="row"><div class="col-md-6  "><img alt="Existing domains" src="/assets/img/shlink-web-client-3/domains-existing.png" class=""/><p class="text-center"><small>Existing domains</small></p></div><div class="col-md-6  "><img alt="New domain" src="/assets/img/shlink-web-client-3/domains-new.png" class=""/><p class="text-center"><small>New domain</small></p></div></div><h3>Other improvements</h3><p>The changes described so far were the major ones. However, v3.0.0 includes a few other minor improvements.</p><ul><li>Result and loading messages have been standardized: They are now more consistent and use the same set of components, which also ensures consistency in the future.</li><li>Error messages are now more descriptive: Instead of showing generic error messages when API calls fail, the actual error message is now always displayed, so that it&#x27;s easier to know what failed.</li><li>Dropped support for Shlink v1: All major versions come with some breaking changes that improve code maintainability. This one drops Shlink v1 support, which allows removing some conditional paths here and there.</li></ul><p>Other than the last point, Shlink v3.0.0 is mostly backwards compatible, so it&#x27;s safe to upgrade if you are already using Shlink v2.x</p><h3>Next steps</h3><p>Of course this doesn&#x27;t end here, and I have a lot of ideas to improve it even further.</p><ul><li>Dark theme: It&#x27;s what the cool kids do these days, and I have already done some experiments around it.</li><li>Shlink dashboard: Next gen web app for Shlink. You can read the whole reasoning in <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-web-client/issues/338">this issue</a>.</li><li>Filtering in lists added to URL: Many sections allow filtering lists, but the state is saved in memory, which means it gets lost if the page is refreshed, and it cannot be bookmarked. The intention is to make all filtering params to become part of the URL.</li><li>Exporting visits in CSV: It&#x27;s cool to see charts full of info, but sometimes admins need to feed other services with the data, and consuming the original API is not always an option. Exporting visits to a CSV file will be supported soon by shlink-web-client.</li></ul>

Released shlink-web-client v3.0.0

<p>Let&#x27;s not deny that <a target="_blank" rel="noopener noreferrer" href="https://github.com/features/actions">GitHub actions</a> are hitting hard in the tech community, and many projects are transitioning from other continuous integration systems like <a target="_blank" rel="noopener noreferrer" href="https://circleci.com/">Circle CI</a> or <a target="_blank" rel="noopener noreferrer" href="https://travis-ci.org/">Travis CI</a>.</p><p>Personally I have been using Travis a lot during the past years, for all my OSS projects, and I have gotten very used to it.</p><p>However, people seem to be &quot;abandoning&quot; travis in favor of GitHub Actions, for different reasons. &quot;It&#x27;s faster&quot;, &quot;I can run smaller tasks in parallel&quot;, etc.</p><p>In my case, I have moved a few very specific tasks there, where travis was becoming a bottleneck, but I still have mixed feelings about the very-verbose yaml-based config needed to configure a few GitHub Actions, compared to the more concise travis one.</p><p>In this article I&#x27;ll explain how I improved a travis build to run several smaller tasks in parallel jobs, without having to completely move away from it.</p><h3>Initial state</h3><p>This project&#x27;s travis config is relatively simple. It is used to build a JS project for the web browser, and it has to run the next tasks:</p><ul><li>Lint the code, to make sure it fulfils the project&#x27;s coding standards.</li><li>Run unit tests.</li><li>Run mutation tests.</li><li>Build the docker image, to make sure none of the new changes broke it.</li><li>Conditionally publish a release on GitHub, with a dist file attached to it, if the build is being run for a git tag.</li></ul><pre><pre><code class="language-yaml">dist: bionic

language: node_js

branches:
  only:
    - /.*/

cache:
  directories:
    - node_modules

services:
  - docker

node_js:
  - &#x27;12.16.3&#x27;

install: npm ci

before_script:
  - echo &quot;Building commit range ${TRAVIS_COMMIT_RANGE}&quot;
  - export MUTATION_FILES=$(git diff ${TRAVIS_COMMIT_RANGE:-origin/main} --name-only | grep -E &#x27;src\/(.*).(ts|tsx)$&#x27; | paste -sd &quot;,&quot;)

script:
  - npm run lint
  - npm run test:ci
  - npm run mutate:ci
  - docker build -t project:test .

after_success:
  - node_modules/.bin/ocular coverage/clover.xml

before_deploy: npm run build ${TRAVIS_TAG#?}

deploy:
  - provider: releases
    api_key:
      secure: &lt;key&gt;
    file: &quot;./dist/project_${TRAVIS_TAG#?}_dist.zip&quot;
    skip_cleanup: true
    on:
      tags: true
</code></pre></pre><h3>Problems with this approach</h3><p>This is more or less ok, but the last two commands in <code>script</code> can be relatively slow and can delay the whole process.</p><p>Also, specially the mutation tests one, which is currently always considered successful regardless the result, can make the build reach the maximum run time allowed by travis, making the whole build falsely fail.</p><blockquote><p>The reason for the <code>before_script</code> step is to determine which source code files changed, and run mutation tests on them only, in order to mitigate this.</p></blockquote><p>Other than this, the build config defines &quot;global&quot; customizations that are actually only used for some tasks:</p><ul><li>Running <code>npm ci</code> is not required for the <code>docker build</code> task.</li><li>Getting the <code>docker</code> service is not required by any of the other tasks.</li><li>Determining changed files is only used by the <code>mutation tests</code> task.</li></ul><h3>Making use of jobs</h3><p>One cool feature travis supports is the use of a job <a target="_blank" rel="noopener noreferrer" href="https://docs.travis-ci.com/user/build-matrix/">build matrix</a>, that lets you define multiple jobs that run in parallel as part of the same build, and make each one of them do unrelated tasks.</p><p>Using this feature it is possible to improve the <code>.travis.yml</code> file, making it look like this:</p><pre><pre><code class="language-yaml">dist: bionic

language: node_js

branches:
  only:
    - /.*/

cache:
  directories:
    - node_modules

node_js:
  - &#x27;12.16.3&#x27;

jobs:
  fast_finish: true
  allow_failures:
    - name: &#x27;Mutation tests&#x27;
  include:

    - name: &#x27;Lint&#x27;
      install: npm ci
      script: npm run lint

    - name: &#x27;Unit tests&#x27;
      install: npm ci
      script: npm run test:ci
      after_success:
        - node_modules/.bin/ocular coverage/clover.xml

    - name: &#x27;Mutation tests&#x27;
      install: npm ci
      before_script:
        - echo &quot;Building commit range ${TRAVIS_COMMIT_RANGE}&quot;
        - export MUTATION_FILES=$(git diff ${TRAVIS_COMMIT_RANGE:-origin/main} --name-only | grep -E &#x27;src\/(.*).(ts|tsx)$&#x27; | paste -sd &quot;,&quot;)
      script: npm run mutate:ci

    - name: &#x27;Build docker image&#x27;
      services:
        - docker
      install: skip
      script: docker build -t project:test .

    - name: &#x27;Publish release&#x27;
      if: tag IS present
      install: skip
      script: skip
      before_deploy: npm run build ${TRAVIS_TAG#?}
      deploy:
        - provider: releases
          api_key:
            secure: &lt;key&gt;
          file: &quot;./dist/project_${TRAVIS_TAG#?}_dist.zip&quot;
          skip_cleanup: true
          on:
            tags: true
</code></pre></pre><p>This approach introduces all these benefits:</p><ul><li>Now linting, testing, mutation testing and building the docker image run in parallel, making the build finish earlier.</li><li>The mutation testing step is allowed to fail, and thanks to the <code>fast_finish</code> flag, travis won&#x27;t even wait for it to finish, considering the build successful once the rest have passed.</li><li>Configs that are only meaningful for certain jobs are defined in the job itself, instead of globally.</li><li>The &quot;publish release&quot; job only runs conditionally if the build is for a tag, thanks to the <code>if: tag IS present</code> condition. (More info about <a target="_blank" rel="noopener noreferrer" href="https://docs.travis-ci.com/user/conditional-builds-stages-jobs/">conditional jobs</a> in travis).</li><li>As soon as any of the jobs fails, the whole build fails and you don&#x27;t have to continue waiting. Before this, if the third task failed, you still had to wait for the previous two to finish.</li><li>Each job can have a name, making it more clear in travis UI what tasks are being run:</li></ul><p><img src="/assets/img/travis-multiple-jobs/travis-jobs.png" alt="Travis CI jobs"/></p><h3>Conclusion</h3><p>The main purpose of this article was to showcase that travis still has a lot to give. Github Actions are definitely cool, but there&#x27;s usually no silver bullet for everything, and a combination with other tools might be more suitable.</p><p>Also, this allows people using travis to improve their builds without having to change everything. Then, once you have split your tasks, if you still want to move to Github Actions, it should be easier.</p><blockquote><p>If you are curious, this is the project in which I use this travis configuration: <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-web-client">shlinkio/shlink-web-client</a>.</p></blockquote>

Running several steps of a Travis CI build in parallel

<p>Asynchronous and non-blocking runtimes are pretty usual in many programming languages, as well as long-lived web apps that stay in memory and are capable of dispatching multiple HTTP requests without having to be fully bootstrapped every time. This has not been traditionally the case with PHP apps. </p><p>However, many projects are starting to get some adoption, that bring this long-lived approach to the PHP world.</p><p>The problem with this is that PHP developers are not used to this, and they tend to continue acting as they have always done. Also, libraries might have made some assumptions, making them not work as expected when used under this context.</p><p>Some of these projects are <a target="_blank" rel="noopener noreferrer" href="https://www.swoole.co.uk">swoole</a>, <a target="_blank" rel="noopener noreferrer" href="https://reactphp.org/">ReactPHP</a> or <a target="_blank" rel="noopener noreferrer" href="https://amphp.org/">Amphp</a>, to give some examples.</p><p>In this article, I will be focusing on the first one, and explain how to approach some of the pain points I have found after using it for a year and a half on an <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink">existing PHP project</a>.</p><h3>Introducing Swoole</h3><p>The main difference swoole has compared to other similar projects is that it&#x27;s written as a native PHP extension, and needs to be installed with <code>pecl</code>.</p><p>This is obviously a bit cumbersome, but the benefit is that, since it&#x27;s written in C++, it has a better handling of memory allocation, which is very important to achieve what it tries to do.</p><p>What swoole does is running a main process which bootstraps the PHP application once, and then keeps it in memory so that it can continue dispatching requests without having to load resources every time. This makes it really fast.</p><p>Those HTTP requests are handled by a set of child processes called &quot;web workers&quot;. Swoole forks the contents in memory for the main process to each one of the workers, so each worker has a &quot;copy&quot; of the application in memory.</p><p>Each worker is independent, but each one of them can only handle one request at a time, so you have to increase the number of workers if you expect a lot of concurrent traffic.</p><p>Other than web workers, swoole also has another set of child processes called &quot;task workers&quot;, which can be used to delegate long tasks at runtime to prevent web workers from getting blocked.</p><p>Here you can see a diagram of swoole&#x27;s architecture.</p><p><img src="https://www.swoole.co.uk/images/how-swoole-works.png" alt="How Swoole works"/></p><blockquote><p>More information can be found in <a target="_blank" rel="noopener noreferrer" href="https://www.swoole.co.uk/how-it-works">Swoole&#x27;s website</a>.</p></blockquote><h3>Problems to solve</h3><p>Now that we know how swoole works, let&#x27;s see some of the challanges I have faced in th past, which are usually not that ovbious when you are used to think in terms of &quot;blocking&quot; and &quot;short-lived&quot;.</p><h4>Database connections are kept open</h4><p>One of the first things you find out is that, since the application stays in memory, any database connection that has been opened by a worker (either web or task worker), will be kept open and reused in subsequent executions or HTTP requests served by the app.</p><p>This, which is in theory good, has a side effect, which is not obvious during development: <strong>the opened connection will eventually expire</strong>.</p><p>Since most of the popular database abstraction layers in PHP are coming from the times in which PHP was just a bootstrap-on-every-request technology, they never cared too much about expired database connections, as they would be opened again on next request.</p><p>However, when using swoole and the like, you have to implement some mechanism which takes care of reconnecting or gracefully recovering when this happens.</p><blockquote><p>I wrote an article explaining how I solved this on a middleware-based app served with swoole and using doctrine: <a href="/2019/11/04/how-to-properly-handle-a-doctrine-entity-manager-on-an-expressive-application-served-with-swoole/">How to properly handle a doctrine entity manager on an expressive application served with swoole</a></p></blockquote><p>Obviously, swoole (and also others) provides some tools to work with databases (like this <a target="_blank" rel="noopener noreferrer" href="https://www.swoole.co.uk/docs/modules/swoole-async-mysql-client">MySQL client</a>), which I&#x27;m sure do not have this problem (although, I haven&#x27;t personally tried this), but they are more limited than other well adopted libraries.</p><h4>In-memory caches don&#x27;t behave as expected</h4><p>It is a frequent practice to use local in-memory caches, like APCu, to improve applications performance in production.</p><p>Usually APCu is faster than other options, like redis or memcached, so it&#x27;s the choice when you only have one instance of your app running (meaning, you don&#x27;t have a cluster behind a load balancer).</p><p>However, even if you are running your app on just one server with swoole, because of the web worker system explained earlier, you actually have several copies of your app in memory, and each one of them will have a separated and not shared cache when using APCu.</p><p>This can lead to inconsistencies which are hard to debug, so you need to have it in mind.</p><p>Also, since the information that is being loaded is kept in memory anyway, you will not notice such a big improvement by using APCu or other in-memory caches.</p><blockquote><p>Notice that this problem does not happen when using distributed/centralized caching solutions, such as redis or memcached, as in that case, the caching technology is shared by all workers.</p></blockquote><h4>No need to optimize file including/requiring</h4><p>Many projects have complex configuration systems which are spread into several files. While this make it more maintainable, it also has a performance impact when all those files need to be loaded on every request.</p><p>Because of that, there are libraries like <a target="_blank" rel="noopener noreferrer" href="https://github.com/laminas/laminas-config-aggregator">laminas-config-aggregator</a> that can dynamically load config from many sources and, if some conditions are met, they can merge the result in a single bigger file that can be used in later requests.</p><p>In the case of swoole, as all the files will be kept in memory once loaded for the first time, there&#x27;s no need for this kind of optimization.</p><h4>Class autoloading happens once</h4><p>This point is very related with the previous one.</p><p>Composer offers a lot of optimization options so that class autoloading is faster in production. While you usually will still want to use some of those so that the first time a class is being loaded it is still reasonably fast, any subsequent hits will just use the class which is already in memory.</p><p>One of the composer flags you will probably don&#x27;t want use with swoole is <code>--apcu-autoloader</code>, because of the reasons mentioned two points above this one.</p><h4>Services should be truly stateless</h4><p>This is something that you should always do anyway, but it&#x27;s specially important when a service instance is kept in memory between HTTP requests.</p><p>Sometimes it&#x27;s easy to end up having some small piece of internal state on a service, where we save something as a side effect of a method call, to end up using it when another one of the public methods is invoked.</p><p>While this could seem harmless when the whole app is bootstrapped on every request, it can have very dangerous side effects when the app is served with swoole, because the instance will persist until another request is dispatched by the same web worker.</p><p>Just imagine you saved some user data on a service, and it ends up being served to another user on the next request. That won&#x27;t look good.</p><p>Because of this, make sure your services are truly stateless, and if for any reason that&#x27;s not possible, at least remember to have some mechanism in place that flushes the data just before finishing the request.</p><h4>Hot reloading for dev envs is not that good</h4><p>One thing we usually take for granted is that we can modify a PHP file and just by making a new request, the new code will be executed. However, when the code is kept in memory, it&#x27;s not that straightforward.</p><p>When you are developing a project which is served with swoole, you will have to restart the server every time to get your changes applied.</p><p>Swoole has a mechanism to do a light server reload, which is faster. You can find the docs in <a target="_blank" rel="noopener noreferrer" href="https://www.swoole.co.uk/docs/modules/swoole-server-reload">it&#x27;s website</a>.</p><p>The problem with this is that you need to implement the mechanism to invoke that when a file is changed, or use some library that does it for you.</p><p>Also, I have noticed that even calling that and seeing that the server is reloaded, does not always have the expected effect, and you have to still end up doing a hard reload. It&#x27;s something I don&#x27;t fully understand and I still need to investigate a bit further.</p><h3>Conclusion</h3><p>As it always happens when new technologies appear, it takes some time to get used to them, but I strongly suggest you to give a try to swoole or any other non-blocking runtime, because it makes your apps super fast.</p><p>Just remember to have all of the above in mind ;-)</p><p>That said, here you can find some libraries that will make your life easier when trying to serve existing projects with swoole, without having to couple with it and being able to continue using your favourite framework:</p><ul><li>IDE helper: <a target="_blank" rel="noopener noreferrer" href="https://github.com/swoft-cloud/swoole-ide-helper">https://github.com/swoft-cloud/swoole-ide-helper</a></li><li>Mezzio (formerly Zend Expressive): <a target="_blank" rel="noopener noreferrer" href="https://github.com/mezzio/mezzio-swoole">https://github.com/mezzio/mezzio-swoole</a></li><li>Symfony: <a target="_blank" rel="noopener noreferrer" href="https://github.com/k911/swoole-bundle">https://github.com/k911/swoole-bundle</a></li><li>Laravel: <a target="_blank" rel="noopener noreferrer" href="https://github.com/swooletw/laravel-swoole">https://github.com/swooletw/laravel-swoole</a></li></ul><p>Also, here you can find some more articles talking about this topic:</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://www.zend.com/blog/why-you-should-use-asynchronous-php">https://www.zend.com/blog/why-you-should-use-asynchronous-php</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://samsonasik.wordpress.com/2020/04/05/using-swoole-in-mezzio-application-with-sdebug/">https://samsonasik.wordpress.com/2020/04/05/using-swoole-in-mezzio-application-with-sdebug/</a></li></ul>

Considerations when working with async PHP runtimes like swoole

<p>Some of you probably know that I have a pet project in which I like to work from time to time. This project is a self-hosted URL shortener called <a target="_blank" rel="noopener noreferrer" href="https://shlink.io">Shlink</a>.</p><p>Shlink is built using expressive as the base framework for the HTTP-dispatching task. A while ago, an expressive module was released to officially support serving expressive apps with <a target="_blank" rel="noopener noreferrer" href="https://www.swoole.co.uk/">swoole</a>, and I decided to include it on Shlink.</p><h3>How does swoole work?</h3><p>Swoole is an asynchronous non-blocking I/O framework which works in a similar way as node.js, but for PHP apps.</p><p>It has been conceived in a way that the app stays in memory between requests, removing the bootstrapping step and making apps to be served much faster.</p><p>Swoole can serve as many simultaneous requests as you tell it to. For this it makes use of workers. Each worker can serve one request at a time, and every worker has a separate instance of the app loaded in memory (this is something I <a target="_blank" rel="noopener noreferrer" href="https://twitter.com/dominikzogg/status/1189403552399134720">learned recently</a>).</p><p>Because of this, you have to sometimes change a bit the way you design your code. For example, if a service needs to be stateful (it shouldn&#x27;t, but shit happens), you will have to remember that the state will persist between requests, and you will want to somehow reset it to avoid unexpected side effects.</p><h3>How does this affect the EntityManager?</h3><p>One of the main services that keeps this internal state in Shlink is the doctrine <a target="_blank" rel="noopener noreferrer" href="https://www.doctrine-project.org/projects/doctrine-orm/en/current/tutorials/getting-started.html#obtaining-the-entitymanager">EntityManager</a>.</p><p>Due to the fact that it was originally designed to be used on a classical web server + CGI context, assuming it would be recreated on every request, there are a few considerations to take into account:</p><ul><li>It implements the unit of work pattern. It internally keeps track of all the entities that have been created/changed until you decide to flush it.</li><li>If something fails, the <code>EntityManager</code> can get closed, which makes it unusable after that moment.</li><li>If the database connection expires, the <code>EntityManager</code> will not gracefully recover. Instead, it will fail, since it&#x27;s not aware of the connection not being usable, getting yourself in previous point.</li></ul><h3>How to workaround it</h3><p>I learned about these issues by try and error, by using it on the app and getting the failures. Now, I want to share how I workaround those issues.</p><p>The first one I faced was the one making the app unusable after the <code>EntityManager</code> was closed. In order to solve this I created a decorated instance (naming, an object implementing <code>EntityManagerInterface</code> which in turn was wrapping an actual <code>EntityManager</code>), which was making sure the wrapped instance was transparently recreated if at some point it was closed.</p><p>However, this caused some side effects, and I finally realized that it was better to make the <code>EntityManager</code> to have to be reopened from the outside, and only once per request (this is closer to how it would behave on a classical set-up).</p><p>That&#x27;s how I came up with the <code>ReopeningEntityManager</code> Shlink is currently using (as of v1.20.0):</p><pre><pre><code class="language-php">&lt;?php

declare(strict_types=1);

namespace Shlinkio\Shlink\Common\Doctrine;

use Doctrine\ORM\Decorator\EntityManagerDecorator;

class ReopeningEntityManager extends EntityManagerDecorator
{
    /** @var callable */
    private $createEm;

    public function __construct(callable $createEm)
    {
        parent::__construct($createEm());
        $this-&gt;createEm = $createEm;
    }

    public function open(): void
    {
        if (! $this-&gt;wrapped-&gt;isOpen()) {
            $this-&gt;wrapped = ($this-&gt;createEm)();
        }
    }
}
</code></pre></pre><blockquote><p>You can find the actual code in <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-common/blob/master/src/Doctrine/ReopeningEntityManager.php">shlink-common</a>.</p></blockquote><p>It just expects a factory that creates the actual <code>EntityManager</code>, and exposes an <code>open</code> public method which re-creates the wrapped instance if closed.</p><p>This method is called from a <a target="_blank" rel="noopener noreferrer" href="https://www.php-fig.org/psr/psr-15/">PSR-15</a> middleware which needs to be registered early in the middleware pipeline. It makes sure the method is called before the next middleware on the stack is invoked:</p><pre><pre><code class="language-php">&lt;?php

declare(strict_types=1);

namespace Shlinkio\Shlink\Common\Middleware;

use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Http\Server\MiddlewareInterface;
use Psr\Http\Server\RequestHandlerInterface;
use Shlinkio\Shlink\Common\Doctrine\ReopeningEntityManager;

class CloseDbConnectionMiddleware implements MiddlewareInterface
{
    /** @var ReopeningEntityManager */
    private $em;

    public function __construct(ReopeningEntityManager $em)
    {
        $this-&gt;em = $em;
    }

    public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
    {
        $this-&gt;em-&gt;open();

        try {
            return $handler-&gt;handle($request);
        } finally {
            $this-&gt;em-&gt;getConnection()-&gt;close();
            $this-&gt;em-&gt;clear();
        }
    }
}
</code></pre></pre><blockquote><p>Again, the code can be found in <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-common/blob/master/src/Middleware/CloseDbConnectionMiddleware.php">shlink-common</a></p></blockquote><p>This middleware also does two more things (this time, after calling the next middleware on the stack, and making sure it does it even if an exception is thrown).</p><ul><li>It first closes the database connection. By doing it so, we make sure the <code>EntityManager</code> will reconnect the next time it needs to interact with the database, and therefore, we avoid the &quot;expired connection&quot; issue.</li><li>Then it clears the <code>EntityManager</code> itself. This avoids memory leaks, and also prevents that any orphan/non-flushed entity &quot;transcends&quot; to the next request, solving the first problem.</li></ul><h3>Further considerations</h3><p>This approach has been working well so far, but it&#x27;s important to assume that the <code>EntityManager</code> was not really designed for this non-blocking kind of context.</p><p>You need to know that the worker making use of it will not be able to serve any request until the <code>EntityManager</code> finishes its job. Because of this you might want to increase the number of swoole workers.</p><p>Also, closing the DB connection on every request is not the most optimal approach. It would be perfect to have a connection pool that was able to recreate connections when expired, and to keep them open as long as possible for faster operations. However, the <code>EntityManager</code> was not designed for that, so it&#x27;s better to close it.</p><p>You also need a <code>callable</code> to pass to the <code>ReopeningEntityManager</code>. The way you solve this depends on your implementation and the dependencies you use.</p><p>In my case, in Shlink I make use of the <code>ServiceManager</code>, so I have a delegator factory which takes the actual factory and passes it to the <code>ReopeningEntityManager</code>:</p><pre><pre><code class="language-php">&lt;?php

declare(strict_types=1);

namespace Shlinkio\Shlink\Common\Doctrine;

use Psr\Container\ContainerInterface;

class ReopeningEntityManagerDelegator
{
    public function __invoke(ContainerInterface $container, string $name, callable $createEm): ReopeningEntityManager
    {
        return new ReopeningEntityManager($createEm);
    }
}
</code></pre></pre><blockquote><p>You can find the actual code in <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-common/blob/master/src/Doctrine/ReopeningEntityManagerDelegator.php">shlink-common</a>.</p></blockquote><h3>Conclusion</h3><p>I wanted to write this article as a recap for myself too, since I had to ask and investigate many times, and I couldn&#x27;t find a single source of truth explaining everything.</p><p>I hope that, if you ended up here, this article served you well.</p>

How to properly handle a doctrine entity manager on an expressive application served with swoole

<p>For a long time, I have been trying to include tests in every project in which I&#x27;ve worked on.</p><p>There are several types of automated tests (or what should actually be called <a target="_blank" rel="noopener noreferrer" href="https://abstracta.us/insights/guide-continuous-testing/automated-checks">automated checks</a>). From unit tests, integration and functional tests, to end-to-end tests.</p><p>Each one of them differs from the rest by the scope they try to cover. From small units of code which are tested completely detached from external dependencies, to a whole app which is tested as a black box like if an end consumer was using it.</p><p>I&#x27;m not going to deeply discuss the semantics behind this, or where an integration test ends and an end-to-end test starts, because it&#x27;s quite frequently a blurry line.</p><p>I want to focus on unit tests, and a frequently used practice (even in my own tests), which I have read (and felt) several times, it might not be the best approach.</p><blockquote><p>This article is not focused on any particular technology, since I have found this in a lot of different projects.</p></blockquote><h3>What&#x27;s a unit test</h3><p>According to <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Unit_testing">Wikipedia</a>, unit testing is a &quot;method by which individual units of source code [...] are tested to determine whether they are fit for use&quot;.</p><p>While this might look like a clear definition, it&#x27;s actually quite open because, what&#x27;s a &quot;unit&quot; in the context of unit testing?</p><p>Very often, in object oriented programming projects, people take a class as a unit to test, and create a unit test for every class. In other paradigms, like functional programming, you might consider a function as the unit to be tested, or a set of functions which you have decided to put together in a module.</p><p>This is not really important for the purpose of this article. What&#x27;s really important is to take into account that, in order to properly &quot;unit test&quot; a &quot;unit of source code&quot;, you have to do it by isolating it from any dependency of that unit.</p><p>One of the better ways to achieve this is by following the <strong><a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Dependency_inversion_principle">Dependency Injection Principle</a></strong>. Without it you will probably end up trying to hack yourself in order to detach a &quot;unit&quot; from its dependencies.</p><blockquote><p>You can find other articles about <a href="/tag/dependency-injection/">Dependency Injection</a> in my blog.</p></blockquote><h3>Replacing dependencies</h3><p>A very frequent practice when unit testing a unit of source code is to replace its dependencies with <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Test_double">test doubles</a>, in order to properly isolate every unit when testing it.</p><p>Test doubles are basically objects compatible with the dependencies of the subject under test, that are created just for the purpose of the test and have some useful capabilities, like capturing how they have been called, or what should they return when called.</p><p>When you use one of those test doubles to assert how they were called as part of the test itself, you are using them as &quot;spies&quot;.</p><h3>Over-trusting spies</h3><p>All this introduction and contextualization had a single purpose.</p><p>I find myself (and others) in the situation in which you prepare a test double to behave in a certain way, then you call your subject under test, and then assert that your expectations on how the test double was called are correct, by spying on it.</p><p>This sometimes feels wrong, since it seems you are testing the test double/s instead of the subject under test.</p><p>I have read many times that, in a unit test, we should not actually know the internal implementation of the subject under test, but only test that, based on certain input, the expected output is returned (so, trust its public API).</p><p>However, there are cases in which some function/method does not really have any output, and we can only verify everything is going right by spying on the test doubles.</p><p>So, at the end of the day, I have mixed feelings about this practice.</p><h3>Conclusion</h3><p>Sadly, I cannot state a clear conclusion. This is actually an open article, and I would love to get your point of view in the form of a <a href="#comments-box">comment</a>.</p><p>It would be great to see how others face this, if you use other kinds of test doubles instead of spies, or if you are just ok with the implications.</p>

Is spying on mocks a bad practice for unit tests?

<p>There&#x27;s no doubt that having tests in a project allows you to find potential bugs earlier and more easily.</p><p>Lots of OSS projects require a minimum code coverage in order to accept new pull requests from contributors, and proprietary projects also tend to have some sort of continuous integration workflow which requires certain metrics to be fulfilled in order to get builds passing.</p><p>However, the code coverage can lead to a false sense of security, which makes you think that if certain class has a 100% code coverage, it is also 100% bug-free.</p><p>This is not always true, since you could be calling a method and yet not being properly testing its output or its real behavior. The code coverage will mark it as covered, but you might introduce a bug and still have a green test.</p><p>This is where mutation testing comes in.</p><h3>Mutation testing</h3><p>According to <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Mutation_testing">wikipedia</a>, mutation testing involves applying small modifications to a software. These modifications are called mutations.</p><p>Then, if you pass your tests and any of them fails because of this &quot;mutation&quot;, then you have &quot;killed the mutant&quot;. This is what should happen in code which is covered by tests. If tests keep passing after applying this modification, then your tests covering that part of the project are not so good, and you should improve them to <strong>really</strong> test what they are covering.</p><p>So in other words, mutation testing is a quality assurance practice for your tests.</p><p>In PHP projects, there used to be one library which purpose was working with the mutation testing approach, <a target="_blank" rel="noopener noreferrer" href="https://github.com/humbug/humbug">humbug/humbug</a>. However, it is mostly abandoned now, and it has motivated the appearance of another project, <a target="_blank" rel="noopener noreferrer" href="https://infection.github.io/">infection/infection</a>.</p><h3>Infection</h3><p>Infection is a mutation testing framework for PHP.</p><p>It is easy to start using it in your project, since you just need to install it (<code>composer require infection/infection --dev</code>), and create a configuration file. Then, it uses your existing test suite in order to apply mutations to your code and check your tests.</p><p>It currently integrates with <a target="_blank" rel="noopener noreferrer" href="https://github.com/sebastianbergmann/phpunit/">phpunit</a> and <a target="_blank" rel="noopener noreferrer" href="https://github.com/phpspec/phpspec">phpspec</a>, but I suppose there will be compatibility with other testing frameworks in the future.</p><h4>Configuring infection</h4><p>It consumes an <code>infection.json</code> configuration file which could have a structure like this:</p><pre><pre><code class="language-json">{
    &quot;source&quot;: {
        &quot;directories&quot;: [
            &quot;module/*/src&quot;
        ]
    },
    &quot;timeout&quot;: 10,
    &quot;logs&quot;: {
        &quot;text&quot;: &quot;build/infection/infection-log.txt&quot;,
        &quot;summary&quot;: &quot;build/infection/summary-log.txt&quot;,
        &quot;debug&quot;: &quot;build/infection/debug-log.txt&quot;
    },
    &quot;tmpDir&quot;: &quot;build/infection/temp&quot;,
    &quot;phpUnit&quot;: {
        &quot;configDir&quot;: &quot;.&quot;
    }
}
</code></pre></pre><p>This is the configuration of one of my company&#x27;s projects. A modular application where every module is located inside the <code>module</code> directory and has its own <code>src</code> folder with source files.</p><p>This config file basically defines where the sources are located, where infection&#x27;s logs should be placed, a temporary folder which will be used instead of using system&#x27;s default tmp folder, and where is the <code>phpunit.xml[.dist]</code> file.</p><p>That&#x27;s all you need to get infection working.</p><blockquote><p>You can find detailed information of all the configuration options <a target="_blank" rel="noopener noreferrer" href="https://infection.github.io/guide/usage.html#Configuration">here</a></p></blockquote><h4>Running infection</h4><p>Now that we have the configuration file, we just need to run <code>vendor/bin/infection</code>, and it will execute our test suite and then apply mutations to our source files.</p><p>Then it will display the result, telling how many mutants have been killed.</p><p>Infection supports lots of mutators, like replacing <code>++</code> by <code>--</code>, <code>true</code> by <code>false</code>, changing method visibilities, return values, etc. You can find the whole list <a target="_blank" rel="noopener noreferrer" href="https://infection.github.io/guide/mutators.html">here</a>.</p><p>The command line tool supports a list of modifiers, some of which are very interesting. This is how I would recommend you to run it if you want to address the whole project.</p><p><code>vendor/bin/infection --threads=4 --min-msi=70 --only-covered --log-verbosity=2</code>.</p><ul><li><code>--threads</code>: It runs tests and mutators in parallel processes, so you can take advantage of all the cores in your machine. Set a value that makes sense your you. If you have 8 cores, maybe you want to set it with value 8.</li><li><code>--min-msi</code>: This flag makes the process fail if the number of killed mutants is below certain percentage. In this case we require 70% of mutants to be killed. Adapt the value to your needs, and try to gradually improve your tests and increase this value (MSI stands for <em>Mutation Score Indicator</em>).<br/>This flag is really useful in continuous integration environments.</li><li><code>--only-covered</code>: This makes infection address only covered code. We know mutants won&#x27;t be killed in uncovered code, so I think it makes no sense to apply mutators there.<br/>I prefer to separate the <em>required code coverage</em> metric from the <em>required mutation score indicator</em> metric.</li><li><code>--log-verbosity</code>: By default, infection logs all the mutations it has applied to your code. This could lead to a log with thousands of lines if the project is big, so I prefer to set the log verbosity to <strong>2</strong>, which makes it log only non-killed mutants.</li></ul><p>And that&#x27;s mainly it. You can now automate this task in your continuous integration pipeline, or just run it locally, but at least you will discover some parts both in your source files and your tests that can be improved.</p><h3>Troubleshooting</h3><p>While this could be enough for an introduction to infection, the title of this article says something about <strong>big projects</strong>, so I&#x27;m going to explain how we integrated infection in one of my company&#x27;s biggest projects and how we addressed some inherent problems.</p><h4>Already unstable</h4><p>The first problem while working with infection is that, at the moment of writing this article, it&#x27;s not in a stable version yet. That means you could find bugs.</p><p>We found one in which infection wasn&#x27;t properly parsing our phpunit config file and it was excluding almost the whole project from the code coverage report. This caused all mutations to be skipped.</p><p>We reported it and they fixed it very fast, but they haven&#x27;t tagged a version including the fix yet, so we require one specific commit from master while we wait for v0.8 to be released.</p><h4>Time consuming</h4><p>Another problem is that infection consumes a lot of memory and CPU, and takes a lot of time to be run when you have a lot of classes and tests.</p><p>For example, our project has around 1500 unit tests, which take about 1&#x27;5min-2min to be run. Our build process also passes some coding style checks and functional repository tests. It usually takes about 3 minutes.</p><p>The first time we run infection for the whole project, it took more than 16 minutes. That would make a build which is taking 3 minutes to take almost 20 minutes from now on, which is not an option because our build is run a lot of times every day.</p><p>Our solution was making the build run infection only on changed sources, taking advantage of infection&#x27;s <code>--filter</code> flag, which lets you pass a comma-separated list of files and it gets applied only to them.</p><p>The way in which we find which files have changed from previous build is by making a <code>git diff</code> between a commit from previous successful build and the new commit after updating the branch in which the build is being run (the commit identifiers are defined by jenkins&#x27; GIT plugin as environment variables).</p><p>If it is the first time the build is run for this branch, we make the diff with <code>origin/develop</code> instead.</p><p>The result is a bash script like this:</p><pre><pre><code class="language-bash"># Get files which have changed from latest processed commit, including only those inside sources
# If there&#x27;s no previous commit, diff with develop
INFECTION_FILTER=$(git diff ${GIT_PREVIOUS_SUCCESSFUL_COMMIT:-origin/develop} $GIT_COMMIT --name-only | grep /src/ | paste -sd &quot;,&quot; -)

# Check mutations over those files, if any, and require a 70% MSI
if [ -n &quot;$INFECTION_FILTER&quot; ]; then
    vendor/bin/infection --threads=4 --min-msi=70 --only-covered --log-verbosity=2 --filter=&quot;${INFECTION_FILTER}&quot;
else
    echo &quot;No source files affected. Infection skipped.&quot;
fi
</code></pre></pre><p>As you can see, we perform a diff by including only file names, then we exclude those which do not contain &quot;/src/&quot; as part of the name, and then we convert them into a comma-separated list.</p><p>With this approach, the build usually takes 1 extra minute in the worst case scenario, which can be assumed.</p><p>However, I&#x27;m sure the performance of this project will be improved as it grows and gets more stable.</p><h4>Test run duplication</h4><p>While previous approach filters the files where mutations are applied, it is still running all tests beforehand, which makes all tests to be run twice on every build.</p><ul><li>First, our own execution, which generates a code coverage report to be published later, and makes the build fail if tests do not pass.</li><li>Then, infection runs tests again, on its own &quot;conditions&quot; (they take your phpunit config file and generate a new one, which has some custom configuration entries).</li></ul><strike>If it were possible to pass infection an already generated code coverage, we could prevent this duplication, but, as far as I know, it does not support that.</strike> **It does now**.<p>Another solution would be being able to run only tests which affect changed sources, instead of running all tests suites, but phpunit does not allow to filter by a list of files, only a specific file or folder.</p><strike>This is the only problem for which we have not yet found a solution, so If you know any workaround, a comment will be very welcome :-)</strike><p><strong>Update 2018-02-22</strong></p><p>Following what Maks suggested in this <a href="#comment-3762900802">comment</a>, we have changed the process a little bit, and using a more recent commit from <code>dev-master</code>, we can take advantage of a feature which allows an existing code coverage to be passed to infection.</p><p>Now, our phpunit execution includes these two flags: <code>--coverage-xml=build/coverage-xml --log-junit=build/phpunit.junit.xml</code></p><p>And then, infection is executed with <code>--coverage=build</code>.</p><p>That&#x27;s it, no need to run tests twice anymore :-)</p><h4>Xdebug vs phpdbg</h4><p>Since infection needs access to a code coverage report, you need some tool which is capable of generating it.</p><p>People usually use xdebug, since it is the most extended and feature-rich debugging tool.</p><p>However, there&#x27;s a simpler tool, which is designed to debug php console executions only, and comes bundled with php. It is <a target="_blank" rel="noopener noreferrer" href="https://github.com/krakjoe/phpdbg">phpdbg</a>.</p><p>In our project, it usually takes 30%-50% less time to generate a code coverage report than xdebug. The result is not 100% the same, but it is very similar, and it&#x27;s worth it.</p><p>So, instead of running infection like this:</p><pre><pre><code>vendor/bin/infection --threads=4 --min-msi=70 --only-covered --log-verbosity=2 --filter=&quot;${INFECTION_FILTER}&quot;
</code></pre></pre><p>We have to do it like this</p><pre><pre><code>phpdbg -qrr vendor/bin/infection --threads=4 --min-msi=70 --only-covered --log-verbosity=2 --filter=&quot;${INFECTION_FILTER}&quot;
</code></pre></pre><p>It is explained in <a target="_blank" rel="noopener noreferrer" href="https://infection.github.io/guide/usage.html#Running-with-phpdbg">infection&#x27;s docs</a>.</p><p>We have even started to use it to run our own phpunit execution.</p><h3>Conclusion</h3><p>We have seen how to use infection, and how to integrate it in a continuous integration pipeline, even for <strong>big projects</strong>.</p><p>I think it is a very useful tool, and the team behind it is doing a great job. It is also not hard to start using it, since you don&#x27;t really need to write new code, just add a config file and run the CLI tool.</p><p>Finally, I recommend you to read this article, from infection&#x27;s author. It explains how it internally works, and how to use it in a development workflow <a target="_blank" rel="noopener noreferrer" href="https://medium.com/@maks_rafalko/infection-mutation-testing-framework-c9ccf02eefd1">https://medium.com/@maks_rafalko/infection-mutation-testing-framework-c9ccf02eefd1</a>.</p>

Mutation testing with infection in big PHP projects

<p><a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/">Composer</a> is <strong>The Tool</strong> in any modern PHP project. Nowadays I can&#x27;t imagine to work without it.</p><p>It is much more powerful than some people think, easily solving the integration of third party components in our projects, but there are some advanced features that are less known.</p><p>I&#x27;m going to try to explain some of the best practices and mechanisms bundled with composer.</p><blockquote><p>If you are not familiar with composer, some time ago I wrote an introduction article that you should read before continuing with this one. <a href="/2014/07/19/dependency-management-and-autoloading-in-php-projects-with-composer/">Dependency management and autoloading in php projects with composer</a>.</p></blockquote><h3>Globally installing composer</h3><p>It is a good practice to have composer installed globally, instead of having a binary file per project.</p><p>After downloading the <code>composer.phar</code> file from the <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/download/">composer</a> website, you can start using it by running <code>php composer.phar [...]</code>. Instead, we are going to place it in a global location so that we can run the command <code>composer [...]</code> from this moment on.</p><p>In UNIX systems, just make the file executable and move it to a folder that&#x27;s already in the path.</p><pre><pre><code class="language-bash">sudo chmod +x composer.phar
sudo mv composer.phar /usr/local/bin/composer
</code></pre></pre><p>Under Windows systems, there is an installer that will make the work for you. Download it <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/doc/00-intro.md#installation-windows">here</a>.</p><p>Now, go to any location and run <code>composer</code>. You should see composer&#x27;s help.</p><h3>Create the composer.json file</h3><p>The first thing you should do when starting a new PHP project is to create the composer.json file. It is the main configuration file, where metadata, dependencies and autoloading are defined.</p><p>You probably don&#x27;t remember the main parts of this file, and end up copying the file from othe projects. There is no need to do that, just run <code>composer init</code> and the Composer config generator will be executed, guiding you over the process of creating the file.</p><h3>Add dependencies to an existing composer.json file</h3><p>In the documentation of many libraries, they tell you to manually edit your composer.json file to add new dependencies. That can be problematic, because you could have a syntax error, forget to add a trailing comma, etc.</p><p>The best practice to add dependencies to your composer.json file is to run the command <code>composer require vendor-name/library-name ~3.0</code>. That will add the new dependency to your file (avoiding syntax errors) and then install it.</p><p>If you don&#x27;t include the version name, composer will try to find out what&#x27;s the best version to use, by following <a target="_blank" rel="noopener noreferrer" href="http://semver.org/">Semantic Versioning</a>. If it is not able to know which version to use, it will ask you to decide.</p><h3>Production environments</h3><p>The management of dependencies and new classes is not the same in our development environment and the production environment. In development we need new classes to be automatically autoloaded, and we want to have certain dependencies, like phpunit or php code sniffer that are not of any use in production.</p><p>In development we will usually install or update dependencies by running <code>composer install</code> or <code>composer update</code>. That will include all dependencies in <strong>require</strong> and <strong>require-dev</strong> blocks, and generate an autoloader by following different strategies. Some of those autoloading strategies imply the iteration of directories in order to find class files.</p><p>That is ok for development, but in production we need something more efficient than that, and also, we don&#x27;t want <strong>require-dev</strong> dependencies to be installed.</p><p>To get this done, in a deployment process, we should run this command instead.</p><pre><pre><code class="language-bash">composer install --no-dev --optimize-autoloader --prefer-dist --no-interaction
</code></pre></pre><p>This is what each flag makes:</p><ul><li><strong><code>--no-dev</code></strong>: prevents dependencies defined in the <strong>requre-dev</strong> block to be installed.</li><li><strong><code>--optimize-autoloader</code></strong>: generates a classmap autoloader, that will map each class with the file that contains it, preventing unnecessary directory iterations.</li><li><strong><code>--prefer-dist</code></strong>: installs distributable versions of the libraries, instead of source code, when possible.</li><li><strong><code>--no-interaction</code></strong>: will always try to make the proper decission in case any &quot;conflict&quot; is produced, so that the process is completly unatended.</li></ul><p>This is the command you should use when deploying a project, just after clonning a clean repository.</p><h3>Private repositories</h3><p>This is probably one of the most powerful features while working on many private projects that depend on each other.</p><p>By default, composer fetches dependencies from a single repository, <a target="_blank" rel="noopener noreferrer" href="https://packagist.org/">Packagist</a>, which needs to have access to a public repository (usually github). But what happens if my code is private, and I don&#x27;t want to (or can&#x27;t) open source it? May I use composer to install private dependencies? The answer is yes.</p><p>The composer.json file can include a <strong>repositories</strong> block, where we define other repositories (either public or private) where composer will try to find dependencies.</p><p>There are several repository types supported by composer, from a VCS repository to a plain local directory containing zip files.</p><pre><pre><code class="language-javascript">{
    &quot;repositories&quot;: [
        {
            &quot;type&quot;: &quot;vcs&quot;,
            &quot;url&quot;: &quot;https://my-repos.com/something-private&quot;
        },
        {
            &quot;type&quot;: &quot;pear&quot;,
            &quot;url&quot;: &quot;http://pear2.php.net&quot;
        },
        {
            &quot;type&quot;: &quot;artifact&quot;,
            &quot;url&quot;: &quot;path/to/directory/with/zips/&quot;
        }
    ]
}
</code></pre></pre><p>Composer will look for dependencies on each defined repository in order, and Packagist as a last resource.</p><p>Defining repositories is very flexible and a little bit complex in some cases, so I recommend you to read the <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/doc/05-repositories.md">documentation</a>.</p><h3>CLI scripts</h3><p>Sometimes a library includes some command line interface scripts that the user can use to automate configuring some stuff.</p><p>If you want to include your own CLI scripts and allow any user to find them when installing your library as a dependency, you will want to define them in the <strong>bin</strong> block.</p><pre><pre><code class="language-javascript">{
    &quot;bin&quot;: [
        &quot;scripts/generate-foo&quot;,
        &quot;scripts/create-bar&quot;,
        &quot;scripts/do-something&quot;
    ]
}
</code></pre></pre><p>It is just a list of script paths that composer will &quot;copy&quot; to the <code>vendor/bin</code> directory after installation (it doesn&#x27;t really copy them, it creates symlinks in UNIX systems and generates bat files under Windows).</p><p>This makes all the CLI scripts of all dependencies to be &quot;placed&quot; in the same location, so it is easier to find them.</p><p>This makes sense only in non-root projects (those that will be installed as dependencies of other projects and are not base applications).</p><h3>Events</h3><p>Composer will trigger some events during the process of installing dependencies, generating autoloaders and such. You can define some scripts or code snippets that will be automatically executed when those events are triggered.</p><p>These event listeners are defined in the <strong>scripts</strong> block. They can be either CLI commands or static class methods.</p><pre><pre><code class="language-javascript">{
    &quot;scripts&quot;: {
        &quot;post-package-install&quot;: &quot;Acelaya\\MyClass::postPkgInstall&quot;,
        &quot;post-install-cmd&quot;: [
            &quot;Acelaya\\MyClass::postInstall&quot;,
            &quot;phpunit -c app/&quot;
        ]
    }
}
</code></pre></pre><p>The complete list of events can be found in composer&#x27;s <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/doc/articles/scripts.md#event-names">documentation</a>. Take a look at them and see if you can optimize your workflow.</p><h3>Applications installation</h3><p>At this point we all know that we can use composer to install libraries at project level, but what happens with applications, known as <strong>root projects</strong> in composer&#x27;s terms.</p><p>We can also use composer to distribute complete applications, allowing them to be easily installed by using this command.</p><pre><pre><code class="language-bash">composer create-project vendor-name/app-name
</code></pre></pre><p>This will download the application (as if we would have cloned the repository) and then install its dependencies. We could even take advantage of composer&#x27;s events, and initialize a database or create some non-tracked directories.</p><h3>Conclusion</h3><p>And that&#x27;s pretty much all.</p><p>Use composer, it will improve your PHP projects.</p>

Composer advanced concepts

<p>One of the common problems we have to confront when starting a new PHP project is how to handle the different dependencies we are going to have.</p><p>One could think the easier solution is to download all the libraries we are going to need, put them in a <code>lib</code> directory and add them to the version control of the project, but this could be problematic. Let&#x27;s see why.</p><p>The main problems of using this practice are this.</p><ul><li><strong>Dependencies of dependencies:</strong> Our dependencies may have their own dependencies that we will also need to download.</li><li><strong>Updates:</strong> What happens if we need a new feature in the last version of one dependendency? We are going to have to update it manually, and that can be a hard task. Baybe even the new version has new dependencies of its own that we will need to download.</li><li><strong>Autoloading:</strong> How do we deal with autoloading the classes in the dependencies? Some of them may have its own autoloading system, but anyway, we are going to need to <code>include</code> all the autoloaders of each dependency.</li></ul><p>How do we solve all of this? The answer is <a target="_blank" rel="noopener noreferrer" href="http://getcomposer.org">Composer</a>.</p><h3>Starting with composer</h3><p>Composer has become the dependency manager in the main PHP projects. It uses its own library repository (<a target="_blank" rel="noopener noreferrer" href="https://packagist.org/">packagist</a>) to handle dependencies, but you can setup private repositories too.</p><p>It uses a small configuration file, the <code>composer.json</code>, to know which are our dependencies, and it will download their dependencies recursively without us having to do anything special.</p><p>We can define an autoloading strategy in that file too, and forget about that task for our project and our dependencies.</p><p>This are some examples of famous PHP project&#x27;s <code>composer.json</code> files:</p><ul><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/zendframework/zf2/blob/master/composer.json">Zend Framework</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/symfony/symfony/blob/master/composer.json">Symfony</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/doctrine/doctrine2/blob/master/composer.json">Doctrine</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/aws/aws-sdk-php/blob/master/composer.json">AWS PHP SDK</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/Seldaek/monolog/blob/master/composer.json">Monolog</a></li><li><a target="_blank" rel="noopener noreferrer" href="https://github.com/thephpleague/flysystem/blob/master/composer.json">Flysystem</a></li></ul><p>Now lets see a simpler composer file:</p><pre><pre><code class="language-json">{
    &quot;name&quot; : &quot;acelaya/zf2-acmailer&quot;,
    &quot;description&quot; : &quot;Mail sending module for Zend Framework 2&quot;,
    &quot;type&quot; : &quot;library&quot;,
    &quot;authors&quot; : [
        {
            &quot;name&quot; : &quot;Alejandro Celaya Alastrué&quot;,
            &quot;email&quot; : &quot;alejandro@alejandrocelaya.com&quot;,
            &quot;homepage&quot; : &quot;http://www.alejandrocelaya.com&quot;,
            &quot;role&quot; : &quot;Developer&quot;
        }
    ],
    &quot;keywords&quot; : [
        &quot;mail&quot;,
        &quot;zend&quot;,
        &quot;php&quot;
    ],
    &quot;homepage&quot; : &quot;https://github.com/acelaya/ZF2-AcMailer&quot;,
    &quot;license&quot; : [
        &quot;BSD&quot;
    ],
    &quot;require&quot; : {
        &quot;php&quot; : &quot;&gt;=5.3.0&quot;,
        &quot;zendframework/zendframework&quot;: &quot;&gt;=2.2.2&quot;
    },
    &quot;require-dev&quot;: {
        &quot;phpunit/phpunit&quot;: &quot;&gt;=3.7&quot;,
        &quot;squizlabs/php_codesniffer&quot;: &quot;1.*&quot;,
        &quot;phploc/phploc&quot;: &quot;*&quot;,
        &quot;pdepend/pdepend&quot; : &quot;1.1.0&quot;,
        &quot;phpmd/phpmd&quot; : &quot;1.4.*&quot;,
        &quot;sebastian/phpcpd&quot;: &quot;*&quot;,
        &quot;theseer/phpdox&quot;: &quot;0.6.5&quot;
    },
    &quot;autoload&quot; : {
        &quot;psr-0&quot; : {
            &quot;AcMailer&quot; : &quot;src/&quot;
        },
        &quot;classmap&quot; : [
            &quot;./Module.php&quot;
        ]
    }
}
</code></pre></pre><p>This is the composer file of my Zend Framework 2 module <a target="_blank" rel="noopener noreferrer" href="https://github.com/acelaya/ZF2-AcMailer">AcMailer</a>. Let&#x27;s explain it.</p><p>The properties <strong>name</strong>, <strong>description</strong>, <strong>type</strong>, <strong>authors</strong>, <strong>keywords</strong>, <strong>homepage</strong> and <strong>license</strong> are just metadata to provide useful information to other users and ease finding the project in <a target="_blank" rel="noopener noreferrer" href="https://packagist.org/">packagist</a>.</p><p>The <strong>require</strong> property is the place where we define our dependencies and the version we want to install. We can define here the minimum required PHP version and even the php modules we need (like <code>php-pdo</code>, <code>php-intl</code> and such).</p><p>The <strong>require-dev</strong> property allows us to define other dependencies that won&#x27;t be necessery in a production environment, like testing or documentation tools.</p><p>Finally the <strong>autoload</strong> property is where we define how our own classes are loaded. It is very easy to define a <code>psr-0</code> or <code>psr-4</code> autoloading strategy, but we can even define a <code>classmap</code> with a list of paths to files.</p><blockquote><p>You don&#x27;t have to remember the <code>composer.json</code> schema. You can interactively create it by running <code>php composer.phar init</code>.</p></blockquote><h3>Installing dependencies</h3><p>Once this is defined we just need the composer binary file (usually in phar format), that can be found <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/download/">here</a>, and run <code>php composer.phar install</code>.</p><p>This will create a <code>vendor</code> folder in the root of our project, download our dependencies there, the dependencies of our dependencies, and set-up an autoloader that will be able to load all our dependencies and our own classes as well.</p><p>After this it&#x27;s enough with a single <code>include</code> statement to <code>vendor/autoload.php</code> to be able to access any class.</p><p>If later in the project we need to update an existing dependency, add new dependencies or change the autoloading, we just need to update the information in the <code>composer.json</code> file and run <code>php composer.phar update</code> to update dependencies and autoloader.</p><blockquote><p>It is recommended to include the composer binary in the PATH to be able to run <code>composer</code> instead of <code>php composer.phar</code>.</p></blockquote><blockquote><p>If you are using a VCS like <a target="_blank" rel="noopener noreferrer" href="https://git-scm.com/">git</a> in your project you will probably want to <em>ignore</em> your <code>vendor</code> folder.</p></blockquote><h3>Advanced usages</h3><p>This should be enough to start working with composer, but it is a much more powerful tool with more features. It is not a bad idea to take a quick read to its <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/doc/">documentation</a>, but let&#x27;s see the main advanced features.</p><ul><li><strong>Classmap autoloader:</strong> By running <code>composer dumpautoload --optimize</code> we can generate a much faster autoloader which defines a class =&gt; file map array with all the classes handled by composer. This is the more suitable autoloader for production environments.</li><li><strong>Production dependencies:</strong> If we include de modifier <code>--no-dev</code> while installing or updating dependencies, it will ignore (or remove) the <strong>require-dev</strong> dependencies. For example <code>composer update --no-dev</code></li><li><strong>Private repositories:</strong> If you want to install dependencies from repositories other than <a target="_blank" rel="noopener noreferrer" href="https://packagist.org/">packagist</a>, you can do it by using the <strong>repositories</strong> property. It supports many types of repositories, from VCS to PEAR repositories. Complete documentation <a target="_blank" rel="noopener noreferrer" href="https://getcomposer.org/doc/05-repositories.md">here</a>.</li><li><strong>Global packages:</strong> If you want to install dependencies globally, just add the <code>global</code> keyword before the <code>install</code> or <code>update</code>. That will install the dependencies in the <code>~/.composer/vendor</code> directory instead of installing them in the local <code>vendor</code> directory. For example <code>composer global update</code>.</li></ul><p>And that&#x27;s all you need to know to start working with composer.</p><p><strong>Update 2015-04-25</strong>: I&#x27;ve written an article addressing advanced concepts that is a good continuation to this one. <a href="/2015/04/25/composer-advanced-concepts/">Composer advanced concepts</a>.</p>

Dependency management and autoloading in php projects with composer

<p>Dependency management is very important in any kind of project. Any application will need to use third party libraries to avoid to reinvent the wheel.</p><p>The problem is that getting each one of them from a different place takes time, and it&#x27;s not easy to be sure we get the correct version.</p><p>To solve this in java projects we can use an automated dependency manager, like <a target="_blank" rel="noopener noreferrer" href="https://maven.apache.org/">Maven</a>, which is really a building system with dependcy management included. The problem with maven is that it could be hard to use, and not very intuitive.</p><p>Instead of Maven, I rather prefer to use <a target="_blank" rel="noopener noreferrer" href="https://ant.apache.org/ivy/">Ivy</a>. It is a more simple and agile dependency management system, designed to be used with <a target="_blank" rel="noopener noreferrer" href="https://ant.apache.org/">Ant</a>, but it takes dependencies from Maven repositories, like <a target="_blank" rel="noopener noreferrer" href="https://mvnrepository.com/">Maven central</a>.</p><h3>Getting started</h3><p>To use Ivy we need to create an <code>ivy.xml</code> file in the project root directory. It will be used to define which dependencies are needed by our project, and will need to be downloaded at building time.</p><p>This is the <a target="_blank" rel="noopener noreferrer" href="https://github.com/acelaya/subclient-subconnector">Subconnector</a> Ivy script.</p><pre><pre><code class="language-xml">&lt;ivy-module version=&quot;2.0&quot;&gt;
    &lt;info organisation=&quot;net.subclient&quot; module=&quot;subconnector&quot; /&gt;

    &lt;dependencies&gt;
        &lt;dependency org=&quot;junit&quot; name=&quot;junit&quot; rev=&quot;4.10&quot; /&gt;
        &lt;dependency org=&quot;com.google.code.gson&quot; name=&quot;gson&quot; rev=&quot;2.2.2&quot; /&gt;
        &lt;dependency org=&quot;log4j&quot; name=&quot;log4j&quot; rev=&quot;1.2.17&quot; /&gt;
        &lt;dependency org=&quot;org.apache.commons&quot; name=&quot;commons-lang3&quot; rev=&quot;3.1&quot;/&gt;
    &lt;/dependencies&gt;
&lt;/ivy-module&gt;
</code></pre></pre><p>As you can see, it is very simple. We just defined an info element with the basic project information, and a dependencies element with nested information of each of the dependencies of the project.</p><p>For each one of them we have to know its organization, name and revision. With this we can be sure we will get the desired library and revision.</p><p>But how do we know what to put in <code>org</code>, <code>name</code> and <code>rev</code> attributes? The <a target="_blank" rel="noopener noreferrer" href="https://mvnrepository.com/">maven central repository</a> webpage will help us with this.</p><p>At the top search bar we can write the name of a library, for example gson. The first result corresponds to the gson artifacts in the maven central. By selecting the desired version we will get the dependency element for an Ivy script, as well as other dependency management systems, like Maven or <a target="_blank" rel="noopener noreferrer" href="https://gradle.org/">Gradle</a>.</p><p><img src="/assets/img/dependency.jpg" alt="Maven dependency example"/></p><p>We just need to copy and paste it. Nice and easy.</p><h3>Automate retrieving dependencies</h3><p>Once our Ivy script is ready, it is time to include retrieving dependencies in our building process. The easiest way to do it is to create an Ant target in our <code>build.xml</code> script, like this.</p><pre><pre><code class="language-xml">&lt;target name=&quot;retrieve-dependencies&quot;
        description=&quot;Retrieve dependencies with Apache IVY&quot;&gt;

    &lt;ivy:retrieve type=&quot;jar&quot; /&gt;

&lt;/target&gt;
</code></pre></pre><p>This will look for the ivy.xml script and download dependencies defined on it in a lib folder, but only binary jar files. If <code>type=&quot;jar&quot;</code> is removed, all docs, sources and binaries will be retrieved.</p><p>Make sure to add Ivy binaries to Ant classpath to make it properly work. The <code>project</code> element in the build script will need also to look like this.</p><pre><pre><code class="language-xml">&lt;project basedir=&quot;.&quot; default=&quot;build&quot; name=&quot;Subconnector&quot; xmlns:ivy=&quot;antlib:org.apache.ivy.ant&quot;&gt;

    ...

&lt;/project&gt;
</code></pre></pre><p>Notice the <code>xmlns:ivy</code> attribute.</p><p>Now we are ready to go.</p><p>For an advanced Ivy configuration take a look at its <a target="_blank" rel="noopener noreferrer" href="https://ant.apache.org/ivy/history/2.3.0/index.html">official documentation</a>. It explains how to use custom repositories, how to set up a proxy server, how to change default local lib folder, etc.</p>

Dependency management in Java projects with Ant and Ivy

<p><a target="_blank" rel="noopener noreferrer" href="http://nsis.sourceforge.net/Main_Page">NSIS</a> is a well known system used to create Windows installers for any type of application based on scripts.</p><p>It is good system. Once we have defined our script we can automatically build it to create application installers in minutes, including creation of Windows register entries, installation folders, application shortcuts, etc</p><p>The problem is that only Windows binaries are officially provided to compile this kind of scripts. This can be a problem in a <a target="_blank" rel="noopener noreferrer" href="https://en.wikipedia.org/wiki/Continuous_integration">continuous integration</a> environment, where continuous integration servers use to be Linux based.</p><p>At this article I am going to explain how to compile NSIS source code in Linux to get it working there and allow it to be invoked from automated tasks run in a continuous integration server.</p><h3>Download NSIS</h3><p>The first step is to download the windows distributable zip file and the source code. Both of them can be downloaded from their Sourceforge project: <a target="_blank" rel="noopener noreferrer" href="https://sourceforge.net/projects/nsis/files/?source=navbar">https://sourceforge.net/projects/nsis/files/?source=navbar</a>.</p><p>At the moment of writing this article the last stable version is NSIS 2.46, so we will need to download nsis-2.46.zip and nsis-2.46-src.tar.bz2. We will put them in a folder were users who are going to need it are allowed to access. I am going to put it at <code>/home/alejandro/nsis</code></p><p>Now that we have downloaded both windwos distributable package and source code we are going to decompress them creating two folders, nsis-2.46 and nsis-2.46-src.</p><h3>System requirements</h3><p>NSIS official documentation says we need this tools to compile it:</p><ul><li>Python, version 1.6 or higher. I am using Python 2.7.3</li><li>Scons version 0.96.93 or higher. This application can be downloaded from their official website <a target="_blank" rel="noopener noreferrer" href="http://www.scons.org/download.php">http://www.scons.org/download.php</a></li><li>A C compiler, in our case <code>gcc</code> and <code>g++</code></li></ul><p>All of this tools can be easily installed from repository, and probably we already have some of them.</p><p>The scons installer downloaded from their website comes in RPM format which could be a problem for systems not based in Ubuntu. For Debian I used the alien [<code>aptitude install alien</code>] tool, which converts RPM packages into DEB packages by running this command: <code>alien -k filename.rpm</code></p><p>Anyway, I uploaded the DEB resultant package for you to <a href="/assets/downloads/scons_2.3.0-1_all.deb_.zip">download</a> it.</p><h3>Compile NSIS</h3><p>Once all the requirements are clear we are going to compile the NSIS binary that is responsible for interpreting the scripts, the so called <code>makensis</code>.</p><p>We have to be at the source code directory, in my case <code>/home/alejandro/nsis/nsis-2.46-src</code>, and we execute this command.</p><pre><pre><code>scons SKIPSTUBS=all SKIPPLUGINS=all SKIPUTILS=all SKIPMISC=all NSIS_CONFIG_CONST_DATA=no PREFIX=/path/where/distributable/was/extracted install-compiler
</code></pre></pre><p>Notice that <em>install-compiler</em> is an argument of the command that is placed at the end, and is not part of the distributable path.</p><p>In my case, using my real path, the resulting command would be this:</p><pre><pre><code>scons SKIPSTUBS=all SKIPPLUGINS=all SKIPUTILS=all SKIPMISC=all NSIS_CONFIG_CONST_DATA=no PREFIX=/home/alejandro/nsis/nsis-2.46 install-compiler
</code></pre></pre><p>The execution of this command will compile the code, generating the <code>makensis</code> binary at the bin directory inside the distributable  package folder, in my case <code>/home/alejandro/nsis/nsis-2.46/bin</code>.</p><h3>Test makensis and create an installer</h3><p>Now that we have makensis created its a good idea to make it available from anywhere, for example adding it to a bin directory, and make sure it can be executed.</p><p>Using the paths from our example, running this commands will make the job.</p><pre><pre><code>chmod +x /home/alejandro/nsis/nsis-2.46/bin/makensis
ln -s /home/alejandro/nsis/nsis-2.46/bin/makensis /usr/local/bin/makensis
</code></pre></pre><p>Now it is time to test the command. Running <code>makensis</code> from anywhere should output something like this.</p><p><img src="/assets/img/nsis.png" alt="Nsis help"/></p><p>If we have an nsis script, runing the command makensis /path/to/script.nsi will compile the script, creating the installer. The syntax is the same used in windows.</p><p><strong>Atention!</strong> While running this command there could be this error.</p><pre><pre><code>Error: opening stub &quot;/home/alejandro/nsis/nsis-2.46/share/nsis/Stubs/zlib
Error initalizing CEXEBuild: error setting default stub
</code></pre></pre><p>This could happen because of a path error, but solving it is very simple, just go to the NSIS distributable folder, <code>/home/alejandro/nsis/nsis-2.46</code> in my case, and create the folder share [<code>mkdir share</code>], go to that folder an create a symbolic link running ln -s <code>/home/alejandro/nsis/nsis-2.46 nsis</code>.</p><p>That should fix the problem.</p><p>With this easy steps we have now a NSIS compilation system under a Linux machine ready to build and deploy Windows installers.</p>

Alejandro Celaya | Blog

Software development, agile methodologies and open source projects.

tag Archive — "continuous-integration"

— 2020 —

— 2018 —

— 2015 —

— 2014 —