A couple of weeks ago I migrated a React project using redux to <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/">Redux Toolkit</a>.
Redux has always been my preferred state management tool for React (which doesn&#x27;t mean I &quot;love&quot; it), but one of its main problems is the amount of boilerplate code it requires.
Because of that I had a couple of helper functions, but there was still some things not properly handled, like proper type inference on reducers and actions (I use TypeScript).
A bit by accident (I think it was on Twitter), I found out about Redux Toolkit (RTK from now on), an official library from the redux team, which provides some opinionated helpers to reduce the amount of boilerplate, and I decided to give it a chance.
These are my impressions.
<h3>TL;DR</h3>
I first started to use it in a proof of concept, and I quickly loved it.
RTK does not &quot;replace&quot; any of the redux principles, so you can progressively adopt it, by replacing small bits of code. My plan was to use it just for some reducers, but I liked it so much that I ended up migrating the whole project in a series of consecutive pull requests.
<h3>A bit of context</h3>
Before the migration, I used to follow the <a target="_blank" rel="noopener noreferrer" href="https://github.com/erikras/ducks-modular-redux">ducks modular pattern</a>, which promotes putting all your action types, action creators and the reducer handling those actions on the same module.
This fits pretty well with RTK, as in most of the cases you will use <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/api/createSlice">slices</a>, and end up exporting the reducer and action creators returned by it, from the same module.
If you use <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/api/createAsyncThunk">async thunks</a>, they usually need to be used by that same slice, so it&#x27;s not a bad idea to export them also from there.
<h3>Why I love it</h3>
There are a couple of reasons for which I loved RTK right away. The main one is that they make an impressive work on making type inference in TypeScript work properly.
<ul>
<li>Slices detect the type of the initial state in order to know what is expected to be passed as <code>state</code> to reducers.</li>
<li>You can infer the type of your store from what is generated by <code>configureStore</code> or <code>combineReducers</code> functions, allowing you to also properly type the <code>dispatch</code> function.</li>
<li>Action creators are generated with the proper param types based on the reducers provided to <code>createSlice</code> or the callback provided to <code>createAsyncThunk</code>.</li>
<li><code>createSlice</code> can consume all the action creators generated by <code>createAsyncThunk</code>, and infer the proper payload for each one of them.</li>
</ul>
Thanks to this, the code is super predictable and potential bugs can be caught sooner.
<blockquote>
More details on how to use RTK with typescript https://redux-toolkit.js.org/usage/usage-with-typescript
</blockquote>
But this is not the only benefit. Other reasons that make it very nice to work with are:
<ul>
<li>It keeps the same concepts from Redux, but removing the boilerplate or your own helper functions if you have them (yes, I didn&#x27;t want to keep all those <code>switch</code> statements everywhere).</li>
<li>It is mostly transparent for the components dispatching the actions or consuming the state.</li>
<li>All the tests covering your reducers/actions/components should keep passing, as the outcome from RTK is the same as if you built everything manually.</li>
<li>It introduces a couple of extra helpers, like <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/rtk-query/overview">RTK query</a> or the <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/api/createListenerMiddleware">listener middleware</a>, which is very useful to dispatch actions as a result of other actions being dispatched.</li>
</ul>
<h3>Challenges I found</h3>
Of course, adopting a new tool always comes with some challenges.
The main &quot;problem&quot; I found is that RTK is a bit opinionated, but that&#x27;s precisely how they manage to reduce boilerplate code, and it&#x27;s also &quot;advertised&quot; in their website, so not a big surprise.
I usually prefer to be in control of everything (configuration over convention), but that&#x27;s just a personal preference, and sometimes it&#x27;s ok to accept a bit of magic for the sake of simplicity.
This caused the next challenges for my particular case:
<ul>
<li>
RTK assumes your project follows a strict <a target="_blank" rel="noopener noreferrer" href="https://redux.js.org/tutorials/fundamentals/part-7-standard-patterns#flux-standard-actions">Flux pattern</a>, meaning your actions always have the properties <code>type</code>, <code>payload</code> and optionally <code>meta</code> and <code>error</code>.
I was not following this pattern (my actions had the payload &quot;mixed&quot; on the first object level), so I had to adapt all my actions first, fix the tests, then migrate the reducer to RTK and make sure the tests kept passing.
<div></div>
</li>
<li>
Immutability is no longer promoted in reducers. Instead, the library requires <a target="_blank" rel="noopener noreferrer" href="https://immerjs.github.io/immer/">immer</a> internally, and using proxies they automatically ensure immutability, even if you &quot;mutate&quot; the state.
This feels a bit counterintuitive at first, because redux always promoted an immutable approach, and the use of immer is not obvious.
However, they have <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/usage/immer-reducers#why-immer-is-built-in">good reasons</a> to do it, and if your reducers return a state object, you can still use the immutable approach. It&#x27;s up to you.
<div></div>
</li>
<li>
Action creators resulting of <code>createAsyncThunk</code> can only have one argument.
This is because the async thunk callback receives a second argument containing the <code>dispatch</code> and <code>getState</code> functions, as well as a couple of other goodies, enforcing this convention to avoid human errors.
<div></div>
</li>
<li>
You cannot dispatch other actions after the callback passed to <code>createAsyncThunk</code>, because it has to return the payload and RTK dispatches the action for you.
However, this is probably something you usually should not be doing, and in my case, I resolved it with the <a target="_blank" rel="noopener noreferrer" href="https://redux-toolkit.js.org/api/createListenerMiddleware">listener middleware</a>, listening for the second action in order to dispatch the second one.
</li>
</ul>
<h3>Conclusion</h3>
My conclusion is that RTK is definitely worth it if you use redux.
It makes code simpler, more predictable and better typed. However, it still requires knowing how redux works internally and what are its patterns, in order to understand why the library exposes the helpers it exposes and why.
For existing redux projects, you can migrate progressively, addressing the challenges bit by bit and not all at once, which simplifies its adoption.
For new projects in which you want to use redux, I would recommend starting right away with RTK.

A couple of weeks ago I migrated a React project using redux to Redux Toolkit. Redux has always been my preferred state management tool for React (which doesn't mean I "love" it), but one of its main problems is the amount of boilerplate code it requires.

Because of that I had a couple of helper…

My experience migrating to Redux Toolkit (RTK)

In 2019, GitHub published their own solution to run automated workflows called GitHub Actions, which allowed those hosting their code in GitHub, to be able to define and run their CI/CD pipelines in the same platform. When it was released, one of the main pain points to use it was that defining…

How to reduce duplication in your GitHub Actions workflows

Recently ReactJS v18 was released. I started to look into its improvements, and checked which of my projects could benefit from them.
The main react-based project I maintain at the moment of writing this article is <a target="_blank" rel="noopener noreferrer" href="https://github.com/shlinkio/shlink-web-client">shlink-web-client</a>, so I naturally created a new branch and started the process.
The update was quite smooth, as most of the other dependencies I was using from the react ecosystem were already updated to support v18. All but one, <a target="_blank" rel="noopener noreferrer" href="https://enzymejs.github.io/enzyme/">enzyme</a>, the testing framework I was using to unit-test react components.
To be precise, I was able to continue running my tests, but some of them (those using <code>mount</code> instead of <code>shallow</code>) were still rendering components with the &quot;old&quot; React v17 approach, causing them to behave as in that version and a warning being printed in the console.
<h3>First fix attempt</h3>
The way enzyme integrates with the react version you are using is through an adapter library. The last official one supports only React v16, but someone released an <a target="_blank" rel="noopener noreferrer" href="https://github.com/wojtekmaj/enzyme-adapter-react-17">unofficial adapter for v17</a> which, with close to 650k weekly downloads, quickly became the way to go.
I checked if there was one for React 18, but sadly there wasn&#x27;t.
Also, the author of the adapter for v17 posted an article indicating <a target="_blank" rel="noopener noreferrer" href="https://dev.to/wojtekmaj/enzyme-is-dead-now-what-ekl">enzyme is dead</a>, he was not going to release one for v18, and publishing one for v17 was a mistake.
With those facts in mind, I started to face what was probably going to be the moment to migrate to <a target="_blank" rel="noopener noreferrer" href="https://testing-library.com/docs/react-testing-library/intro">react testing library</a>.
<h3>Migration strategy</h3>
I decided to follow a migration strategy similar to the one suggested in the article above, but with a couple of modifications:
<ul>
<li>I update to React 18 anyway, as tests still pass.</li>
<li>Migrate right away all tests using <code>mount</code>, as those are the ones behaving differently.</li>
<li>New tests should be written directly with react testing library (RTL from now on).</li>
<li>When an existing test needs to be changed, consider migrating it to RTL, and do it if there&#x27;s time.</li>
<li>Introduce some PR from time to time just migrating a handful of tests.</li>
<li>Once everything is migrated, get rid of everything related with enzyme.</li>
</ul>
<h3>First contact and impressions</h3>
I was a bit sceptical with RTL, as it was focused more on integration tests, instead of unit tests, which meant I had to completely change the mindset to write tests.
It also didn&#x27;t support shallow rendering, meaning that you will end up sometimes rendering a bigger component tree when testing container-like components, potentially causing side effects that could affect your test.
On the other hand there&#x27;s more and more people stating that <a target="_blank" rel="noopener noreferrer" href="https://kentcdodds.com/blog/why-i-never-use-shallow-rendering">shallow rendering is an anti-pattern</a>, and that&#x27;s probably the reason of enzyme getting &quot;discontinued&quot;.
Also, RTL has become the <a target="_blank" rel="noopener noreferrer" href="https://reactjs.org/docs/testing.html#tools">officially recommended framework</a> to test react components, it has become very popular, and it has even joined a wider group of testing libraries under the <a target="_blank" rel="noopener noreferrer" href="https://testing-library.com/"><code>@testing-library</code></a> organization.
The thing is that I started to use it, and I was quickly surprised with how it felt to work with it, and all the benefits it was bringing to my tests.
<h3>Benefits I have experienced</h3>
Once I started migrating the first tests I noticed some benefits:
<ul>
<li>Changing my mindset was actually easier than I thought. The docs are very well written, so it&#x27;s relatively easy to start using RTL.</li>
<li>My tests got simpler, and required less of those nasty global mocks that jest allows (I&#x27;m looking at you, <a target="_blank" rel="noopener noreferrer" href="https://jestjs.io/es-ES/docs/mock-functions#mocking-modules"><code>jest.mock</code></a>).</li>
<li>Tests were way less coupled with implementation details. I moved from testing the component props, to test what the user would see on the screen.</li>
<li>Related with previous point, I started to write my tests by looking at the screen instead of the component&#x27;s implementation.</li>
<li>Changes in source code led to fewer tests requiring to be changed.</li>
<li>A progressive migration was possible, where some tests still use enzyme and others are already using RTL.</li>
</ul>
And overall, it felt good and I have already dedicated a couple of PRs just to migrate more tests.
<h3>Challenges I faced</h3>
But of course, migrating to a new tool always comes with its own challenges, and this was not an exception. I&#x27;ll explain the main ones I have faced so far and how I tackled them:
<ul>
<li>
RTL lets you do some things in a couple of different ways, and the docs may not be super clear about what&#x27;s the recommended option.
For this I recommend reading <a target="_blank" rel="noopener noreferrer" href="https://acel.me/DCMb1">this article</a> from Kent C. Dodds, RTL&#x27;s author, which explains some common mistakes when using the library.
</li>
<li>
At first, I struggled a bit with asynchronous side effects and state changes in components, which can easily end up printing a warning like <code>Warning: An update to MyComponent inside a test was not wrapped in act</code>.
The warning also explains how you are supposed to solve this by wrapping your code in <code>act()</code>, but this warning comes from React, not RTL, and what it is not telling you is that RTL already wraps all needed operations in <code>act()</code>.
What this error usually mean is that you are not waiting for something to happen. <a target="_blank" rel="noopener noreferrer" href="https://acel.me/jnUdT">This other article</a>, also from Kent C. Dodds, explains everything you need to know about the things that can lead to this warning, and how to solve them.
</li>
<li>
Since you start testing from the user point of view, some use cases can be challenging to test without coupling with implementation details. For example, when you have a component which dynamically renders different images or icons, I used to directly check the component properties.
For this I found the best solution was to use <a target="_blank" rel="noopener noreferrer" href="https://jestjs.io/docs/snapshot-testing">jest snapshots</a>. I didn&#x27;t want to manually check which SVG was being rendered in the DOM, just that different ones were being used in every case.
<pre class="language-tsx"><code class="language-tsx">// I went from this...
test(&#x27;...&#x27;, () =&gt; {
 const wrapper = shallow(&lt;MyComp dynamicValue=&quot;foo&quot; /&gt;);
 expect(wrapper.find(FontAwesomeIcon).prop(&#x27;icon&#x27;)).toEqual(someIcon);
})

// To this
test(&#x27;...&#x27;, () =&gt; {
 const { container } = render(&lt;MyComp dynamicValue=&quot;foo&quot; /&gt;);
 expect(container.firstChild).toMatchSnapshot();
})
</code></pre>
</li>
<li>
Something very similar happens if you use some library that renders canvas. There&#x27;s no way to test that from a DOM point of view, which is what RTL does.
In my case I was using <a target="_blank" rel="noopener noreferrer" href="https://www.chartjs.org/">Chart.js</a> to render some charts, so I used <a target="_blank" rel="noopener noreferrer" href="https://github.com/hustcc/jest-canvas-mock">jest-canvas-mock</a>, which exposes a method on the canvas to see which are all the events invoked from the library.
Then I used snapshots again to verify they had the expected &quot;shape&quot;.
<pre class="language-tsx"><code class="language-tsx">// I went from this...
test(&#x27;...&#x27;, () =&gt; {
 const wrapper = shallow(&lt;MyCompWithCanvas /&gt;);
 expect(wrapper.find(Chart).prop(&#x27;data&#x27;).datasets).toEqual(...);
})

// To this
test(&#x27;...&#x27;, () =&gt; {
 const { container } = render(&lt;MyCompWithCanvas /&gt;);
 const events = container.querySelector(&#x27;canvas&#x27;)?.getContext(&#x27;2d&#x27;)?.__getEvents();

 expect(events).toBeTruthy();
 expect(events).toMatchSnapshot();
})
</code></pre>
</li>
</ul>
<h3>Conclusion</h3>
For better or worst, enzyme should no longer be considered an option. If you are starting a new project or are using it on small ones, consider migrating to RTL as soon as possible.
If you are using it in bigger projects, following the process described in this article will probably help you.
In any case, you will get surprised. React testing library is a great tool and provides many benefits over enzyme.
Its main issue is that it&#x27;s not a drop-in replacement for enzyme, so you will have to change a bit your mindset.

Recently ReactJS v18 was released. I started to look into its improvements, and checked which of my projects could benefit from them. The main react-based project I maintain at the moment of writing this article is shlink-web-client, so I naturally created a new branch and started the process.

The…

My experience migrating from enzyme to react testing library

Capturing code coverage for a test suite is a very useful way to know which parts of your source code are actually getting executed by tests. This is useful not only to know if you need to add more tests to your project (in case the coverage is too low), but also, if you want to apply technics like…

Capturing remote code coverage in API tests with PHPUnit

In the last couple of days I have been playing around with the idea of combining GitHub Actions and GitHub Pages to be able to dynamically build preview environments every time a pull request is opened against a repository hosting a client-side web app.
The idea is that you can quickly preview the changes made by some contributor, without having to clone their branch and running everything locally, something specially useful for open source projects.
In this article I will explain how to set everything up, and what are some considerations and problems you may face during the process.
<h3>Enable github pages</h3>
The first thing you need to do is enable your project to publish pages on some branch. I recommend creating a new branch for this, let&#x27;s say <code>preview-env</code>, and then enabling publishing on it.
In order to do that go to your repository and then Settings -&gt; Pages, look for the Source section, and you should see something like this:
<img src="/assets/img/preview-envs/github-pages-before-enabling.png" alt="GitHub pages before enabling"/>
Click in the dropdown and select your new branch, <code>preview-env</code>. Then click Save.
<img src="/assets/img/preview-envs/github-pages-after-enabling.png" alt="GitHub pages after enabling"/>
After saving you will see a message at the top of this section telling you what&#x27;s the URL in which the pages are published. It always follows the same pattern. If your organization is <code>my-org</code> and your repository is <code>my-repo</code>, the url will be <code>https://my-org.github.io/my-repo/</code>.
<h3>Configure the workflow</h3>
Now we need to configure a workflow that will &quot;deploy&quot; the preview environment for the branch every time a new pull request is created or updated:
Let&#x27;s imagine we have a React web app that is built with node.js. The configuration could look like this.
<pre class="language-yml"><code class="language-yml"># .github/workflows/deploy-preview.yml
name: Deploy preview

on:
 pull_request: null

jobs:
 deploy:
 runs-on: ubuntu-20.04
 continue-on-error: true
 steps:
 - name: Checkout code
 uses: actions/checkout@v2
 - name: Use node.js 14.15
 uses: actions/setup-node@v1
 with:
 node-version: 14.15
 - name: Generate slug
 id: generate_slug
 run: echo &quot;##[set-output name=slug;]$(echo ${GITHUB_HEAD_REF#refs/heads/} | sed -r &#x27;s/[~\^]+//g&#x27; | sed -r &#x27;s/[^a-zA-Z0-9]+/-/g&#x27; | sed -r &#x27;s/^-+\|-+$//g&#x27; | tr A-Z a-z)&quot;
 - name: Build
 run: npm ci &amp;&amp; npm run build
 - name: Deploy
 uses: JamesIves/github-pages-deploy-action@4.1.1
 with:
 branch: preview-env
 folder: build
 target-folder: ${{ steps.generate_slug.outputs.slug }}
 - name: Publish env
 uses: marocchino/sticky-pull-request-comment@v2
 with:
 header: Preview environment
 message: |
 ## Preview environment
 https://my-org.github.io/my-repo/${{ steps.generate_slug.outputs.slug }}/
</code></pre>
<blockquote>
If you are not familiar with GitHub Actions yet, take a look at the <a target="_blank" rel="noopener noreferrer" href="https://docs.github.com/en/actions">official documentation</a>.
</blockquote>
This workflow will follow the next steps:
<ul>
<li>Checkout the code from the repository.</li>
<li>Set-up node.js 14.15</li>
<li>Based on the branch name, generate a slug that we will use to have separated deployment folders for every pull request we receive. The slug is exposed to next steps as an output.</li>
<li>Build the project. This step will obviously depend on the needs of your project.</li>
<li>Using <a target="_blank" rel="noopener noreferrer" href="https://github.com/JamesIves/github-pages-deploy-action">JamesIves/github-pages-deploy-action</a>, deploy the contents we previously built in the GitHub page, under a sub-folder matching the slug we generated a few steps above, and under the branch <code>preview-env</code> (remember we enabled GitHub pages for this branch).</li>
<li>Finally, using <a target="_blank" rel="noopener noreferrer" href="https://github.com/marocchino/sticky-pull-request-comment">marocchino/sticky-pull-request-comment</a>, post a comment in the pull request with a link to the preview environment, so that you can just click on it and see the deployment.</li>
</ul>
The steps may change depending on your needs and how you build your project. For example, I have an extra step to update a property in the <code>package.json</code> that is then used to allow serving the app in a sub-path instead of the root of the domain.
You may also not build your site with node.js, as you could be using any of the multiple existing static site generators out there, but you get the idea.
Finally, you have probably noticed the config includes <code>continue-on-error: true</code>. This will allow the pull request to be merged even if this process fails for some reason.
<h3>Considerations when using forks</h3>
While the configuration above will work perfectly when creating pull requests inside the same repository, there are some limitations when the pull request comes from a fork (which is the usual in OSS projects).
For security reasons, the <code>GITHUB_TOKEN</code> injected in those cases doesn&#x27;t have write permissions, which will make the &quot;deploy&quot; step fail.
Also, you cannot define your own token with more permissions and inject it as a secret, as the secrets are not exposed to workflows running on forks.
However, with a couple small config changes, we can work around these limitations:
<h4>Change event</h4>
The first thing we need to do is use the <code>pull_request_target</code> event instead of the <code>pull_request</code> one.
<pre class="language-diff"><code class="language-diff">on:
- pull_request: null
+ pull_request_target: null
</code></pre>
According to the documentation, this is equivalent to the <code>pull_request</code> event, but it runs in the scope of the base repository.
Thanks to this, the <code>GITHUB_TOKEN</code> will come with write permissions, and we will have access to secrets if needed.
<h4>Checkout the fork</h4>
The second thing we are going to change is that we now need to specify the repository to checkout.
In order to do that, we need to add two inputs to the &quot;Checkout code&quot; step, like this:
<pre class="language-diff"><code class="language-diff"> steps:
 - name: Checkout code
 uses: actions/checkout@v2
+ with:
+ repository: ${{ github.event.pull_request.head.repo.full_name }}
+ ref: ${{ github.event.pull_request.head.ref }}
</code></pre>
This will ensure we checkout from the fork repository, instead of the base one, which is the default, as it&#x27;s the one where the <code>pull_request_target</code> event is scoped to.
And that&#x27;s it. With these two changes you can now deploy preview environments for pull requests coming from forks.
<blockquote>
IMPORTANT! The limitations of the <code>pull_request</code> event are there for a reason. People could potentially get access to your secrets or manipulate your code. Proceed at your own risk.
</blockquote>
<h3>Other considerations</h3>
Other than the limitations for forks, there are a few other minor things to take into consideration.
<ul>
<li>Changes in GitHub Pages do not reflect immediately. It may take up to a couple of minutes for the deployment to be available, but considering you will usually not review a PR right after it has been created, this should not be a big issue.</li>
<li>As you have seen, every time a pull request is created, the deployment will generate a new folder in your GitHub Pages branch. This may result in a lot of old folders to pile-up, so you may want to set up some way to clean them up periodically or after the pull request has been merged.</li>
<li>Due to the fact that GitHub Pages only allows static pages to be served, this process cannot be used with apps that require some kind of back-end or server-side logic.</li>
<li>If your project is already using GitHub Pages for something else (serve the project&#x27;s website or docs), that may conflict with this, preventing you from having this kind of preview envs.</li>
</ul>

In the last couple of days I have been playing around with the idea of combining GitHub Actions and GitHub Pages to be able to dynamically build preview environments every time a pull request is opened against a repository hosting a client-side web app. The idea is that you can quickly preview the…

Setting-up a preview environment for your client-side apps with GitHub Actions and GitHub Pages

At some point, any enterprise project will probably need a message queue.
A message queue is used to publish information (usually known as messages) that a different &quot;node&quot; (usually known as worker) will consume in order to perform a specific action.
It is frequently used in web applications to pass information to background workers that consume the queue and perform long tasks, but it is also an important part when applying concepts like <a target="_blank" rel="noopener noreferrer" href="https://martinfowler.com/eaaDev/EventSourcing.html">Event Sourcing</a> to your architecture.
It is also really important when working with microservices, since it is a way to enable data exchange between each service.
Whatever the case, it is very likely that you will eventually need to decide which message queue to use.
<h3>Why AWS Simple Queue Service</h3>
In my company we had to decide this some time ago.
We started testing different services, seeing what their strengths and weaknesses were.
If I remember correctly, we tested <a target="_blank" rel="noopener noreferrer" href="https://www.rabbitmq.com/">RabbitMQ</a>, <a target="_blank" rel="noopener noreferrer" href="https://kr.github.io/beanstalkd/">Beanstalkd</a>, <a target="_blank" rel="noopener noreferrer" href="https://redis.io/">Redis</a> and <a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/sqs/">AWS Simple Queue Service</a> (SQS from now on).
The conclusion was that RabbitMQ is the one with the more features, by far. However, it is also the most complicated to configure and interact with.
Also, we would have to maintain the RabbitMQ clusters ourselves (the same happens with Beanstalkd).
Since we work with AWS, we decided to give SQS a try, and its simplicity convinced us.
It is super simple to create a new queue, and using one of their SDKs, you can interact with it in minutes.
Also, the price is absurdly cheap. It is virtually free for &quot;small&quot; amounts of requests (and when I say small, I mean that performing 1 request to their API every 20 seconds during 1 year costs about $0.30 at the end of the year).
It is by far cheaper than maintaining your own instances with a different service, and I&#x27;m not counting the cost in time you will have to invest in maintaining those servers.
<h3>The &quot;small&quot; problem with SQS</h3>
One of the most interesting features with RabbitMQ is that you can open one socket with the queue server, and once a new message gets in, the worker is &quot;notified&quot; and the message can be consumed. This way, you have a process which is always running, listening for a queue, and processing messages as soon as possible.
This is usually the desired behavior. You don&#x27;t want any delay on processing messages (or the delay should be as short as possible).
On the other hand, SQS bills by API request. This means that the worker has to manually check if there&#x27;s any message in the queue, in small intervals.
Our first approach was configuring cron jobs in our workers, so that they could check if there were any message in the queue, and process all of them, if any.
This was problematic, since it applies a delay of up to one minute (you can&#x27;t configure a cron to be run with an interval smaller than that), which is not desirable, particularly when the job takes 5-10 seconds to complete. Adding an extra 600% of waiting time can affect the user experience, or have other side effects, depending on the task.
<h3>The long polling to the rescue</h3>
The first solution that came to our mind was making the workers run requests on an infinite loop, but that&#x27;s not a very clean approach, and will result in our workers being at a 100% of CPU usage all the time.
We started investigating and found out about a feature in SQS called <a target="_blank" rel="noopener noreferrer" href="https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-long-polling.html">long polling</a>. It lets you tell the queue to wait up to 20 seconds before returning a response, if the message queue is empty, and if during that time a new message enters the queue, return a response immediately.
Using it, you can make a worker continuously perform requests at 20 second intervals, but if a message is in the queue, the response will be returned immediately.
The final behavior is very similar to a socket, in which you can process a message as soon as it enters the queue, and the worker does not consume any resource while it is waiting for a response.
<h3>Configure a queue with long polling</h3>
There are two ways of configuring the long polling. You can do it in AWS, and make the queue always have it, or you can do it client-side, and add a <code>WaitTimeSeconds</code> param with the time in seconds (with a maximum value of 20 seconds).
The second option is very easy to implement when using any of the official SDKs, and is more flexible, since you can decide when to wait for a message and when to fail immediately when the queue is empty.
The first one is very easy too. Just set a value for the &quot;Receive Message Wait Time&quot; param, when configuring the queue.
<img src="/assets/img/sqs-socket/receive-message-wait-time.png" alt="SQS Receive Message Wait Time"/>
That should be all, but there&#x27;s one important consideration.
If you have several workers consuming the same queue with long polling and one message enters the queue, all the workers will receive it until one of them deletes it. This is usually problematic, since it could lead to one task being processed more than once.
In order to prevent this, you should set a small value in the &quot;Default Visibility Timeout&quot; param, which tells SQS to deliver the message just to one consumer, and wait that amount of time before delivering it to the rest.
This way, you can make one of the workers receive the message, delete it from the queue, and then process it, and that will prevent other workers from receiving it.
Usually, setting a value of 3-5 seconds should be more than enough for the first worker to remove it from the queue. However, if you expect your workers to need a longer time, just increase the value.
Remember, that value will delay delivering the message to all the workers but the first that gets it.
<h3>Conclusion</h3>
We have seen an efficient way of working with SQS, the most simple, yet powerful message queue.
If you have more complex needs, it might end being too simple, but if not, it is possible to tweak it enough to use it on large applications and services.

At some point, any enterprise project will probably need a message queue. A message queue is used to publish information (usually known as messages) that a different "node" (usually known as worker) will consume in order to perform a specific action.

It is frequently used in web applications to…

How to simulate a socket connection with AWS Simple Queue Service

In a project I&#x27;m working on, we recently needed to add some kind of encryption system that allowed us to store sensitive information in a secure manner, but being able to access to it at runtime in order to pass it to third party services.
Securely storing your own app passwords is easy. You just hash it, and the password can be verified without having to decrypt it, but if you need to &quot;decrypt&quot; it, the solution is not that easy.
We started by building our own solution based on <a target="_blank" rel="noopener noreferrer" href="https://docs.zendframework.com/zend-crypt/">zend/crypt</a>, which is an excellent solution and works like a charm, but you still need to store the master key somehow, and that&#x27;s the big problem.
After some research, someone pointed us to <a target="_blank" rel="noopener noreferrer" href="https://www.vaultproject.io/">vault project</a>, a self-hosted service which manages encrypted key-value pairs, in a more secure manner, where the master key is securely handled. I recommend you read the docs in order to see all its features.
Vault supports different storage backends, each one with its own pros and cons. Some are meant to be used in development environments (like in memory and local file) and others are more suitable for production, with support for high availability (like DynamoDB or <a target="_blank" rel="noopener noreferrer" href="https://www.consul.io/">consul</a>, another hashicorp&#x27;s service).
The official documentation indirectly recommends using vault with consul, which is a service with more features than just storing data. Indeed, it is a service discovery tool widely used this days in container-based infrastructures.
For this reason, we started using both vault and consul in development, in order to get used to both services, but in the end we found it problematic because of different reasons:
<ul>
<li>The consul server needs to get started before vault, otherwise vault fails on start-up. We saw this problem happening in development very frequently, which was annoying.</li>
<li>We talked with an expert sysadmin that was using consul in production environments. He told us it was a magnificent service, but he warn us about the possibility of having random data loss for no apparent reason. 
When using consul as a service discovery tool, that is not a big problem. Just recreating the consul cluster, all the data is retrieved again from running services. In our case it isn&#x27;t that simple. We were going to store sensitive and important data in consul, and loosing it could be a big problem.</li>
<li>I usually prefer to maintain as less services as possible in production. I see it from the point of view of a developer. If there is a cloud service that does the job, I prefer to use it rather than maintaining my own server. 
Be serverless as much as possible.</li>
</ul>
<h3>Finding an alternative</h3>
Ok, consul is discarded. We need another storage alternative.
Looking at vault&#x27;s documentation we saw there are a couple <a target="_blank" rel="noopener noreferrer" href="https://www.vaultproject.io/docs/configuration/storage/dynamodb.html">storage backends</a> that support high availability.
We finally decided to go with <a target="_blank" rel="noopener noreferrer" href="https://aws.amazon.com/dynamodb/">AWS DynamoDB</a>. It is much less prone to data loss due to redundant storage under it, easier to buck-up and restore, and much cheaper than having dedicated servers. Also, the project&#x27;s infrastructure is already in AWS, so communication should be fast.
Using DynamoDB we now only need to manage vault servers.
<h3>Vault&#x27;s high availability</h3>
Another of the vault&#x27;s feature is the possibility to configure several servers in high availability mode, as long as the storage supports it.
This was a must for us, since getting vault down would make the service to not properly work and produce inconsistent data.
The way vault manages high availability is very interesting. The first server that&#x27;s unsealed is set as active, and the rest are set in standby.
If a request is performed to one of the servers in standby, they just redirect the request to the active server.
If the active server is down, then one of the standby servers takes the leadership and is set as active.
The communication between servers does not happen via network communication, but on the shared storage. The active server creates a lock file, and this way, the rest of the servers know they have to standby.
This makes every server to work in an independent manner, and makes it easy to add more servers to the cluster if needed.
The only problem is how do I make my app dynamically read from the active server, but be able to read from others if the active is down.
Do I have to manually manage the load balancing? Do I have to implement some kind of strategy that tries to communicate to each server until one returns a response? That&#x27;s odd.
Vault&#x27;s documentation recommends using another consul&#x27;s feature to solve this problem, some kind of DNS that makes a cluster name to resolve to the active instances IP address. This way you just need to perform requests to the cluster name.
That&#x27;s good, but we weren&#x27;t going to use consul, right?
<h3>AWS internal load balancer to the rescue</h3>
After some investigation we saw contradictory opinions on behalf of hashicorp about using a load balancer with vault. Sometimes they recommend not to use it, sometimes they say they are not against nor in favor.
Because of the way high availability works in vault, if the balancer redirects to a standby instance and it redirects to the balancer again, you could enter an infinite redirection loop.
But what if you could make the load balancer consider standby instances out of service?
Gladly, vault&#x27;s API has an <a target="_blank" rel="noopener noreferrer" href="https://www.vaultproject.io/api/system/health.html">endpoint</a> that performs a health check returning a status 200 only on unsealed active instances. If the instance is not active, it returns a 429 status. This endpoint doesn&#x27;t require authentication.
Using this endpoint for the balancer&#x27;s health check, the balancer will never redirect to a standby instance.
Also, if the active instance has any problem and goes down, as soon as a standby instance takes leadership, the load balancer will notice it, and mark it as InService, starting to redirect requests to that instance.
Pretty nice. We can now configure our app to perform requests to the balancer and the problem is solved.
<h3>Configuring the AWS load balancer</h3>
When configuring the load balancer for the vault instances, we just need to take into account a few things.
First, the balancer should be a classic load balancer, and it has to be configured as internal. This makes the balancer name resolve to the private IP address, and therefore, you prevent anyone accessing vault from the outside.
It should also be configured to forward HTTP traffic on port 8200 (vault works on this port)
<img src="%7B%7Bsite.url%7D%7D/assets/img/vault-with-load-balancer/balancer-first-step.png" alt="Load balancer"/>
Then, when you are asked to configure the health check, use this params.
<img src="%7B%7Bsite.url%7D%7D/assets/img/vault-with-load-balancer/health-check.png" alt="Health check"/>
Configure the interval and healthy threshold according to your needs. Since for me it is a critical service, I reduced the interval to 15 seconds, and set the healthy threshold to 2. This way, when tha active instances goes down, it wont take more than 30 seconds to recover the service.
Finally add the instances to the balancer.
If everything is properly configured, after some seconds, in the instances tab of the balancer, you should see that only one of the instances is InService.
<img src="%7B%7Bsite.url%7D%7D/assets/img/vault-with-load-balancer/instances-status.png" alt="Instances status"/>
Now the balancer and the vault instances are ready to work!

In a project I'm working on, we recently needed to add some kind of encryption system that allowed us to store sensitive information in a secure manner, but being able to access to it at runtime in order to pass it to third party services. Securely storing your own app passwords is easy. You just…

Take advantage of vault project's high availability with an AWS internal load balancer

<a target="_blank" rel="noopener noreferrer" href="https://letsencrypt.org/">Let&#x27;s Encrypt</a> is the latest internet trending. Its a relatively young Certification Authority which intends to provide free and automated SSL certificates so that everybody can set encryption to their sites and applications.
It is a <a target="_blank" rel="noopener noreferrer" href="https://www.linuxfoundation.org/">Linux Foundation</a> project sponsored by most of the biggest Internet companies, such as Google, Facebook, Mozilla or Cisco.
They provide tools to automatically generate, setup and renew SSL certificates, but those tools are still in early stages, and not all the integrations are easy.
In this article I&#x27;m going to explain how to setup one of this certificates in an Amazon Web Services Elastic Load Balancer (ELB from now on).
<h3>Generate the certificate</h3>
There are very good tutorials out there on how to generate a certificate, so I&#x27;m not going to dedicate much to this.
I usually follow the instructions from <a target="_blank" rel="noopener noreferrer" href="https://certbot.eff.org/">certbot</a>.
In there, you have to choose the web server of your choice, in this case &quot;None of the above&quot;, and your operating system, for example Ubuntu 16.04 (the only difference with older Ubuntu versions is that you can install the CLI tool from the standard repositories, otherwise there is a previous step where you have to download the <code>certbot-auto</code> script and run it, but that&#x27;s it).
Once you have installed the <code>letsencrypt</code> tool (or <code>certbot-auto</code>), you have to generate the certificate with the webroot plugin. This plugin just generates the certificate by providing the document root for each domain that will be included in the certificate.
<pre><code>letsencrypt certonly --webroot -w /var/www/mydomain -d mydomain.com -d www.mydomain.com -w /var/www/cool-site -d cool-site.com -d www.cool-site.com
</code></pre>
This will create a certificate valid for mydomain.com, www.mydomain.com, cool-site.com and www.cool-site.com.
All of the domains get verified from let&#x27;s encrypt side by putting a temporary hidden file inside provided web root folders.
In this case, the first two domains will be checked in the /var/www/mydomain folder, and the last two in /var/www/cool-site. If those domains are really served from there, the certificate will get properly created.
The problem with this in an Elastic Load Balancer is that you are running this tool in one server, but you will have more than one balanced server. This could make the previous process fail if the wrong server responds to the let&#x27;s encrypt check request.
To prevent this you will have to leave just one server temporarily. Don&#x27;t worry, it only takes a few seconds. Once the certificate is created you can add the rest of the servers again.
<blockquote>
There are other plugins which simplify deploying certificates in standalone web servers.
</blockquote>
<blockquote>
For example, the apache plugin automatically sets the configuration in the apache virtual host of your choice.
</blockquote>
If everything worked, you should now have 4 files in the <code>/etc/letsencrypt/live/mydomain.com</code> folder.
<pre><code>/etc/letsencrypt/live/mydomain.com
 |-- cert.pem
 |-- chain.pem
 |-- fullchain.pem
 |-- privkey.pem
</code></pre>
<h3>Setup the certificate in an ELB</h3>
Now that you have your certificate files, you need to upload them to the ELB.
Go to the AWS web console, select the EC2 service and go to Load Balancers. Once there select the load balancer of your choice and go to the Listeners tab.
<img src="/assets/img/load-balancers-listeners.png" alt="Load Balancer listeners"/>
Now you have to click on the &quot;Change&quot; link, in the SSL Certificate column of the HTTPS protocol, in order to set the new certificate.
<img src="/assets/img/load-balancers-set-certificate.png" alt="Load Balancer set SSL Certificate"/>
In this window, select the third option, Upload a new SSL certificate to AWS... and set the name of your choice for it.
Then, in the Private key field you have to place the contents of the <code>cert.pem</code> file, and in the Public key certificate field, the contents of the <code>fullchain.pem</code> file. You can leave the Certificate chain field empty.
Save, and you are done. Your ELB is now serving HTTPS request by using the previously generated Let&#x27;s Encrypt certificate.
<h3>Renewal</h3>
Renewing this certificate is easy. Just run the command <code>letsencrypt renew</code> (or <code>certbot-auto renew</code>) in the same server used to create the certificate in the first place.
That will generate the new certificate in the same location, and you can repeat the previous step with it.
The main problem with Let&#x27;s Encrypt certificates is that they are valid for three months only. Auto renewal is a feature that they want to implement at some point, but it&#x27;s not available yet.
By the moment, the best solution for standalone servers is creating a cron job that automatically runs the <code>letsencrypt renew</code> command, but for an ELB that won&#x27;t fix the problem.
You will probably need to create some kind of script that automatically uploads the certificate to the ELB after being renewed, by using the AWS REST API or the CLI interface.
<a target="_blank" rel="noopener noreferrer" href="http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-update-ssl-cert.html#us-update-lb-SSLcert-cli">This documentation</a> should be helpful. Pay special attention to the To replace an SSL certificate with a certificate uploaded to IAM section.

Let's Encrypt is the latest internet trending. Its a relatively young Certification Authority which intends to provide free and automated SSL certificates so that everybody can set encryption to their sites and applications. It is a Linux Foundation project sponsored by most of the biggest Internet…

Alejandro Celaya | Blog

Software development, agile methodologies and open source software.

tag Archive — "aws"

— 2017 —

— 2016 —