Glen Holcomb

Read this first

Microservice Testing, or the MiddleMan

I’ve been thinking about Component Testing recently especially in an environment where the components are not necessarily all written in the same language, or you have some convoluted messaging paths through the components.

There are lots of great tools out there for everything but full integration testing. Component level integrations are hard, ideally you want your application to be running as it would in production but you still want to observe all the communication between Components.

How does one simplify the verification of the interactions of the components without having to make heavy modification to the actual code to facilitate testing, or just hitting a service and assuming if the response is correct then you got there in the way you expected to get there?

There is a layer that knows all of this of course, that layer is the network layer. While you are spending time...

Continue reading →

Orchestrating Dependent Tasks with Elixir

Last time I gave a brief high level overview on managing a job pipeline (DAG) with Elixir. Today I’m going to give a fully native Elixir example. I know I promised a Rails integration example last time but this grew much longer than I had originally though. The Rails + Sidekick + Verk example will have to wait for next time.

This post is a bit on the long side so here are some short cuts in case you want to jump around:

  1. The Pipeline
  2. Native Elixir
  3. A Note On Non-trivial Graphs

The Pipeline

First we need to define our pipeline. Let’s stick to the original simple DAG from the previous example, although I am happy to report that more complex graphs are now supported!

This time lets give the jobs more realistic names to help keep things straight between prose and the diagrams.

  DNS --------|
              |------------> S3 ------------> GEN
  SSL --------|

Native Elixir

Lets...

Continue reading →

DAG Management in Elixir

The other day I was talking to a friend, he was relating the difficulty he was having managing background job pipelines in his Rails application. The biggest issue was tracking and acting on the success/failure states from outside Sidekick while preserving the dependency structure of the DAG.

He was describing the work he’d done simplifying the graph into an algebra to help with the priority and status and I couldn’t help but think this would be a super simple to do in Elixir, it felt like such a natural fit for the language, enter Assembly Line.

Keep in mind this post is only concerned with simple DAGs. AssemblyLine doesn’t yet handle anything with branches or sub-graphs in it, although that is planned for the future.

Today lets talk about a strategy for managing the DAG itself. While the pipeline can be large and complex, we are really only concerned with the current level...

Continue reading →