fireap Build Status

Consul-based rapid propagative task runner for large systems.


What's this?

This program triggers to execute configured tasks on nodes in Consul Cluster. And tasks will be executed in propagative way.
Task propagation time takes O(log N) in theory; as N means node number in cluster.

So this tool can shorten the time of tasks which take O(N) time in system of large number of nodes.
Typical usecase is software deployment.

The name fireap comes from "fire" and "reap".


Here is a benchmark result comparing fireap against GNU Parallel.

GNU Parallel fireap
real 0m46.906s 0m18.992s
user 0m40.407s 0m00.527s
sys 0m04.241s 0m00.046s

The job executed here is a directory sync operation by rsync command which contains a thousand of files up to total 12MB size through 100 t2.micron instances on AWS EC2.

Concurrency numbers of both GNU Parallel and fireap is 5 in this benchmark.

In fireap, the word "concurrency" means the maximum concurrent number that one node can be "pulled" by other nodes.
You will grasp the concept by going under part of this document.

About Consul

Consul is a tool for service discovery and infrastructure automation developed and produced by HashiCorp.

Take a look at how it works

Below is a demo of fireap task propagation at a 10-node Consul cluster.

Fireap Demo

On the top of the screen, fireap monitor command is executed. This continuously shows the application version and updated information of nodes those are stored in Consul Kv.

On the bottom of the screen, fireap fire command is executed which fires Consul Event. The triggered event is broadcasted to cluster members at once.
And it leads cluster members to execute fireap reap command by Consul Watch mechanism.

Eventually, configured tasks are executed on nodes in the cluster.

Task Propagation Procedure

The image below illustrates task propagation procedure by fireap in Consul cluster.

Fireap Task Propagation Illustration

Leader and Followers

One leader node fires Consul events whose Name is FIREAP:TASK. And it is assumed to be the 1st node in the cluster which finishes the task.
All other nodes are followers who receive events and execute tasks.

Concept of leader and follower is not related to role of server and client in Consul.
Server or client in Consul cluster can be either leader or follower.


  1. Leader fires a Consul event.
  2. The event is broadcasted for followers at once.
  3. Followers execute the task in propagative way:
    1. All followers search for a finished node in the cluster to "pull" update information or contents from the node.
      In first stage, there is only leader who finished the task. So they all tries to "pull" from leader, but maximum number of who can "pull" from a particular node is limited by configuration. Then, only several followers succeed to "pull" and execute the task.
    2. In second stage, leader and several followers now finished the task. Their update will be "pulled" by other followers.
    3. Stage goes on until all followers finish the task.

This propagation way looks like tree branching.
But it is rather robust because even if a follower happens to fail the task, the failure does not affect others.

Consul Kv

Leader and followers store information in Consul Kv about task completion and so on.
All keys of data related to this program begin with prefix fireap/.

How to get started?

More documentation will come soon including starter guide.


The MIT License (MIT)

Copyright (c) 2016 YASUTAKE Kiyoshi