Consul-based rapid propagative task runner for large systems.
This program triggers to execute configured tasks on nodes in Consul
And tasks will be executed in propagative way.
Task propagation time takes O(log N) in theory; as N means node number in cluster.
So this tool can shorten the time of tasks which take O(N) time in system of large number
Typical usecase is software deployment.
The name fireap comes from "fire" and "reap".
Here is a benchmark result comparing fireap against GNU Parallel.
The job executed here is a directory sync operation by
rsync command which
contains a thousand of files up to total 12MB size through 100 t2.micron instances on AWS EC2.
Concurrency numbers of both GNU Parallel and fireap is 5 in this benchmark.
In fireap, the word "concurrency" means the maximum concurrent number that one node can be "pulled" by other nodes.
You will grasp the concept by going under part of this document.
Take a look at how it works
Below is a demo of fireap task propagation at a 10-node Consul cluster.
On the top of the screen,
fireap monitor command is executed.
This continuously shows the application version and updated information of nodes
those are stored in Consul Kv.
On the bottom of the screen,
fireap fire command is executed which fires Consul
The triggered event is broadcasted to cluster members at once.
And it leads cluster members to execute
fireap reap command by Consul Watch
Eventually, configured tasks are executed on nodes in the cluster.
Task Propagation Procedure
The image below illustrates task propagation procedure by fireap in Consul cluster.
Leader and Followers
One leader node fires Consul events whose
And it is assumed to be the 1st node in the cluster which finishes the task.
All other nodes are followers who receive events and execute tasks.
Concept of leader and follower is not related to role of server and client
Server or client in Consul cluster can be either leader or follower.
- Leader fires a Consul event.
- The event is broadcasted for followers at once.
- Followers execute the task in propagative way:
- All followers search for a finished node in the cluster to "pull" update
information or contents from the node.
In first stage, there is only leader who finished the task. So they all tries to "pull" from leader, but maximum number of who can "pull" from a particular node is limited by configuration. Then, only several followers succeed to "pull" and execute the task.
- In second stage, leader and several followers now finished the task. Their update will be "pulled" by other followers.
- Stage goes on until all followers finish the task.
- All followers search for a finished node in the cluster to "pull" update information or contents from the node.
This propagation way looks like tree branching.
But it is rather robust because even if a follower happens to fail the task, the failure does not affect others.
Leader and followers store information in Consul Kv about task completion and so on.
All keys of data related to this program begin with prefix
How to get started?
More documentation will come soon including starter guide.
The MIT License (MIT)
Copyright (c) 2016 YASUTAKE Kiyoshi