Pipeline is a Rails plugin/gem to run asynchronous processes in a configurable pipeline.
Execution of sequential user-defined stages in an asynchronous pipeline
Persistence of pipeline instances and stages
Error recovery strategies:
Irrecoverable errors fail the entire pipeline
Recoverable errors are automatically retried (using dj's exponential retry strategy)
Recoverable errors that require user input pause the pipeline for further retry
Cancelling/Resuming of a paused pipeline
Callbacks before and after executing stages and pipeline
Installation and Use
pipeline can be installed as either a RubyGem (recommended) or as a plugin.
pipeline as a RubyGem, add the following lines to your
config.gem "pipeline" config.gem "delayed_job"
rake gems:install rake gems:unpack # Optional, if you want to vendor the gem
To install it as a plugin, run:
script/plugin install git://github.com/dtsato/pipeline.git
Generating the required tables
In order to persist your pipelines and stages, execute the following:
script/generate pipeline # To generate the migration scripts that will store pipelines script/generate delayed_job # To generate the migration scripts for delayed_job rake db:migrate
Starting your workers
You will also need to run your Delayed Job workers that will process the pipeline jobs in the background:
Delayed job (github.com/collectiveidea/delayed_job)
examples for more examples (including error-recovery and cancelling)
class Step1 < :::: def perform puts("Started step 1") sleep 2 puts("Finished step 1") end end class Step2 < :::: def perform puts("Started step 2") sleep 3 puts("Finished step 2") end end class TwoStepPipeline < :: define_stages Step1 >> Step2 end .(TwoStepPipeline.new)
Copyright © 2009 Danilo Sato. See LICENSE for details.