Module: Resque::Plugins::Retry

Includes:
Logging
Included in:
ExponentialBackoff
Defined in:
lib/resque/plugins/retry.rb,
lib/resque/plugins/retry/logging.rb

Overview

If you want your job to retry on failure, simply extend your module/class with this module:

class DeliverWebHook
  extend Resque::Plugins::Retry # allows 1 retry by default.
  @queue = :web_hooks

  def self.perform(url, hook_id, hmac_key)
    heavy_lifting
  end
end

Easily do something custom:

class DeliverWebHook
  extend Resque::Plugins::Retry
  @queue = :web_hooks

  @retry_limit = 8  # default: 1
  @retry_delay = 60 # default: 0

  # used to build redis key, for counting job attempts.
  def self.retry_identifier(url, hook_id, hmac_key)
    "#{url}-#{hook_id}"
  end

  def self.perform(url, hook_id, hmac_key)
    heavy_lifting
  end
end

Defined Under Namespace

Modules: Logging Classes: AmbiguousRetryStrategyException

Instance Attribute Summary (collapse)

Class Method Summary (collapse)

Instance Method Summary (collapse)

Methods included from Logging

#log_message

Instance Attribute Details

- (Number) expire_retry_key_after (readonly)

This method is abstract.

The number of seconds to set the TTL to on the resque-retry key in redis

Returns:

  • (Number)

    number of seconds



251
252
253
# File 'lib/resque/plugins/retry.rb', line 251

def expire_retry_key_after
  @expire_retry_key_after
end

- (Array?) fatal_exceptions (readonly)

This method is abstract.

Controls what exceptions may not be retried

Default: `nil` - this will retry all exceptions.

Returns:

  • (Array, nil)


227
228
229
# File 'lib/resque/plugins/retry.rb', line 227

def fatal_exceptions
  @fatal_exceptions
end

Class Method Details

+ (Object) extended(receiver)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Fail fast, when extended, if the “receiver” is misconfigured



49
50
51
52
53
# File 'lib/resque/plugins/retry.rb', line 49

def self.extended(receiver)
  if receiver.instance_variable_get('@fatal_exceptions') && receiver.instance_variable_get('@retry_exceptions')
    raise AmbiguousRetryStrategyException.new(%{You can't define both "@fatal_exceptions" and "@retry_exceptions"})
  end
end

Instance Method Details

- (Object) after_perform_retry(*args)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resque after_perform hook

Deletes retry attempt count from Redis.



396
397
398
399
# File 'lib/resque/plugins/retry.rb', line 396

def after_perform_retry(*args)
  log_message 'after_perform_retry, clearing retry key', args
  clean_retry_key(*args)
end

- (Object) before_perform_retry(*args)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Resque before_perform hook

Increments `@retry_attempt` count and updates the “retry_key” expiration time (if applicable)



374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
# File 'lib/resque/plugins/retry.rb', line 374

def before_perform_retry(*args)
  log_message 'before_perform_retry', args
  @on_failure_retry_hook_already_called = false

  # store number of retry attempts.
  retry_key = redis_retry_key(*args)
  Resque.redis.setnx(retry_key, -1)
  @retry_attempt = Resque.redis.incr(retry_key)
  log_message "attempt: #{@retry_attempt} set in Redis", args

  # set/update the "retry_key" expiration
  if expire_retry_key_after
    log_message "updating expiration for retry key: #{retry_key}", args
    Resque.redis.expire(retry_key, retry_delay + expire_retry_key_after)
  end
end

- (Object) clean_retry_key(*args)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Clean up retry state from redis once done



447
448
449
450
# File 'lib/resque/plugins/retry.rb', line 447

def clean_retry_key(*args)
  log_message 'clean_retry_key', args
  Resque.redis.del(redis_retry_key(*args))
end

- (Object) inherited(subclass)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Copy retry criteria checks on inheritance.



58
59
60
61
# File 'lib/resque/plugins/retry.rb', line 58

def inherited(subclass)
  super(subclass)
  subclass.instance_variable_set('@retry_criteria_checks', retry_criteria_checks.dup)
end

- (Object) instance_exec(*args, &block)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Used to perform retry criteria check blocks under the job instance's context

Returns:

  • (Object)

    return value of the criteria check



433
434
435
436
437
438
439
440
441
442
# File 'lib/resque/plugins/retry.rb', line 433

def instance_exec(*args, &block)
  mname = "__instance_exec_#{Thread.current.object_id.abs}"
  class << self; self end.class_eval{ define_method(mname, &block) }
  begin
    ret = send(mname, *args)
  ensure
    class << self; self end.class_eval{ undef_method(mname) } rescue nil
  end
  ret
end

- (Object) on_failure_retry(exception, *args)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This hook will only allow execution once per job perform attempt. This was added because Resque v1.20.0 calls the hook twice. IMO; this isn't something resque-retry should have to worry about!

Resque on_failure hook

Checks if our retry criteria is valid, if it is we try again. Otherwise the retry attempt count is deleted from Redis.



411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
# File 'lib/resque/plugins/retry.rb', line 411

def on_failure_retry(exception, *args)
  log_message 'on_failure_retry', args, exception
  if @on_failure_retry_hook_already_called
    log_message 'on_failure_retry_hook_already_called', args, exception
    return 
  end

  if retry_criteria_valid?(exception, *args)
    try_again(exception, *args)
  else
    log_message 'retry criteria not sufficient for retry', args, exception
    clean_retry_key(*args)
  end

  @on_failure_retry_hook_already_called = true
end

- (String) redis_retry_key(*args)

Builds the redis key to be used for keeping state of the job attempts.

Returns:

  • (String)

    redis key



85
86
87
# File 'lib/resque/plugins/retry.rb', line 85

def redis_retry_key(*args)
  ['resque-retry', name, retry_identifier(*args)].compact.join(':').gsub(/\s/, '')
end

- (Array) retry_args(*args)

This method is abstract.

Modify the arguments used to retry the job. Use this to do something other than try the exact same job again

Returns:

  • (Array)

    new job arguments



157
158
159
160
161
162
163
164
165
166
167
# File 'lib/resque/plugins/retry.rb', line 157

def retry_args(*args)
  # Here for backwards compatibility. If an "args_for_retry" method exists
  # invoke it, but warn that it is deprecated (and will be removed in a
  # future revision)
  if respond_to?(:args_for_retry)
    warn "`Resque::Plugins::Retry#args_for_retry` is deprecated, please use `Resque::Plugins::Retry#retry_args` instead."
    args_for_retry(*args)
  else
    args
  end
end

- (Array) retry_args_for_exception(exception, *args)

This method is abstract.

Modify the arguments used to retry the job based on the exception. Use this to do something other than try the exact same job again.

Returns:

  • (Array)

    new job arguments



176
177
178
# File 'lib/resque/plugins/retry.rb', line 176

def retry_args_for_exception(exception, *args)
  retry_args(*args)
end

- (Fixnum) retry_attempt

Number of retry attempts used to try and perform the job

The real value is kept in Redis, it is accessed and incremented using a before_perform hook.

Returns:

  • (Fixnum)

    number of attempts



109
110
111
# File 'lib/resque/plugins/retry.rb', line 109

def retry_attempt
  @retry_attempt ||= 0
end

- (Object) retry_criteria_check {|exception, *args| ... }

Register a retry criteria check callback to be run before retrying the job again

If any callback returns `true`, the job will be retried.

Examples:

Using a custom retry criteria check.


retry_criteria_check do |exception, *args|
  if exception.message =~ /InvalidJobId/
    # don't retry if we got passed a invalid job id.
    false
  else
    true
  end
end

Yields:

  • (exception, *args)

Yield Parameters:

  • exception (Exception)

    the exception that was raised

  • args (Array)

    job arguments

Yield Returns:

  • (Boolean)

    false == dont retry, true = can retry



329
330
331
# File 'lib/resque/plugins/retry.rb', line 329

def retry_criteria_check(&block)
  retry_criteria_checks << block
end

- (Array) retry_criteria_checks

Retry criteria checks

Returns:

  • (Array)


288
289
290
# File 'lib/resque/plugins/retry.rb', line 288

def retry_criteria_checks
  @retry_criteria_checks ||= []
end

- (Boolean) retry_criteria_valid?(exception, *args)

Test if the retry criteria is valid

Parameters:

  • exception (Exception)
  • args (Array)

    job arguments

Returns:

  • (Boolean)


260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
# File 'lib/resque/plugins/retry.rb', line 260

def retry_criteria_valid?(exception, *args)
  # if the retry limit was reached, dont bother checking anything else.
  if retry_limit_reached?
    log_message 'retry limit reached', args, exception
    return false 
  end

  # We always want to retry if the exception matches.
  retry_based_on_exception = retry_exception?(exception)
  log_message "Exception is #{retry_based_on_exception ? '' : 'not '}sufficient for a retry", args, exception

  retry_based_on_criteria = false
  unless retry_based_on_exception
    # call user retry criteria check blocks.
    retry_criteria_checks.each do |criteria_check|
      retry_based_on_criteria ||= !!instance_exec(exception, *args, &criteria_check)
    end
  end
  log_message "user retry criteria is #{retry_based_on_criteria ? '' : 'not '}sufficient for a retry", args, exception

  retry_based_on_exception || retry_based_on_criteria
end

- (Number) retry_delay(exception_class = nil)

This method is abstract.

Number of seconds to delay until the job is retried

Returns:

  • (Number)

    number of seconds to delay



119
120
121
122
123
124
125
126
127
# File 'lib/resque/plugins/retry.rb', line 119

def retry_delay(exception_class = nil)
  if @retry_exceptions.is_a?(Hash)
    delay = @retry_exceptions[exception_class] || 0
    # allow an array of delays.
    delay.is_a?(Array) ? delay[retry_attempt] || delay.last : delay
  else
    @retry_delay ||= 0
  end
end

- (Boolean) retry_exception?(exception)

Convenience method to test whether you may retry on a given exception

also be a Class

Parameters:

  • an (Exception)

    instance of Exception. Deprecated: can

Returns:

  • (Boolean)


189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
# File 'lib/resque/plugins/retry.rb', line 189

def retry_exception?(exception)
  # If both "fatal_exceptions" and "retry_exceptions" are undefined we are
  # done (we should retry the exception)
  #
  # It is intentional that we check "retry_exceptions" first since it is
  # more likely that it will be defined (over "fatal_exceptions") as it
  # has been part of the API for quite a while
  return true if retry_exceptions.nil? && fatal_exceptions.nil?

  # If "fatal_exceptions" is undefined interrogate "retry_exceptions"
  if fatal_exceptions.nil?
    retry_exceptions.any? do |ex|
      if exception.is_a?(Class)
        ex >= exception
      else
        ex === exception
      end
    end
  # It is safe to assume we need to check "fatal_exceptions" at this point
  else
    fatal_exceptions.none? do |ex|
      if exception.is_a?(Class)
        ex >= exception
      else
        ex === exception
      end
    end
  end
end

- (Array?) retry_exceptions

This method is abstract.

Controls what exceptions may be retried

Default: `nil` - this will retry all exceptions.

Returns:

  • (Array, nil)


237
238
239
240
241
242
243
# File 'lib/resque/plugins/retry.rb', line 237

def retry_exceptions
  if @retry_exceptions.is_a?(Hash)
    @retry_exceptions.keys
  else
    @retry_exceptions ||= nil
  end
end

- (String) retry_identifier(*args)

This method is abstract.

You may override to implement a custom retry identifier, you should consider doing this if your job arguments are many/long or may not cleanly convert to strings.

Builds a retry identifier using the job arguments. This identifier is used as part of the redis key

Parameters:

  • args (Array)

    job arguments

Returns:

  • (String)

    job identifier



74
75
76
77
# File 'lib/resque/plugins/retry.rb', line 74

def retry_identifier(*args)
  args_string = args.join('-')
  args_string.empty? ? nil : Digest::SHA1.hexdigest(args_string)
end

- (Object?) retry_job_delegate

This method is abstract.

Specify another resque job (module or class) to delegate retry duties to upon failure

Returns:

  • (Object, nil)

    class or module if delegate on failure, otherwise nil



146
147
148
# File 'lib/resque/plugins/retry.rb', line 146

def retry_job_delegate
  @retry_job_delegate ||= nil
end

- (Fixnum) retry_limit

Maximum number of retrys we can attempt to successfully perform the job

A retry limit of 0 will never retry. A retry limit of -1 or below will retry forever.

Returns:

  • (Fixnum)


97
98
99
# File 'lib/resque/plugins/retry.rb', line 97

def retry_limit
  @retry_limit ||= 1
end

- (Boolean) retry_limit_reached?

Test if the retry limit has been reached

Returns:

  • (Boolean)


297
298
299
300
301
302
303
304
305
# File 'lib/resque/plugins/retry.rb', line 297

def retry_limit_reached?
  if retry_limit == 0
    true
  elsif retry_limit > 0
    true if retry_attempt >= retry_limit
  else
    false
  end
end

- (Number) sleep_after_requeue

This method is abstract.

Number of seconds to sleep after job is requeued

Returns:

  • (Number)

    number of seconds to sleep



135
136
137
# File 'lib/resque/plugins/retry.rb', line 135

def sleep_after_requeue
  @sleep_after_requeue ||= 0
end

- (Object) try_again(exception, *args)

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Retries the job



336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
# File 'lib/resque/plugins/retry.rb', line 336

def try_again(exception, *args)
  log_message 'try_again', args, exception
  # some plugins define retry_delay and have it take no arguments, so rather than break those,
  # we'll just check here to see whether it takes the additional exception class argument or not
  temp_retry_delay = ([-1, 1].include?(method(:retry_delay).arity) ? retry_delay(exception.class) : retry_delay)

  retry_in_queue = retry_job_delegate ? retry_job_delegate : self
  log_message "retry delay: #{temp_retry_delay} for class: #{retry_in_queue}", args, exception

  # remember that this job is now being retried. before_perform_retry will increment
  # this so it represents the retry count, and MultipleWithRetrySuppression uses
  # the existence of this to determine if the job should be sent to the 
  # parent failure backend (e.g. failed queue) or not.  Removing this means
  # jobs that fail before ::perform will be both retried and sent to the failed queue.
  Resque.redis.setnx(redis_retry_key(*args), -1)

  retry_args = retry_args_for_exception(exception, *args)

  if temp_retry_delay <= 0
    # If the delay is 0, no point passing it through the scheduler
    Resque.enqueue(retry_in_queue, *retry_args)
  else
    Resque.enqueue_in(temp_retry_delay, retry_in_queue, *retry_args)
  end

  # remove retry key from redis if we handed retry off to another queue.
  clean_retry_key(*args) if retry_job_delegate

  # sleep after requeue if enabled.
  sleep(sleep_after_requeue) if sleep_after_requeue > 0
end