Class: Gitlab::Counters::FlushStaleCounterIncrementsWorker

Inherits:
Object
  • Object
show all
Includes:
ApplicationWorker, LimitedCapacity::Worker
Defined in:
app/workers/gitlab/counters/flush_stale_counter_increments_worker.rb

Constant Summary collapse

MAX_RUNNING_JOBS = 
1
BATCH_LIMIT = 
1000
ID_RANGES =

We hardcoded these IDs here because the FlushCounterIncrementsWorker was disabled in September 2024 after an incident. In March 2025, we reenabled the worker. These are the leftover entries on gitlab.com that still need to be flushed. Afterwards, we can remove this job.

{
  ProjectDailyStatistic => {
    initial_start_id: 3847138140,
    end_id: 4074016739
  }
}.freeze

Constants included from ApplicationWorker

ApplicationWorker::LOGGING_EXTRA_KEY, ApplicationWorker::SAFE_PUSH_BULK_LIMIT

Constants included from Loggable

Loggable::ANONYMOUS

Constants included from WorkerAttributes

WorkerAttributes::DEFAULT_DATA_CONSISTENCY, WorkerAttributes::DEFAULT_DATA_CONSISTENCY_PER_DB, WorkerAttributes::DEFAULT_DEFER_DELAY, WorkerAttributes::LOAD_BALANCED_DATA_CONSISTENCIES, WorkerAttributes::NAMESPACE_WEIGHTS, WorkerAttributes::VALID_DATA_CONSISTENCIES, WorkerAttributes::VALID_RESOURCE_BOUNDARIES, WorkerAttributes::VALID_URGENCIES

Instance Method Summary collapse

Methods included from LimitedCapacity::Worker

#perform, #remove_failed_jobs, #report_prometheus_metrics

Methods included from Loggable

#build_structured_payload

Methods included from SidekiqVersioning::Worker

#job_version

Methods included from WorkerContext

#with_context

Instance Method Details

#max_running_jobs ⇒ Object 



63
64
65
 
# File 'app/workers/gitlab/counters/flush_stale_counter_increments_worker.rb', line 63

def max_running_jobs
  MAX_RUNNING_JOBS
end

#perform_work ⇒ Object 



34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
 
# File 'app/workers/gitlab/counters/flush_stale_counter_increments_worker.rb', line 34

def perform_work
  return unless ::Gitlab.com_except_jh? # rubocop:disable Gitlab/AvoidGitlabInstanceChecks -- we need to check on which instance this happens

  # Process up to 100 batches per job execution as a compromise between
  # performance (avoiding too many small jobs) and efficiency (preventing
  # jobs from running too long and potentially timing out)
  100.times do
    ID_RANGES.each do |model, attributes|
      min_id = start_id(model)
      remaining_work = [(attributes[:end_id] - min_id), 0].max
      last_id = flush_stale_for_model(model, min_id, attributes[:end_id])
      # we need to add +1 here, because otherwise, we'd process the last record twice.
      update_start_id(model, last_id + 1)
      return if remaining_work == 0 # rubocop:disable Lint/NonLocalExitFromIterator -- return if we don't have anymore work to do
    end
  end
end

#remaining_work_count ⇒ Object 



52
53
54
55
56
57
58
59
60
61
 
# File 'app/workers/gitlab/counters/flush_stale_counter_increments_worker.rb', line 52

def remaining_work_count
  # iterate through all models and see, if there is still work to do
  remaining_work = 0
  ID_RANGES.each do |model, attributes|
    return remaining_work if remaining_work > 0

    remaining_work = [(attributes[:end_id] - start_id(model)), 0].max
  end
  remaining_work
end