Skip to content

Commit

Permalink
VimPerformanceState#capture - don't capture old deleted records
Browse files Browse the repository at this point in the history
VimPerformanceState records only capture active objects and those deleted after this hour's time period
This cuts down on the number of records processed. (by a lot)

Numbers
=======

```
VimPerformanceState.first ; ExtManagementSystem.first ; ContainerImage.first ; Container.first ; nil
bookend(gc:true) { Metric::Rollup.rollup_child_metrics(ContainerImage.find(627), Time.now.utc.iso8601, "historical", :containers) }
```

Numbers run for rollup_child_metrics for a parent object that hasn't been created yet.
If the parent object has been run, then we don't need a capture, and we'll see no difference

|       ms |       bytes |   objects |query |  qry ms |     rows |` comments`
|      ---:|         ---:|       ---:|  ---:|     ---:|      ---:|  ---
|  2,131.6 | 77,558,748* | 1,070,108 |   13 | 1,760.5 |   13,680 |`rollup-create-vps-before`
|  1,632.5 |  5,139,458* |   393,268 |   13 | 1,517.7 |      174 |`rollup-create-vps-after`

* Memory usage does not reflect 175,949 freed objects.
* Memory usage does not reflect 59,056 freed objects.

We've already merged code to reduce the number of records returned from VimPerformanceState#containers
So removing archived from the VimPerformanceState will reduce the size of the query sent to fetch containers.
This will reduce the query ms (simpler/smaller query), but this does not reduce the number of records returned
And VPS#state_data went from 167,234 => 1,752 characters

The savings there is due to the fact that capture does not download all the archived records.

So this commit saves all the id downloads, and the commit from the fetch pr (just merged a few commits back),
reduces all archived records fetch as part of the capture process
  • Loading branch information
kbrock committed Aug 1, 2023
1 parent 65d0a84 commit 3ddd7d6
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 2 deletions.
4 changes: 3 additions & 1 deletion app/models/vim_performance_state.rb
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,9 @@ def fetch_assoc_records(resource, assoc)
end
return false unless resource.respond_to?(method)

resource.send(method)
records = resource.send(method)
records = records.not_archived_before(timestamp) if records.try(:klass).respond_to?(:not_archived_before)
records
end

def capture_parent_cluster
Expand Down
16 changes: 15 additions & 1 deletion spec/models/vim_performance_state_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,17 @@
end
end

describe "#capture_assoc_ids" do
it "captures running containers" do
container_image = FactoryBot.create(:container_image)
containers = create_past_present_future(:container, {:container_image => container_image})

state = VimPerformanceState.capture(container_image)
expect(state.get_assoc(:containers).map(&:to_i)).to match_array(containers.map(&:id))
expect(state.containers.map(&:id)).to match_array(containers.map(&:id))
end
end

describe "#containers" do
let(:capture_time) { 5.hours.ago.utc.beginning_of_hour }

Expand Down Expand Up @@ -135,10 +146,13 @@

it "fetches active container objects" do
container_image = FactoryBot.create(:container_image, :created_on => 6.hours.ago)
containers = FactoryBot.create_list(:container, 1, :container_image => container_image)
containers = create_past_present_future(:container, {:container_image => container_image})

results = fetch_records(container_image, :containers, capture_time)
expect(results).to eq(containers)

state = VimPerformanceState.capture(container_image)
expect(state.get_assoc(:containers).map(&:to_i)).to match_array(containers.map(&:id))
end
end

Expand Down

0 comments on commit 3ddd7d6

Please sign in to comment.