Metrics: batch_duration vs process_duration

[secsilm]

The document briefly explains the differences between the two

mosec_service_batch_duration_second_bucket shows the duration of dynamic batching for each connection in each stage (starts from receiving the first task).

mosec_service_process_duration_second_bucket shows the duration of processing for each connection in each stage (including the IPC time but excluding the mosec_service_batch_duration_second_bucket).

But I'm still not very clear about the specific stages they represent. Assuming we only have one stage, does batch_duration refer to the time from receiving the first request to returning all requests? Then what time does process_duration include? What is the relationship between them?

Thank you!

[kemingy]

• batch_duration: gathering tasks for one batch (count start from receiving the 1st task)
• process_duration: from sending the batch to receiving the results of the batch (including IPC time)

You can check the concept.

Assuming we only have one stage, does batch_duration refer to the time from receiving the first request to returning all requests? Then what time does process_duration include? What is the relationship between them?

requests => batched requests => sending to worker => worker process => receiving from worker => send results to clients

batch_duration + process_duration is roughly (could be less if there are too many requests waiting to be processed) the time spent inside the service.

[secsilm]

Okay, I seem to understand. The batch_duration refers to the time taken to bundle the requests together, while the process_duration refers to the time taken to process that batch. The actual inference time of the model is included in the process_duration.

Am I understanding it correctly?

[kemingy]

Yes, your understanding is correct. process_duration includes model inference and IPC time.