[Serve][Doc] Update metrics & log doc (#34222)

Update the logging & metrics for the 2.4. change.

Co-authored-by: angelinalg <[email protected]>

doc/source/serve/doc_code/monitoring/custom_metric_snippet.py

@@ -1,6 +1,6 @@
 # __start__
 from ray import serve
-from ray.util import metrics
+from ray.serve import metrics
 
 import time
 import requests
@@ -13,9 +13,9 @@ def __init__(self):
         self.my_counter = metrics.Counter(
             "my_counter",
             description=("The number of odd-numbered requests to this deployment."),
-            tag_keys=("deployment",),
+            tag_keys=("model",),
         )
-        self.my_counter.set_default_tags({"deployment": "MyDeployment"})
+        self.my_counter.set_default_tags({"model": "123"})
 
     def __call__(self):
         self.num_requests += 1
@@ -32,6 +32,5 @@ def __call__(self):
 
     # __end__
     break
-
 response = requests.get("http://localhost:8000/")
 assert response.status_code == 200

doc/source/serve/production-guide/monitoring.md

@@ -71,13 +71,12 @@ Run this deployment using the `serve run` CLI command:
 ```console
 $ serve run monitoring:say_hello
 
-2022-08-10 22:58:55,963	INFO scripts.py:294 -- Deploying from import path: "monitoring:say_hello".
-2022-08-10 22:58:57,886	INFO worker.py:1481 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265.
-(ServeController pid=63881) INFO 2022-08-10 22:58:59,365 controller 63881 http_state.py:129 - Starting HTTP proxy with name 'SERVE_CONTROLLER_ACTOR:SERVE_PROXY_ACTOR-1252fc7fbbb16ca6a80c45cbb5fe4ef182030b95aa60b62604151168' on node '1252fc7fbbb16ca6a80c45cbb5fe4ef182030b95aa60b62604151168' listening on '127.0.0.1:8000'
-The new client HTTP config differs from the existing one in the following fields: ['location']. The new HTTP config is ignored.
-(ServeController pid=63881) INFO 2022-08-10 22:58:59,999 controller 63881 deployment_state.py:1232 - Adding 1 replicas to deployment 'SayHello'.
-(HTTPProxyActor pid=63883) INFO:     Started server process [63883]
-2022-08-10 22:59:00,979	SUCC scripts.py:315 -- Deployed successfully.
+2023-04-10 15:57:32,100	INFO scripts.py:380 -- Deploying from import path: "monitoring:say_hello".
+[2023-04-10 15:57:33]  INFO ray._private.worker::Started a local Ray instance. View the dashboard at http://127.0.0.1:8265 
+(ServeController pid=63503) INFO 2023-04-10 15:57:35,822 controller 63503 deployment_state.py:1168 - Deploying new version of deployment SayHello.
+(HTTPProxyActor pid=63513) INFO:     Started server process [63513]
+(ServeController pid=63503) INFO 2023-04-10 15:57:35,882 controller 63503 deployment_state.py:1386 - Adding 1 replica to deployment SayHello.
+2023-04-10 15:57:36,840	SUCC scripts.py:398 -- Deployed Serve app successfully.
 ```
 
 `serve run` prints a few log messages immediately. Note that a few of these messages start with identifiers such as
@@ -97,11 +96,14 @@ curl -X GET http://localhost:8000/
 This causes the HTTP proxy and deployment replica to print log statements to the terminal running `serve run`:
 
 ```console
-(HTTPProxyActor pid=63883) INFO 2022-08-10 23:10:08,005 http_proxy 127.0.0.1 http_proxy.py:315 - GET / 200 2.4ms
-(ServeReplica:SayHello pid=63885) INFO 2022-08-10 23:10:08,004 SayHello SayHello#JYbzqP monitoring.py:15 - Hello world!
-(ServeReplica:SayHello pid=63885) INFO 2022-08-10 23:10:08,004 SayHello SayHello#JYbzqP replica.py:482 - HANDLE __call__ OK 0.2ms
+(ServeReplica:SayHello pid=63520) INFO 2023-04-10 15:59:45,403 SayHello SayHello#kTBlTj HzIYOzaEgN / monitoring.py:16 - Hello world!
+(ServeReplica:SayHello pid=63520) INFO 2023-04-10 15:59:45,403 SayHello SayHello#kTBlTj HzIYOzaEgN / replica.py:527 - __CALL__ OK 0.5ms
 ```
 
+:::{note}
+Log messages include the logging level, timestamp, deployment name, replica tag, request ID, route, file name, and line number.
+:::
+
 Find a copy of these logs at `/tmp/ray/session_latest/logs/serve/`. You can parse these stored logs with a logging stack such as ELK or [Loki](serve-logging-loki) to be able to search by deployment or replica.
 
 Serve supports [Log Rotation](ray-log-rotation) of these logs through setting the environment variables `RAY_ROTATION_MAX_BYTES` and `RAY_ROTATION_BACKUP_COUNT`.
@@ -341,7 +343,7 @@ The emitted logs include:
 ```
 # HELP ray_my_counter The number of odd-numbered requests to this deployment.
 # TYPE ray_my_counter gauge
-ray_my_counter{..., deployment="MyDeployment"} 5.0
+ray_my_counter{..., deployment="MyDeployment",model="123",replica="MyDeployment#rUVqKh"} 5.0
 ```
 
 See the [Ray Metrics documentation](ray-metrics) for more details, including instructions for scraping these metrics using Prometheus.