remoteproc: do cache invalidation before reading rsc_table status #500

iuliana-prodan · 2023-07-12T11:42:48Z

Do a cache invalidation before reading the resource table's status since this ca be in a cacheable region.

iuliana-prodan · 2023-07-12T12:02:24Z

cc @carlocaione @arnopo

iuliana-prodan · 2023-07-12T15:05:25Z

Changes since v1:

make cache invalidation optional based on VIRTIO_CACHED_RSC_TABLE;
add VIRTIO_CACHED_RSC_TABLE as option.

tnmysh · 2023-07-12T15:35:31Z

lib/remoteproc/remoteproc_virtio.c

@@ -40,6 +41,11 @@ static unsigned char rproc_virtio_get_status(struct virtio_device *vdev)
 	rpvdev = metal_container_of(vdev, struct remoteproc_virtio, vdev);
 	vdev_rsc = rpvdev->vdev_rsc;
 	io = rpvdev->vdev_rsc_io;
+
+#ifdef VIRTIO_CACHED_RSC_TABLE


Can we move #ifdef part to relevant header file ?
May be we can introduce function to check if cache is on or not and then based on that we can decide if we need to call metal_cache_invalidate or not.

So far, in OpenAMP there are 2 ways of doing cache invalidation:

rpmsg: https://github.com/OpenAMP/open-amp/blob/main/lib/rpmsg/rpmsg_virtio.c#L138

virtqueue: https://github.com/OpenAMP/open-amp/blob/main/lib/virtio/virtqueue.c#L32

I've chosen the first one.
You're suggesting the second one - just to put the defines in a header, not in .c file, right?

May be we can introduce function to check if cache is on or not and then based on that we can decide if we need to call metal_cache_invalidate or not.

That's way too wasteful IMO, you are going to have a huge hit in performance if for every cache op you are going to check whether the cache is on or not.

To make this nicer we can do what we did here:

open-amp/lib/virtio/virtqueue.c

Lines 32 to 38 in 7f90610

#ifdef VIRTIO_CACHED_VRINGS

#define VRING_FLUSH(x) metal_cache_flush(&x, sizeof(x))

#define VRING_INVALIDATE(x) metal_cache_invalidate(&x, sizeof(x))

#else

#define VRING_FLUSH(x) do { } while (0)

#define VRING_INVALIDATE(x) do { } while (0)

#endif /* VIRTIO_CACHED_VRINGS */

@carlocaione yes, will do like this.

@carlocaione yes, will do like this.

bonus point if you can clean up these two ways of doing the same thing 😜 (in a different PR)

Thanks. Yes second way works.

cmake/options.cmake

iuliana-prodan · 2023-07-13T11:14:13Z

Changes since v2:

use defines, added in header, for cache flush/invalidate functions.

LE: the checkpatch issue is a false-positive.

carlocaione

LGTM but looking forward to the cleanup PR.

lib/include/openamp/remoteproc_virtio.h

arnopo · 2023-07-13T14:27:50Z

Please also update https://github.com/OpenAMP/open-amp/blob/main/README.md

iuliana-prodan · 2023-07-13T16:12:46Z

Changes since v3:

fix checkpatch warning;
update README in new commit.

carlocaione · 2023-07-13T16:46:33Z

lib/include/openamp/remoteproc_virtio.h

+#define RSC_TABLE_FLUSH(x) \
+	({void *_x = (x); \
+	metal_cache_flush(&_x, sizeof(_x)); })


Are you sure this is doing what you think it is doing? Looking at it it looks like you are flushing the pointer here, not the actual struct also with a wrong sizeof().

You're right. I've fixed it.
Also, to pass checkpatch I've added 2 arguments for RSC_TABLE_FLUSH and RSC_TABLE_INVALIDATE

iuliana-prodan · 2023-07-13T22:56:50Z

Changes since v4:

in order to fix checkpatch warnings I've added 2 arguments for RSC_TABLE_FLUSH and RSC_TABLE_INVALIDATE,

lib/remoteproc/remoteproc_virtio.c

iuliana-prodan · 2023-07-14T10:22:37Z

Changes since v5:

call RSC_TABLE_INVALIDATE with sizeof(struct..);
do not use address of the pointer for metal_cache_flush/metal_cache_invalidate;

carlocaione · 2023-07-14T10:28:04Z

lib/remoteproc/remoteproc_virtio.c

+
+	RSC_TABLE_INVALIDATE(vdev_rsc, sizeof(struct fw_rsc_vdev));
+


This is fine now, thanks.

Shouldn't we also RSC_TABLE_FLUSH in rproc_virtio_set_status() though?

I've added RSC_TABLE_FLUSH/RSC_TABLE_INVALIDATE for status, dfeatures and gfeatures.
I don't know if I should do the same for read_config/write_config.
What do you think?
For Zephyr, config is not used: https://github.com/zephyrproject-rtos/zephyr/blob/main/lib/open-amp/resource_table.h#L50

@arnopo I'm not familiar with remoteproc at all. What are the structs that are actually r/w from both sides that needs to be flushed / invalidated?

@iuliana-prodan, @carlocaione

Yes we need cache operation even in read config and write config as well.
One more question, should we perform only flush or invalidate and flush both during writing resource table? Or it is really platform dependent?

"Flush" and "invalidate" should be sufficient if the structures are aligned on a cache line. We can improve this in the future if needed.

Right @TanmayShah-xilinx,
The cache shoule also be managed in rproc_virtio_read_config andrproc_virtio_read_config

iuliana-prodan · 2023-07-17T08:48:10Z

Changes since v6:

do flush/invalidate also for write/read config;

carlocaione

In general this LGTM. I'm a bit concerned about the alignment of these structs. If the structs are not aligned to the cache line, the invalidation will corrupt neighboring structs in memory. It really is a silent killer.

See for example zephyrproject-rtos/zephyr#59456

carlocaione · 2023-07-17T08:53:53Z

lib/remoteproc/remoteproc_virtio.c

+
+	RSC_TABLE_INVALIDATE(vdev_rsc, sizeof(struct fw_rsc_vdev));
+


nit: can you keep the newlines consistent among all the RSC_TABLE_* calls?

Do a cache invalidation before reading the resource table's info since this ca be in a cacheable region. Make this optional, based on VIRTIO_CACHED_RSC_TABLE. Signed-off-by: Iuliana Prodan <[email protected]>

Add info for WITH_DCACHE_RSC_TABLE option. Signed-off-by: Iuliana Prodan <[email protected]>

iuliana-prodan · 2023-07-17T11:16:28Z

Changes since v7:
keep the newlines consistent among all the RSC_TABLE_* calls;

arnopo

LGTM

tnmysh

LGTM.

iuliana-prodan mentioned this pull request Jul 12, 2023

Need to do a dcache invalidation when reading status from resource table #499

Closed

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 146f72b to ddf6fb1 Compare July 12, 2023 11:46

arnopo requested review from carlocaione, edmooring, arnopo and tnmysh July 12, 2023 12:55

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from ddf6fb1 to a2b48ce Compare July 12, 2023 15:03

iuliana-prodan closed this Jul 12, 2023

iuliana-prodan reopened this Jul 12, 2023

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from a2b48ce to 125b19d Compare July 12, 2023 15:12

tnmysh reviewed Jul 12, 2023

View reviewed changes

cmake/options.cmake Show resolved Hide resolved

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 125b19d to 364ebc5 Compare July 13, 2023 11:12

carlocaione approved these changes Jul 13, 2023

View reviewed changes

arnopo reviewed Jul 13, 2023

View reviewed changes

lib/include/openamp/remoteproc_virtio.h Outdated Show resolved Hide resolved

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 364ebc5 to 45c9ec4 Compare July 13, 2023 16:11

carlocaione suggested changes Jul 13, 2023

View reviewed changes

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch 2 times, most recently from cd05c20 to 3f1dd55 Compare July 13, 2023 22:45

carlocaione suggested changes Jul 14, 2023

View reviewed changes

lib/remoteproc/remoteproc_virtio.c Outdated Show resolved Hide resolved

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 3f1dd55 to 4614bdf Compare July 14, 2023 10:06

carlocaione suggested changes Jul 14, 2023

View reviewed changes

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 4614bdf to ecd7ee3 Compare July 14, 2023 13:38

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from ecd7ee3 to 9c5537d Compare July 17, 2023 08:45

carlocaione approved these changes Jul 17, 2023

View reviewed changes

iuliana-prodan added 2 commits July 17, 2023 14:14

remoteproc: do cache invalidation before reading rsc_table info

af40a57

Do a cache invalidation before reading the resource table's info since this ca be in a cacheable region. Make this optional, based on VIRTIO_CACHED_RSC_TABLE. Signed-off-by: Iuliana Prodan <[email protected]>

README: add new cache option for resource table

25ffebe

Add info for WITH_DCACHE_RSC_TABLE option. Signed-off-by: Iuliana Prodan <[email protected]>

iuliana-prodan force-pushed the fix-rsc-table-cache-invalidate branch from 9c5537d to 25ffebe Compare July 17, 2023 11:15

carlocaione approved these changes Jul 17, 2023

View reviewed changes

arnopo approved these changes Jul 17, 2023

View reviewed changes

iuliana-prodan mentioned this pull request Jul 17, 2023

Add support for HiFi4 DSP on openamp_rsc_table sample zephyrproject-rtos/zephyr#60455

Merged

arnopo requested a review from tnmysh July 17, 2023 14:47

tnmysh approved these changes Jul 17, 2023

View reviewed changes

arnopo merged commit 5891cb4 into OpenAMP:main Jul 17, 2023
2 checks passed

arnopo added this to the Release V2023.10 milestone Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

remoteproc: do cache invalidation before reading rsc_table status #500

remoteproc: do cache invalidation before reading rsc_table status #500

iuliana-prodan commented Jul 12, 2023

iuliana-prodan commented Jul 12, 2023

iuliana-prodan commented Jul 12, 2023 •

edited

Loading

tnmysh Jul 12, 2023 •

edited

Loading

iuliana-prodan Jul 12, 2023 •

edited

Loading

carlocaione Jul 12, 2023

iuliana-prodan Jul 12, 2023

carlocaione Jul 12, 2023

tnmysh Jul 12, 2023

iuliana-prodan commented Jul 13, 2023 •

edited

Loading

carlocaione left a comment

arnopo commented Jul 13, 2023

iuliana-prodan commented Jul 13, 2023

carlocaione Jul 13, 2023

iuliana-prodan Jul 13, 2023

iuliana-prodan commented Jul 13, 2023

iuliana-prodan commented Jul 14, 2023

carlocaione Jul 14, 2023

iuliana-prodan Jul 14, 2023

carlocaione Jul 14, 2023

tnmysh Jul 14, 2023

arnopo Jul 17, 2023

arnopo Jul 17, 2023

iuliana-prodan commented Jul 17, 2023

carlocaione left a comment

carlocaione Jul 17, 2023

iuliana-prodan Jul 17, 2023

iuliana-prodan commented Jul 17, 2023

arnopo left a comment

tnmysh left a comment

	#ifdef VIRTIO_CACHED_VRINGS
	#define VRING_FLUSH(x) metal_cache_flush(&x, sizeof(x))
	#define VRING_INVALIDATE(x) metal_cache_invalidate(&x, sizeof(x))
	#else
	#define VRING_FLUSH(x) do { } while (0)
	#define VRING_INVALIDATE(x) do { } while (0)
	#endif /* VIRTIO_CACHED_VRINGS */

remoteproc: do cache invalidation before reading rsc_table status #500

remoteproc: do cache invalidation before reading rsc_table status #500

Conversation

iuliana-prodan commented Jul 12, 2023

iuliana-prodan commented Jul 12, 2023

iuliana-prodan commented Jul 12, 2023 • edited Loading

tnmysh Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

iuliana-prodan Jul 12, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iuliana-prodan commented Jul 13, 2023 • edited Loading

carlocaione left a comment

Choose a reason for hiding this comment

arnopo commented Jul 13, 2023

iuliana-prodan commented Jul 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iuliana-prodan commented Jul 13, 2023

iuliana-prodan commented Jul 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iuliana-prodan commented Jul 17, 2023

carlocaione left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iuliana-prodan commented Jul 17, 2023

arnopo left a comment

Choose a reason for hiding this comment

tnmysh left a comment

Choose a reason for hiding this comment

iuliana-prodan commented Jul 12, 2023 •

edited

Loading

tnmysh Jul 12, 2023 •

edited

Loading

iuliana-prodan Jul 12, 2023 •

edited

Loading

iuliana-prodan commented Jul 13, 2023 •

edited

Loading