Magento 2: Cache key id prefix

[valguss]

• Bug fix #…?

• New feature?

• BC breaks?

• Tests added?

• Docs added?

  Please, regenerate docs by running next command:
  $ php bin/docgen
  

[valguss]

@antonmedv This is an updated version of this #3388

It now does all the operations on the server and should be a bit easier to see what's going on

The general gist is that during deployment, we copy the configuration and update the cache keys for the new deployment so that it uses it own totally separate cache to the current live site. It then moves the updated configuration in place to be the live configuration ready for next deployment

[antonmedv]

Looks good to me. 👍🏻

[antonmedv]

@peterjaap please review

[valguss]

Just rebasing the docs again

[antonmedv]

@peterjaap merging?

[valguss]

Rebased docs again

[peterjaap]

@valguss I'm still unsure what problem this exactly solves. I understand what it does, but I don't know which issues this fixes. So I'm hesitant to include new code for something that doesn't effect everyone.

[valguss]

Ahh, The issue is based from here davidalger/capistrano-magento2#151
Essentially, it's a way to run a deployed instance of magento on a totally fresh cache.
The main issue we see is that on high traffic sites where this is not implemented, the code generation part of magento setup:di:compile uses the currently deployed cache rather than a fresh cache which can cause it to incorrectly compile the site. When this happens, another redeploy is required to compile correctly

Hope that explains it more

Cheers

Tom

[peterjaap]

@valguss so this is in the scenario where you run setup:di:compile on the target production machine instead of in a build pipeline?

[valguss]

So it runs in the pipeline but because the cache in redis would be shared between the current version and the deploying version, the deploying version wouldn't necessarily generate all the classes required to run. We see this happening where we add new modules and the dependency injection is not created because the production cache dosen't know about the new module.

[peterjaap]

@valguss ok so it does run in a pipeline, but it runs on the production machine and not on an isolated build runner?

[valguss]

That's correct. Most of the Magento commands run in the deployment folder on the deployment server

[peterjaap]

In that case I'm not in favor of merging this into the default recipe since running those commands on the production server instead of a build pipeline isn't standard (or advised) practice. What do you think @schmengler / @Schrank ?

[valguss]

Not sure I follow. All the magent2 commands in the recipe run on a target server before being pushed live? Maybe I've confused the situation

[peterjaap]

@valguss ok I misunderstood, this recipe does indeed run every command straight on the production server (I work on another one that does a push-only approach from a runner instance, hence the confusion).

I've added some comments.

[valguss]

Thanks @peterjaap I'll work through those and get another push later today. Appreciate the input :)

[schmengler]

@valguss what is the reason of manipulating env.php like this instead of using bin/magento setup:config:set? Looks overly complicated to me but I'm probably missing something

[valguss]

It's like this because the env.php is shared across deployments. If we set the config update during deployment then the currently deployed site can also use the updated value and pollute the cache

[schmengler]

it could be set after the symlink has been switched

[schmengler]

Oh, I should have read the whole conversation

the code generation part of magento setup:di:compile uses the currently deployed cache rather than a fresh cache which can cause it to incorrectly compile the site. When this happens, another redeploy is required to compile correctly

I'm only using artifact deployment where setup:di:compile is not executed on the target server, so as long as this is not affected by default, I'm fine with whatever helps your use case

[valguss]

Unfortunately, it needs to be set before then as the fresh cache is required during the Magento compile section. Without this change, the currently live and deploying codebases share the config and cache during magento:compile. It's specifically this section where having a fresh non-polluted cache is essential for the compilation to work correctly

[valguss]

Now updated to use a config flag to enable

[antonmedv]

@peterjaap @schmengler let's merge it and release?

[peterjaap]

@valguss could you move the if check if (get('use_redis_cache_id')) { to around where the hook is defined?

That way, the task itself doesn't start if use_redis_cache_id is not true. Now, it does start and finishes immediatly, showing up in the task log.

So:

if (get('use_redis_cache_id')) {
    after('deploy:shared', 'magento:set_cache_prefix');
}

and

if (get('use_redis_cache_id')) {
    after('deploy:magento', 'magento:cleanup_cache_prefix');
}

[valguss]

@peterjaap That's now added. Cheers

[peterjaap]

Ok, LGTM!

[guvra]

FYI, I don't mind this feature because it's optional, but it will break your env.php file if you follow good practices, such as using preload keys

cf. https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/cache/redis/redis-pg-cache.html#redis-preload-feature

'cache' => [
    'frontend' => [
        'default' => [
            'id_prefix' => '061_',
            'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
            'backend_options' => [
                'preload_keys' => [
                    '061_EAV_ENTITY_TYPES',
                    '061_GLOBAL_PLUGIN_LIST',
                    '061_DB_IS_UP_TO_DATE',
                    '061_SYSTEM_DEFAULT',
                ],
        ],
        'page_cache' => [
            'id_prefix' => '061_'
        ]
    ]
]

The prefix of the preload keys must match the cache id prefix so it's a very bad idea to change it during each deployment.

[valguss]

Thanks @guvra i'll work on updating the repload keys as well for a future update

[schmengler]

By the way, this is also helpful in edge cases if you don't compile on the target server, for example when you deploy an update where a cache type was removed:

[staging] run /usr/bin/php7.4 /.../releases/.../src/bin/magento cache:flush
[staging] err In ClassReader.php line 34:
[staging] err Class Vertex\Tax\Model\Cache\Type does not exist