Docker friendly changes

[di-repo]

Hi, and thanks for the wonderful module.
I've spent some time making the module work on Docker, and I can confirm it's working fine. But I've stumbled upon some issues. The module makes two php calls, which obviously can't work inside a node container, and I don't find it a good idea to use mixed purpose containers (php+node). Fortunately, the calls are really simple so I've just run them and saved the output:
php -r "require 'app/vendor/autoload.php'; foreach ((new \Magento\Framework\Component\ComponentRegistrar)->getPaths('module') as \$m) echo \$m.PHP_EOL;" > modules.cfg
php -r "echo json_encode( require 'app/etc/env.php' ?? []);" > settings.cfg
This can be easily run inside the php container and stored in a location of your choice. After that, I've just rewritten the cache-clean.js to use the two saved files. I am not fluent enough with closurescript to do a PR, but it will be a nice option to check first for a static file with configuration, and if not present then proceed with php calls. This will allow some further customizations like having a more fine-grained filter for watch locations. I don't think I need to watch for files inside vendor, or at least not for all of them.

[Vinai]

That is a great idea! I'll definitely implement this.
One nice side effect I like is that it also makes startup faster, shelling out to PHP takes too long, but so far I haven't found a nice solution.
There are a couple of thoughts I'd like to decide on before starting:

• pass file names to script as arguments?
• convention (name and location)?
• how to handle new module while the watcher is running (watch config file and reload)
• There is a third call to PHP to list the theme locations, just like the module locations.
  Those should be combined in one file.

Other ideas:

• supply a Magento plugin in a module (not enabled automatically) to regenerate the list when module:disable or module:unable is executed.

[Vinai]

Just to cross reference because it's also about Docker: #4

[di-repo]

Thanks for the quick answer, glad you like the idea.
Atm your module is supplied as magento module so the call to generate the config file can be a bin/magento one.

• I don't see any benefit in passing script names as arguments. I prefer to think about this files as a local caching mechanism.
• Probably I would create only one file and put it in module space. why not config-cache.json
• I can't think of a graceful way to achieve this, but it's a fair compromise, after all when you add new module you can restart the tool/rerun config cache. I don't expect someone to run this in an environment different than dev. Probably you can achieve the same via a custom API call to Magento, that will return only modules and themes location, but I don't find it a particularly good idea. Too complicated.
• I have missed the third call but from your words, I understand it won't be an issue to implement.

I think your other idea will fix the third point if you intercept enable disable you can just rewrite the config.json and watch for it in the cache-clean app.

[Vinai]

Thanks for your additional feedback.
Right now I'm thinking to write a stand-alone PHP shell script to generate the config file. This script will be supplied as part of the mage2tv/magento-cache-clean package in vendor/bin.

Then, a separate - optional - package for a Magento module can supply the bin/magento command to generate the file, and an option to automate the rebuilding of the cache file when a new module is added.

I don't want to write into the module directory really. non-composer code writing to vendor/ dosn't feel right. I'd rather keep the cache file in var/ or maybe the Magento base dir.

I'm itching to build this right now, but unfortunately it will have to wait until after https://magetestfest.com/ - sorry.

[di-repo]

This will be OK, simple and working as all code should be. Maybe we will meet at the test fest ;)

[Vinai]

Would be great to see you there - please say hi if you are there!

[Vinai]

Hi @dimitar-ivanov-tryzens I've implemented a first prototype.

I would like to ask you to check it out and provide feedback, how it might be improved.
The prototype can be installed with the following commands:

composer remove --dev mage2tv/magento-cache-clean
composer require --dev mage2tv/magento-cache-clean:dev-app-config-file

If you installed the utility globally, please use composer global instead. Leave out the --dev option if you installed it as a regular dependency.

The PHP script to generate the dump file can be found in the package at
vendor/mage2tv/bin/generate-cache-clean-config.php.
For a global install that would be $HOME/.composer/vendor/mage2tv/bin/generate-cache-clean-config.php.

The script will create the file var/cache-clean-config.json.
If that file is present, cache-clean.js will parse it to get the app config and the module and theme directories.

If the file is not present, cache-clean.js will get the required information via php.

When running the watcher with debug output enabled (cache-clean.js -w -vv) you can see how it is accessing the information, by printing for example Listing themes from var/cache-clean-config.json.

Please let me know if it works for you, solves the intended problem and how it might be improved.
Thank you!

[peterjaap]

@Vinai almost, the PHP script path is vendor/mage2tv/magento-cache-clean/bin/generate-cache-clean-config.php :)

So I gave it a whirl and got this;

 ✘  ~/development/workspace/client/magento2   master ●  vendor/bin/cache-clean.js -w -vv 
Release 0.0.35 sponsored by https://www.mage2.tv

Magento dir /home/peterjaap/development/workspace/client/magento2
Listing modules from var/cache-clean-config.json
Listing themes from var/cache-clean-config.json
[ERROR] ENOENT: no such file or directory, watch '/data/client/magento2/vendor/magento/theme-frontend-blank'

I then opened var/cache-clean-config.json and removed all occurences of /data/client/magento2/ so all paths would be relative. Then ran it again and voila;

Release 0.0.35 sponsored by https://www.mage2.tv

Magento dir /home/peterjaap/development/workspace/client/magento2
Listing modules from var/cache-clean-config.json
Watching module module-authorization
Watching module module-customer
Watching module module-require-js
Watching module module-backend
Watching module module-sales-sequence
Watching module module-bundle
Watching module module-sales
Watching module module-catalog-inventory
Watching module module-indexer
Watching module module-catalog
Watching module module-cms
Watching module module-email
Watching module module-config
Watching module module-directory
Watching module module-store
Watching module module-media-storage
Watching module module-eav
Watching module module-security
Watching module module-integration
Watching module module-user
Watching module module-ui
Watching module module-theme
Watching module module-rule
Watching module module-variable
Watching module module-widget
Watching module module-catalog-rule
Watching module module-sales-rule
Watching module module-page-cache
Watching module module-checkout
Watching module module-downloadable
Watching module module-rss
Watching module module-newsletter
Watching module module-wishlist
Watching module module-review
Watching module module-reports
Watching module module-tax
Watching module module-quote
Watching module module-payment
Watching module module-import-export
Watching module module-contact
Watching module module-shipping
Watching module module-vault
Watching module module-catalog-import-export
Watching module module-msrp
Watching module module-catalog-url-rewrite
Watching module module-instant-purchase
Watching module module-paypal
Watching module module-configurable-product
Watching module module-cron
Watching module module-core
Watching module module-developer
Watching module module-url-rewrite
Watching module module-search
Watching module module-cookie
Watching module module-gift-message
Watching module module-grouped-product
Watching module module-product-alert
Watching module module-robots
Watching module module-ordermanagement
Watching module module-catalog-search
Watching module module-google-analytics
Watching module module-multishipping
Watching module module-offline-shipping
Watching module module-send-friend
Watching module module-usps
Watching module module-webapi
Watching module module-backup
Watching module module-cms-url-rewrite
Watching module module-deploy
Watching module module-sitemap
Watching module module-translation
Watching module Core
Watching module Login
Watching module Payment
Watching module dotmailer-magento2-extension
Watching module module-kp
Watching module module-community
Watching module module-admin-notification
Watching module module-advanced-pricing-import-export
Watching module module-analytics
Watching module module-authorizenet
Watching module module-braintree
Watching module module-bundle-import-export
Watching module module-cache-invalidate
Watching module module-captcha
Watching module module-catalog-analytics
Watching module module-catalog-rule-configurable
Watching module module-catalog-widget
Watching module module-checkout-agreements
Watching module module-configurable-import-export
Watching module module-configurable-product-sales
Watching module module-currency-symbol
Watching module module-customer-analytics
Watching module module-customer-import-export
Watching module module-dhl
Watching module module-downloadable-import-export
Watching module module-encryption-key
Watching module module-fedex
Watching module module-google-adwords
Watching module module-google-optimizer
Watching module module-grouped-import-export
Watching module module-layered-navigation
Watching module module-marketplace
Watching module module-new-relic-reporting
Watching module module-offline-payments
Watching module module-persistent
Watching module module-product-video
Watching module module-quote-analytics
Watching module module-release-notification
Watching module module-review-analytics
Watching module module-sales-analytics
Watching module module-sales-inventory
Watching module module-sample-data
Watching module module-signifyd
Watching module module-swagger
Watching module module-swatches
Watching module module-swatches-layered-navigation
Watching module module-tax-import-export
Watching module module-ups
Watching module module-version
Watching module module-webapi-security
Watching module module-weee
Watching module module-wishlist-analytics
Watching module common
Watching module security-suite-common
Watching module src
Watching module src
Watching module src
Watching module src
Watching module src
Watching module src
... etc
Listing themes from var/cache-clean-config.json
Watching theme theme-frontend-blank
Watching theme theme-adminhtml-backend
Watching theme theme-frontend-luma
Watching theme client-theme
Monitoring app/etc/config.php for new modules
Listening for hotkeys
Hot-keys for manual cache cleaning:
[c]onfig [b]lock_html [l]ayout [t]ranslate [f]ull_page [v]iew [a]ll

Hot-keys for cleaning static content areas:
[F]rontend [A]dminhtml

Watcher initialized (Ctrl-C to quit)

And it also processes files 🎉

Processing /home/peterjaap/development/workspace/client/magento2/app/design/frontend/Elgentos/client/web/css/source/pages/_catagory_page.less
Processing /home/peterjaap/development/workspace/client/magento2/app/code/client/Shipping/etc/di.xml
Cleaning cache type(s) config
Using :default cache_backend
Reading app config from var/cache-clean-config.json
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/cache, :id_prefix 717_}
Cleaning tag CONFIG
Tag-file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---717_CONFIG

[di-repo]

Hi @Vinai, thanks works perfect!
On the issue @peterjaap is facing, it's rather self-explanatory paths are generated based on the php container file structure in your case /data/mage........ and you are running node on the bare metal. I can think of few solutions here, I don't know how well will filewatcher work with relative paths or if they are converted to absolute from cache-clean script (@Vinai can give us ome insights here I believe). If this works well enough relative paths might be good enough. Just in case you can change paths from php docker to "full real ones".
I'll suggest go full docker, and just spin docker node container to run cache-clean with the same volume and mapping php is using. I am using it in this way and seems to run very well.

[Vinai]

So... I updated the script to use relative paths in the dump file.
Does this till work for you @dimitar-ivanov-tryzens? Is this a helpful change @peterjaap?

[di-repo]

This is totally ok as long as it's not breaking cache-clean.js. I've tested relative paths so far so good.
How I use this with docker?
I've added to docker-compose these lines:

  node:
   image: node:10.15.1-alpine
   volumes:
     - ../www:/var/www/html # or just attach same volumes from php with * and &
   working_dir: /var/www/html

This way node container starts and exits, when you need it you can just call

docker-compose run --rm node vendor/bin/cache-clean.js -w

Same can be achieved just with docker and the following command run from magento root:

docker run -it -v $(pwd):/var/www/html -w /var/www/html node:10.15.1-alpine vendor/bin/cache-clean.js -w

In both examples change paths so they reflect your infrastructure.

[di-repo]

I am happy with those changes and think you can successfully close the ticket.
Sehr gut gemacht, danke!

[Vinai]

Thank you for the feedback @dimitar-ivanov-tryzens!
Before closing the issue I would like to hear what @peterjaap has to say, too.

May I incorporate the docker examples you posted above into the README?

[peterjaap]

@Vinai great, works!!! Ship iiiiit! :)

[peterjaap]

@Vinai hm no wait. It looks like it works, but this is my log;

Processing /home/peterjaap/development/workspace/client/magento2/app/code/Elgentos/ProductionTracking/view/layout/customer_account.xml
Cleaning cache type(s) layout full_page
Using :default cache_backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/cache, :id_prefix 7d7_}
Cleaning tag LAYOUT_GENERAL_CACHE_TAG
Tag-file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---7d7_LAYOUT_GENERAL_CACHE_TAG
Using :page_cache cache backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/page_cache, :id_prefix 7d7_}
Cleaning dir /home/peterjaap/development/workspace/client/magento2/var/page_cache/
Varnish request: #js {:protocol http:, :hostname 127.0.0.1, :port 6081, :method PURGE, :path /, :headers #js {:X-Magento-Tags-Pattern .*}, :timeout 10000}
Processing /home/peterjaap/development/workspace/client/magento2/app/code/Elgentos/ProductionTracking/view/frontend/layout/customer_account.xml
Cleaning cache type(s) layout full_page
Using :default cache_backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/cache, :id_prefix 7d7_}
Cleaning tag LAYOUT_GENERAL_CACHE_TAG
Tag-file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---7d7_LAYOUT_GENERAL_CACHE_TAG
Using :page_cache cache backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/page_cache, :id_prefix 7d7_}
Cleaning dir /home/peterjaap/development/workspace/client/magento2/var/page_cache/
Varnish request: #js {:protocol http:, :hostname 127.0.0.1, :port 6081, :method PURGE, :path /, :headers #js {:X-Magento-Tags-Pattern .*}, :timeout 10000}
Varnish request error:  connect ECONNREFUSED 127.0.0.1:6081
No further Varnish PURGEs will be attempted
Varnish request error:  connect ECONNREFUSED 127.0.0.1:6081
No further Varnish PURGEs will be attempted
Processing /home/peterjaap/development/workspace/client/magento2/app/code/Elgentos/ProductionTracking/view/layout/customer_account.xml
Cleaning cache type(s) layout full_page
Using :default cache_backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/cache, :id_prefix 7d7_}
Cleaning tag LAYOUT_GENERAL_CACHE_TAG
Tag-file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---7d7_LAYOUT_GENERAL_CACHE_TAG
Using :page_cache cache backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/page_cache, :id_prefix 7d7_}
Cleaning dir /home/peterjaap/development/workspace/client/magento2/var/page_cache/
Processing /home/peterjaap/development/workspace/client/magento2/app/code/Elgentos/ProductionTracking/view/frontend/layout/customer_account.xml
Cleaning cache type(s) layout full_page
Using :default cache_backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/cache, :id_prefix 7d7_}
Cleaning tag LAYOUT_GENERAL_CACHE_TAG
Tag-file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---7d7_LAYOUT_GENERAL_CACHE_TAG
Using :page_cache cache backend
Cache storage  {:backend Cm_Cache_Backend_File, :cache_dir /home/peterjaap/development/workspace/client/magento2/var/page_cache, :id_prefix 7d7_}
Cleaning dir /home/peterjaap/development/workspace/client/magento2/var/page_cache/

But when I refresh the page, the menu item I'm trying to change isn't actually updated...

[di-repo]

May I incorporate the docker examples you posted above into the README?

sure

[Vinai]

@peterjaap Did it work before?

[peterjaap]

@Vinai probably not, I just assumed (my bad) it worked because of the message it gave.

[Vinai]

@peterjaap I'm happy I didn't release this version already :)

I assume it is building the path to the files incorrectly. Maybe the calculation of the cache ID prefix is wrong (it's based on the path to the app/etc/ dir).

Can you confirm that the path /home/peterjaap/development/workspace/client/magento2/var/cache is correct for the system that the watcher is running on?

Can you please check what the correct path to the file /home/peterjaap/development/workspace/client/magento2/var/cache/mage-tags/mage---7d7_LAYOUT_GENERAL_CACHE_TAG is?

The cache id prefix might be different for example.

[Vinai]

The problem with the Varnish FPC Varnish request error: connect ECONNREFUSED 127.0.0.1:6081 is that it isn't configured to allow connections from a different IP by default. Depending on your setup you would have to expose the port 6081 or tunnel connections to that port into the Varnish container.

[di-repo]

Hi, sorry for being late to the party, but took some days off.
I usually don't develop with Varnish on, but gave it a try and it seems to work well. But needed some little tweaks.
For cache-clean to read the proper settings for Varnish you need them exported to env.php (@Vinai correct me if I'm wrong), to do that you have to:
bin/magento app:config:dump
and after this
bin/magento setup:config:set --http-cache-hosts varnish:80
where varnish is the service name of varnish container in docker-compose.
Now we are almost there but you won't be able to clean the cache due to Error 405 Method not allowed from Varnish. This means you are not allowed to purge the cache.
To allow this you have to edit the varnish.vcl that magento generates:

acl purge {
    "localhost"; # this entry will be here by default but doesn't work in docker context
    "fpm";  # this is my php-fpm container allows me to clean varnish from admin panel
    "cli"; # this is my php-cli container allows me to clean varnish from bin/magento
    "node"; # this is the node container used for cache-clean
}

So now this will work and you will be able to clean cache from all the listed containers. Unfortunately, there is one issue that may or may not bother you. Varnish resolves the DNS addresses on start, and if your cli/node containers are not running it will error. I don't want the cli and node running all the time, so I had to figure something.
First and simplest solution comment the ACL and the if in which it's used a few lines below:

        #if (client.ip !~ purge) {
        #    return (synth(405, "Method not allowed"));
        #}

this will allow everyone to run purge commands, not acceptable for serious work but totally ok for dev.
Second most plausible solution keep all referenced containers running so the DNS will resolve.
Third you can add there the address of the docker network, and just exclude the gateway address.
Fourth you can put the addresses in brackets like ("node") this will just ignore an address that can't be resolved, but you need to change the logic of the if to try to rescan the list of allowed addresses when it receives a purge request.
Probably there are even more solutions to this, but I am not really good at vcl.

@Vinai I've made a small mistake in the docker run command, it's better to add the container to the network all others are using. Without it works for redis, but not for varnish.

docker run -it --network docker_default -v $(pwd):/var/www/html -w /var/www/html node:10.15.1-alpine vendor/bin/cache-clean.js -w

In this case, I am using docker_default network.

@peterjaap if you can provide some more details of your Varnish config, I can try to help.

[Vinai]

@dimitar-ivanov-tryzens Thank you for your excellent and detailed description! You are a lot more familiar with setting up docker than me, so I really appreciate your help.
Sorry for the late reply - I'm currently traveling and I sometimes struggle to keep up with communication.
I think it's good to work with all caches including the full page cache enabled if possible, because otherwise it's easy to accidentally build something in Magento that doesn't work well with FPC enabled (at least for me). So that's why I'm happy about your information.

@peterjaap Any chance you can have another look at the feature and try if fixing the Varnish configuration makes it work for you? With a bit of luck your changes only didn't show up because of the cache-clean watcher was blocked by vcl.
@dimitar-ivanov-tryzens is right in that the Varnish configuration needs to be present in app/etc/env.php or in the config dump JSON file if that is used.

[Vinai]

Yesterday I released a new version with some bugfixes. I've rebased the branch and added a new build.

@peterjaap could you please do me a favor and update the watcher and try if the watcher still doesn't actually clear affected caches? It would be great to finally be able to close this issue for good :)

Here are the commands to update again:

composer remove --dev mage2tv/magento-cache-clean
composer require --dev mage2tv/magento-cache-clean:dev-app-config-file

[Vinai]

After some more tests I think in general things should work as expected. I'll go ahead and merge this branch into master and make a new release. The main motivation is I would like to limit the lifetime of feature branches.
If something then still needs tweaking I can then start from there.

[Vinai]

Published as release 0.0.41