A few issues when you use /pub as DocumentRoot

[hostep]

Preconditions (*)

Steps to reproduce:

robots.txt

Update: The robots.txt part is no longer valid, was fixed in Magento 2.2.0

sitemap.xml

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Marketing => Site Map
3. Click 'Add Sitemap' (small inconsistency btw, no space here)
4. Enter 'sitemap.xml' into the Filename field, enter '/' into the Path field
5. Click 'Save & generate'

Actual result (*)

Problem: sitemap.xml is created in the Magento root directory and in the column 'Link for Google' it says: http://example.com/sitemap.xml, which is incorrect

Continuing:
6. Click on the sitemap entry in the table
7. Change the Path to '/pub/'
8. Click 'Save & generate'

Problem: sitemap.xml is created in the pub directory, but the column 'Link for Google' now says: http://example.com/pub/sitemap.xml, which is also incorrect

In this case a solution with a redirect/rewrite in the .htaccess/nginx config file isn't really possible as the path and filename of the sitemap are dynamic.

Would be great if Magento (or somebody else) can come up with a way to solve this :)

Thanks!

Expected result (*)

[piotrekkaminski]

@buskamuza is it being addressed?

[gordonknoppe]

+1 this issue needs attention. We should be able to use pub/ to serve from production.

To clarify what I found:

sitemap.xml:

By setting up my sitemap.xml to generate into the pub/ directory I was able to place a file in the sub-directory fine and the resulting XML doesn't have any issues with pub/ in paths the file.

robots.txt:

I see two issues here: Magento will always write to the root folder and there is no way that I can see to change this via configuration. And additionally, if you have set the path of your sitemap.xml to pub/ as above then the robots.txt file created in the root has pub/ in it's "Sitemap: *" path.

[hostep]

I just tested this with Magento 2.0.0 GA, and the issues still remain.

robots.txt

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Stores => Configuration => General => Design => Search Engine Robots
3. Enter something in the Edit custom instruction of robots.txt File field
4. Save

Problem: the robots.txt file is created in the Magento root directory, not in the pub directory, so it isn't accessible for web requests.

Possible solution for serving Magento from both the root and the pub directory:
Always generate the robots.txt file in the pub directory, and add some line to the .htaccess/nginx config file only in the root directory which redirects you to the robots.txt file in the pub directory when requesting /robots.txt.
Downside: this might confuse shop owners/developers, if they want to add a robots.txt file to the root directory themselves without using the Magento backend, their file won't be used.
What do you think?

sitemap.xml

Steps to reproduce:

1. Make sure the Magento docroot is set to the pub directory
2. In the backend, go to Marketing => Site Map
3. Click 'Add Sitemap' (small inconsistency btw, no space here)
4. Enter 'sitemap.xml' into the Filename field, enter '/' into the Path field
5. Click 'Save & generate'

Problem: sitemap.xml is created in the Magento root directory and in the column 'Link for Google' it says: http://example.com/sitemap.xml, which is incorrect

Continuing:
6. Click on the sitemap entry in the table
7. Change the Path to '/pub/'
8. Click 'Save & generate'

Problem: sitemap.xml is created in the pub directory, but the column 'Link for Google' now says: http://example.com/pub/sitemap.xml, which is also incorrect

In this case a solution with a redirect/rewrite in the .htaccess/nginx config file isn't really possible as the path and filename of the sitemap are dynamic.

Would be great if Magento (or somebody else) can come up with a way to solve this :)

Thanks!

[mcspronko]

Hi @tanya-soroka and @piotrekkaminski

Do you guys have any updates for this ticket?

Thanks,
Max

[mcspronko]

Hi @piotrekkaminski

Do you have any update about this bug?

Thanks,
Max

[Marko-M]

Luckily you can symlink your way around it, but yep, it's an issue.

[mcspronko]

Hi @Marko-M

Imagine every time marketing department will be asking developer to symlink generated sitemap.xml files :)

[Marko-M]

@mcspronko

Yep, an issue.

[mcspronko]

Hi @tanya-soroka

Do you have any update for this issue?
Thanks,
Max

[markalston]

Still broken in 2.1.0

[atIOCrON]

I too hoped this would be resolved in 2.1... we need a robots.txt file accessible on the frontend urgently and don't want to mess around with the hacks suggested above.

[hostep]

@piotrekkaminski: maybe you should change the label from improvement to bug. I assume it will get picked up earlier if this is tagged as a bug, which it clearly is :)
Also I don't see a MAGETWO- reference number, so this isn't tracked in your internal Jira issues?
I do see a MAGETWO-47607 in #2802 which is related to this issue.

[piotrekkaminski]

I expanded scope of MAGETWO-47607 to include also robots issue.

[mcspronko]

Hi @piotrekkaminski

Any update on this issue?

cc: @pboisvert

[piotrekkaminski]

@mcspronko still in the queue to be fixed.

[mcspronko]

@piotrekkaminski it seems like it is not critical to have proper sitemap.xml and robots.txt files for marketing purposes. Let me fix this issue :), might be faster. So reported 2400 websites will benefit from having proper indexation by search engines.

[piotrekkaminski]

@mcspronko researching why it wasn't included in latest patch. I understand the importance of this issue.

[mcspronko]

Hi @piotrekkaminski

Any luck with findings?

[markalston]

I was able to workaround this problem by adding symlinks in pub to ../robots.txt and ../sitemap.xml respectively. Seems to work fine.

[liquidia]

still broken in 2.1.1

[magento-engcom-team]

Hi @limajasachin, here is your Magento instance.
Admin access: https://i-1146-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[limajasachin]

This issue is not reproducible.

[limajasachin]

I am working on this at #dmcdindia19

[michaeldyl520]

It can still be reproduced on 2.2.6.

[NetStorm84]

@magento give me 2.3-develop instance

[magento-engcom-team]

Hi @NetStorm84. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @NetStorm84, here is your Magento instance.
Admin access: https://i-1146-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[victortodoran]

This is still an issue in 2.3.1

[m2-assistant]

Hi @progreg. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[mcspronko]

Hey, @antonkril is this issue will ever be fixed?
The issue has been opened May 29, 2019, never been fixed and now a Bot requests additional details.

[progreg]

@mcspronko , an issue still reproducible and I have an idea how it can be fixed.

[ihor-sviziev]

Hi @mcspronko @progreg,
This issue actually fixes by adding one line in env.php file.
#1146 (comment)
Also I created issue for documenting it in magento/devdocs#4093, but it still wasn’t done.
I think it’s better to document this option and close this issue, or you prefer better solution?

[Bartlomiejsz]

Hi @ihor-sviziev, sorry to interfere but don't you think it would be good idea to at least add new configuration to installation process that would automatically set this line for new instances?

[progreg]

@ihor-sviziev, Yeap you are right, it will fix the problem. Missed up your comment, thanks for the notice, seems this issue no longer need and it can be closed.

[progreg]

@Bartlomiejsz We can open a new issue for this needs as this one have many unneeded details and also touches on Robots module issue that is no longer relevant.