feat: uri encode processing for attachment paths when querying attachments #1874

FanZeros · 2022-04-21T16:00:42Z

What this PR does?

在上传文件名带有特殊字符（#%）的文件时，在生成链接的过程中，会使用相近的符号代替它们
避免之后在使用或者查看的时候因为转码问题而找不到对应资源

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

修复附件名包含特殊字符无法访问的问题

f2c-ci-robot · 2022-04-21T16:00:51Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ruibaby after the PR has been reviewed.
You can assign the PR to them by writing /assign @ruibaby in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

CLAassistant · 2022-04-21T16:00:53Z

All committers have signed the CLA.

FanZeros · 2022-04-21T16:02:34Z

PS：该改动仅对新上传的内容有效，之前的错误数据并不会被修正

wan92hen · 2022-04-22T01:55:56Z

将生成链接中的文件名进行 urlencode 似乎更合理一些。

FanZeros · 2022-04-22T03:58:48Z

将生成链接中的文件名进行 urlencode 似乎更合理一些。

对文件名#% 进行 urlencode 之后，在文章引用的过程中，%会被二次urlencode，导致引用的错误
以至于需要更改更多内容
并且考虑到文件名中的#%并不是那么重要（比起urlencode成%23%25也更好分辨）

ruibaby · 2022-04-22T06:15:25Z

个人认为这样改可能会有点粗暴，如果说再有另外的字符也会导致此问题怎么办呢。

/cc @halo-dev/sig-halo

FanZeros · 2022-04-22T12:53:06Z

个人认为这样改可能会有点粗暴，如果说再有另外的字符也会导致此问题怎么办呢。

/cc @halo-dev/sig-halo

根据本人测试以及之前的issue中提到，暂时只发现这两个字符的问题
我觉得可以作为暂定的修复策略

JohnNiang

Hi @FanZeros ，如果能够解决这个问题：#1899 ，是不是就可以解决当前 PR 所想要解决的问题了呢。

guqing

guqing

考虑到兼容性问题，应该在此行对文件名进行 url encoding 处理，即只对返回结果编码

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

    
           String fullPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath()); 
        
           String fullThumbPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

可以如下操作

// 将 local 存储的链接中的文件名替换为编码后的文件名
String path = attachmentDTO.getPath()
                .replace(attachmentDTO.getName(), encodeValue(attachmentDTO.getName()));
// ....
// 附件缩略图亦如是
 String basename = FilenameUtils.getBasename(attachmentDTO.getName());
 String extension = FilenameUtils.getExtension(attachmentDTO.getName());
// 得到 thumbail name
 String thumbnailName = String.format("%s-thumbnail%s", basename, extension);
 String thumbnailPath = attachmentDTO.getThumbPath()
      .replace(thumbnailName, encodeValue(thumbnailName));

// ...
 private String encodeValue(String value) {
    return UriUtils.encode(value, StandardCharsets.UTF_8);
}

以上代码中关于缩略图文件名获取的方式需要考虑：后续如果缩略图规则改变不应该影响到此处功能，所以要确保本地上传策略生存缩略图和此处规则一致

ruibaby · 2022-04-28T03:26:38Z

考虑到兼容性问题，应该在此行对文件名进行 url encoding 处理，即只对返回结果编码

如果后端对此做了修改，那么前端所有编码的位置都需要移除。

FanZeros · 2022-04-29T02:26:56Z

Hi @FanZeros ，如果能够解决这个问题：#1899 ，是不是就可以解决当前 PR 所想要解决的问题了呢。

对，这应该也是个解决方法

JohnNiang · 2022-04-29T02:39:33Z

Hi @FanZeros 有兴趣尝试一下这种方法么：#1874 (review)

FanZeros · 2022-04-29T04:56:38Z

Hi @FanZeros 有兴趣尝试一下这种方法么：#1874 (review)

好的，我研究研究

FanZeros · 2022-05-12T14:43:38Z

考虑到兼容性问题，应该在此行对文件名进行 url encoding 处理，即只对返回结果编码

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

String fullPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath());

String fullThumbPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

可以如下操作
// 将 local 存储的链接中的文件名替换为编码后的文件名
String path = attachmentDTO.getPath()
                .replace(attachmentDTO.getName(), encodeValue(attachmentDTO.getName()));
// ....
// 附件缩略图亦如是
 String basename = FilenameUtils.getBasename(attachmentDTO.getName());
 String extension = FilenameUtils.getExtension(attachmentDTO.getName());
// 得到 thumbail name
 String thumbnailName = String.format("%s-thumbnail%s", basename, extension);
 String thumbnailPath = attachmentDTO.getThumbPath()
      .replace(thumbnailName, encodeValue(thumbnailName));

// ...
 private String encodeValue(String value) {
    return UriUtils.encode(value, StandardCharsets.UTF_8);
}
以上代码中关于缩略图文件名获取的方式需要考虑：后续如果缩略图规则改变不应该影响到此处功能，所以要确保本地上传策略生存缩略图和此处规则一致

暂时改动是对fullPath进行url encoding 处理，但thumbnailPath 不处理

前端部分进行如下修改：

https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentSelectModal.vue#L262
选择的时候不能二次编码
https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentDetailModal.vue#L66
https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentDetailModal.vue#L80
不能二次编码，不然用户复制的时候会二次编码出错

不过发现一个问题：encodeURI() 不会对#进行编码，但是java的UriUtils.encode() 会对#进行编码
该问题仅会影响带#的预览图
（目前只发现#，不清楚是否会对其他符号生效）

guqing · 2022-05-13T01:49:18Z

考虑到兼容性问题，应该在此行对文件名进行 url encoding 处理，即只对返回结果编码

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

String fullPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath());

String fullThumbPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

可以如下操作
// 将 local 存储的链接中的文件名替换为编码后的文件名
String path = attachmentDTO.getPath()
                .replace(attachmentDTO.getName(), encodeValue(attachmentDTO.getName()));
// ....
// 附件缩略图亦如是
 String basename = FilenameUtils.getBasename(attachmentDTO.getName());
 String extension = FilenameUtils.getExtension(attachmentDTO.getName());
// 得到 thumbail name
 String thumbnailName = String.format("%s-thumbnail%s", basename, extension);
 String thumbnailPath = attachmentDTO.getThumbPath()
      .replace(thumbnailName, encodeValue(thumbnailName));

// ...
 private String encodeValue(String value) {
    return UriUtils.encode(value, StandardCharsets.UTF_8);
}
以上代码中关于缩略图文件名获取的方式需要考虑：后续如果缩略图规则改变不应该影响到此处功能，所以要确保本地上传策略生存缩略图和此处规则一致
暂时改动是对fullPath进行url encoding 处理，但thumbnailPath 不处理

前端部分进行如下修改：

https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentSelectModal.vue#L262
选择的时候不能二次编码
https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentDetailModal.vue#L66
https://github.com/halo-dev/halo-admin/blob/master/src/components/Attachment/AttachmentDetailModal.vue#L80
不能二次编码，不然用户复制的时候会二次编码出错

不过发现一个问题：encodeURI() 不会对#进行编码，但是java的UriUtils.encode() 会对#进行编码
该问题仅会影响带#的预览图
（目前只发现#，不清楚是否会对其他符号生效）

只需要后端通过 UriUtils.encode() 对文件名部分进行编码即可然后把前端编码的部分删除统一由后端处理应该就可以解决问题了

JohnNiang

Hi @FanZeros ，我的建议也是在构建完整连接的时候，仅对 path 和 thumbnailPath 进行 encode。所有在 halo-admin encodeURL 的地方都需要移除，否则会造成多次 encode，无法正常访问。

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

    
           String fullPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath()); 
        
           String fullThumbPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

FanZeros · 2022-05-13T06:14:33Z

Hi @FanZeros ，我的建议也是在构建完整连接的时候，仅对 path 和 thumbnailPath 进行 encode。所有在 halo-admin encodeURL 的地方都需要移除，否则会造成多次 encode，无法正常访问。

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

String fullPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath());

String fullThumbPath = StringUtils

.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

嗯，我昨晚在尝试你们说的方法，在得到的缩略图链接处，去除encodeURI之后会在年月的“/”出现奇怪的%
https://github.com/halo-dev/halo-admin/blob/master/src/views/attachment/AttachmentList.vue#L88
Windows下测试，如：

理论值 upload/2022\05\2%23%252甘雨-1652370784582-thumbnail.png
去除后 upload/2022%05%02%23%252甘雨-1652370784582-thumbnail.png
去除前 upload/2022%5C05%5C2%2523%25252%25E7%2594%2598%25E9%259B%25A8-1652370784582-thumbnail.png

以及123.png
去除后 upload/2022%05%C4%A3-thumbnail.png
去除前 upload/2022%5C05%5C123-thumbnail.png

暂不清楚该情况是怎么发生的，还在进一步研究中

FanZeros · 2022-05-13T07:00:23Z

"upload/2022\05\2"

thumbPath 貌似有点问题

Windows下生成路径中的“\”感觉需要替换成“/”
如果对fullThumbPath进行replace("\", "/")是可以正常运行的

guqing · 2022-05-13T09:42:20Z

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java
如下图所示，将

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Line 127 in 913002d

attachment.setThumbPath(uploadResult.getThumbPath());

这块attachment.setThumbPath的值写成125行这样即可

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Line 125 in 913002d

attachment.setPath(HaloUtils.changeFileSeparatorToUrlSeparator(uploadResult.getFilePath()));

也就是：

attachment.setThumbPath(HaloUtils.changeFileSeparatorToUrlSeparator(uploadResult.getThumbPath()));

然后根据上面的描述UriUtils.encode处理这里即可

halo/src/main/java/run/halo/app/service/impl/AttachmentServiceImpl.java

Lines 177 to 180 in 913002d

    
           String fullPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath()); 
        
           String fullThumbPath = StringUtils 
        
               .join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

f2c-ci-robot · 2022-05-13T10:59:40Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ruibaby after the PR has been reviewed.
You can assign the PR to them by writing /assign @ruibaby in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

f2c-ci-robot · 2022-05-17T08:17:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

f2c-ci-robot · 2022-05-17T08:17:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

f2c-ci-robot · 2022-05-17T09:51:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing, JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang,guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

f2c-ci-robot · 2022-05-17T12:30:17Z

New changes are detected. LGTM label has been removed.

f2c-ci-robot · 2022-05-17T12:30:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing, JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang,guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ruibaby · 2022-05-17T12:54:12Z

@FanZeros 好像有 checkstyle 的错误哦 https://github.com/halo-dev/halo/pull/1874/checks

设置方法可以看这个 https://docs.halo.run/developer-guide/core/code-style

f2c-ci-robot · 2022-05-17T13:25:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing, JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang,guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

src/test/java/run/halo/app/service/impl/AttachmentServiceImplTest.java

f2c-ci-robot · 2022-05-17T14:27:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

当前该测试用例貌似并不能在全平台上成功运行，所以废弃

f2c-ci-robot · 2022-05-17T14:45:29Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

f2c-ci-robot · 2022-05-17T14:50:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing, JohnNiang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang,guqing]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JohnNiang

/lgtm

ruibaby

/approve

f2c-ci-robot · 2022-05-17T14:53:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guqing, JohnNiang, ruibaby

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JohnNiang,guqing,ruibaby]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JohnNiang · 2022-05-17T14:55:01Z

Hi @FanZeros ，非常感谢花时间修复此功能。在最后，请点击 Details 签署一下 Contributor License Agreenment 吧。

签署成功后，PR 会自动合并。

halo-dev-bot · 2022-05-17T15:22:54Z

@JohnNiang: new pull request created: #2094

In response to this:

/milestone 1.5.x
/cherrypick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Deal with illegal character

676f834

f2c-ci-robot bot requested review from ruibaby and wan92hen April 21, 2022 16:00

Update FilePathDescriptor.java

d278745

FanZeros changed the title ~~Deal with illegal character when upload~~ feat: Deal with illegal character when upload Apr 23, 2022

JohnNiang mentioned this pull request Apr 24, 2022

完善 CONTRIBUTION.md #1888

Closed

JohnNiang reviewed Apr 28, 2022

View reviewed changes

guqing reviewed Apr 28, 2022

View reviewed changes

guqing requested changes Apr 28, 2022

View reviewed changes

This was referenced May 6, 2022

附件列表的图片链接移除 encodeURI 编码 #1918

Closed

获取文件名时特殊字符没有转码 #1205

Closed

JohnNiang reviewed May 13, 2022

View reviewed changes

后端对链接进行encodeURI

455fcbb

f2c-ci-robot bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 17, 2022

guqing approved these changes May 17, 2022

View reviewed changes

Add some testcase for convertToDto

b31806b

f2c-ci-robot bot removed the lgtm Indicates that a PR is ready to be merged. label May 17, 2022

checkstyle

4b6f9ab

JohnNiang requested changes May 17, 2022

View reviewed changes

src/test/java/run/halo/app/service/impl/AttachmentServiceImplTest.java Outdated Show resolved Hide resolved

delete test

89b1d2f

当前该测试用例貌似并不能在全平台上成功运行，所以废弃

JohnNiang approved these changes May 17, 2022

View reviewed changes

JohnNiang reviewed May 17, 2022

View reviewed changes

f2c-ci-robot bot assigned JohnNiang May 17, 2022

f2c-ci-robot bot added the lgtm Indicates that a PR is ready to be merged. label May 17, 2022

ruibaby reviewed May 17, 2022

View reviewed changes

f2c-ci-robot bot merged commit 46220a9 into halo-dev:master May 17, 2022

halo-dev-bot mentioned this pull request May 17, 2022

[release-1.5] feat: uri encode processing for attachment paths when querying attachments #2094

Merged

ruibaby mentioned this pull request Jun 21, 2022

附件上传图片后缩略图方向错误 #2173

Closed

3 tasks

JohnNiang modified the milestones: 1.5.x, 1.5.4 Dec 19, 2022

	String fullPath = StringUtils
	.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getPath());
	String fullThumbPath = StringUtils
	.join(enabledAbsolutePath ? blogBaseUrl : "", "/", attachmentDTO.getThumbPath());

feat: uri encode processing for attachment paths when querying attachments #1874

feat: uri encode processing for attachment paths when querying attachments #1874

Conversation

FanZeros commented Apr 21, 2022 • edited by JohnNiang Loading

What this PR does?

Which issue(s) this PR fixes:

Does this PR introduce a user-facing change?

f2c-ci-robot bot commented Apr 21, 2022

CLAassistant commented Apr 21, 2022 • edited Loading

FanZeros commented Apr 21, 2022

wan92hen commented Apr 22, 2022

FanZeros commented Apr 22, 2022

ruibaby commented Apr 22, 2022

FanZeros commented Apr 22, 2022

JohnNiang left a comment

Choose a reason for hiding this comment

guqing left a comment • edited Loading

Choose a reason for hiding this comment

guqing left a comment • edited Loading

Choose a reason for hiding this comment

ruibaby commented Apr 28, 2022

FanZeros commented Apr 29, 2022

JohnNiang commented Apr 29, 2022

FanZeros commented Apr 29, 2022

FanZeros commented May 12, 2022

guqing commented May 13, 2022

JohnNiang left a comment

Choose a reason for hiding this comment

FanZeros commented May 13, 2022 • edited Loading

FanZeros commented May 13, 2022 • edited Loading

guqing commented May 13, 2022 • edited Loading

f2c-ci-robot bot commented May 13, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

ruibaby commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

f2c-ci-robot bot commented May 17, 2022

JohnNiang left a comment

Choose a reason for hiding this comment

ruibaby left a comment

Choose a reason for hiding this comment

f2c-ci-robot bot commented May 17, 2022

JohnNiang commented May 17, 2022

halo-dev-bot commented May 17, 2022

FanZeros commented Apr 21, 2022 •

edited by JohnNiang

Loading

CLAassistant commented Apr 21, 2022 •

edited

Loading

guqing left a comment •

edited

Loading

guqing left a comment •

edited

Loading

FanZeros commented May 13, 2022 •

edited

Loading

FanZeros commented May 13, 2022 •

edited

Loading

guqing commented May 13, 2022 •

edited

Loading